2025-08-14T21:23:04.3651122Z Current runner version: '2.328.0' 2025-08-14T21:23:04.3656285Z Runner name: 'i-0851ccaad4f014969' 2025-08-14T21:23:04.3657314Z Runner group name: 'default' 2025-08-14T21:23:04.3658060Z Machine name: 'ip-10-0-8-108' 2025-08-14T21:23:04.3660232Z ##[group]GITHUB_TOKEN Permissions 2025-08-14T21:23:04.3662461Z Contents: read 2025-08-14T21:23:04.3662898Z Metadata: read 2025-08-14T21:23:04.3663543Z ##[endgroup] 2025-08-14T21:23:04.3665835Z Secret source: Actions 2025-08-14T21:23:04.3666555Z Prepare workflow directory 2025-08-14T21:23:04.4070637Z Prepare all required actions 2025-08-14T21:23:04.4102931Z Getting action download info 2025-08-14T21:23:04.6931329Z Download action repository 'pytorch/test-infra@main' (SHA:83f58f391e939c10dcb8cb6d745e4cefa3b98a84) 2025-08-14T21:23:06.4201349Z Download action repository 'pytorch/pytorch@main' (SHA:3be70dc30e893b552fc0f23ca06cd8f7949b6d08) 2025-08-14T21:23:21.3694826Z Download action repository 'actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065' (SHA:a26af69be951a213d495a4c3e4e4022e16d87065) 2025-08-14T21:23:21.7563117Z Download action repository 'aws-actions/configure-aws-credentials@ececac1a45f3b08a01d2dd070d28d111c5fe6722' (SHA:ececac1a45f3b08a01d2dd070d28d111c5fe6722) 2025-08-14T21:23:22.0026855Z Download action repository 'aws-actions/amazon-ecr-login@062b18b96a7aff071d4dc91bc00c4c1a7945b076' (SHA:062b18b96a7aff071d4dc91bc00c4c1a7945b076) 2025-08-14T21:23:22.2356975Z Download action repository 'seemethere/upload-artifact-s3@baba72d0712b404f646cebe0730933554ebce96a' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-08-14T21:23:22.5481498Z Getting action download info 2025-08-14T21:23:22.6839882Z Download action repository 'actions/checkout@v4' (SHA:08eba0b27e820071cde6df949e0beb9ba4906955) 2025-08-14T21:23:22.9835165Z Getting action download info 2025-08-14T21:23:23.0893697Z Download action repository 'nick-fields/retry@v3.0.0' (SHA:7152eba30c6575329ac0576536151aca5a72780e) 2025-08-14T21:23:23.2858542Z Getting action download info 2025-08-14T21:23:23.3842381Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482) 2025-08-14T21:23:23.5681640Z Getting action download info 2025-08-14T21:23:23.7138161Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/heads/main (1fc683cf17c8c673044538d10266c00f92987be2) 2025-08-14T21:23:23.7141607Z ##[group] Inputs 2025-08-14T21:23:23.7141914Z build-environment: linux-jammy-py3.9-gcc11-build 2025-08-14T21:23:23.7144187Z test-matrix: {"include": [{"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_avx2_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}]} 2025-08-14T21:23:23.7146941Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:23:23.7147454Z sync-tag: 2025-08-14T21:23:23.7148119Z timeout-minutes: 240 2025-08-14T21:23:23.7148299Z use-gha: 2025-08-14T21:23:23.7148785Z dashboard-tag: 2025-08-14T21:23:23.7149048Z s3-bucket: gha-artifacts 2025-08-14T21:23:23.7149237Z aws-role-to-assume: 2025-08-14T21:23:23.7149661Z disable-monitor: false 2025-08-14T21:23:23.7149873Z monitor-log-interval: 5 2025-08-14T21:23:23.7150076Z monitor-data-collect-interval: 1 2025-08-14T21:23:23.7150289Z ##[endgroup] 2025-08-14T21:23:23.7150693Z Complete job name: linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T21:23:23.7627705Z A job started hook has been configured by the self-hosted runner administrator 2025-08-14T21:23:23.7705080Z ##[group]Run '/home/ec2-user/runner-scripts/before_job.sh' 2025-08-14T21:23:23.7712493Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:23:23.7713005Z ##[endgroup] 2025-08-14T21:23:24.7101066Z Runner Type: linux.8xlarge.amx 2025-08-14T21:23:24.7101577Z Instance Type: m7i-flex.8xlarge 2025-08-14T21:23:24.7101880Z AMI Name: unknown 2025-08-14T21:23:24.7129196Z AMI ID: ami-05ffe3c48a9991133 2025-08-14T21:23:29.0676429Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main 2025-08-14T21:23:29.0676837Z with: 2025-08-14T21:23:29.0677443Z github-secret: *** 2025-08-14T21:23:29.0678028Z instructions: All testing is done inside the container, to start an interactive session run: docker exec -it $(docker container ps --format '{{.ID}}') bash 2025-08-14T21:23:29.0678536Z activate-with-label: false 2025-08-14T21:23:29.0678812Z label: with-ssh 2025-08-14T21:23:29.0679091Z remove-existing-keys: true 2025-08-14T21:23:29.0679354Z fail-silently: true 2025-08-14T21:23:29.0679599Z env: 2025-08-14T21:23:29.0679964Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:23:29.0680231Z ##[endgroup] 2025-08-14T21:23:29.2039413Z Please see https://github.com/pytorch/pytorch/wiki/Debugging-using-with-ssh-for-Github-Actions for more info. 2025-08-14T21:23:29.2040686Z Not on pull request and ciflow reference could not be extracted, skipping adding ssh keys 2025-08-14T21:23:29.2423888Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@main 2025-08-14T21:23:29.2424264Z with: 2025-08-14T21:23:29.2424517Z no-sudo: true 2025-08-14T21:23:29.2424772Z submodules: recursive 2025-08-14T21:23:29.2425054Z fetch-depth: 0 2025-08-14T21:23:29.2425267Z env: 2025-08-14T21:23:29.2425512Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:23:29.2425800Z ##[endgroup] 2025-08-14T21:23:29.2579145Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-08-14T21:23:29.2579734Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-08-14T21:23:29.2588388Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:23:29.2588644Z env: 2025-08-14T21:23:29.2588816Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:23:29.2589006Z ##[endgroup] 2025-08-14T21:23:29.2680741Z ##[group]Run # Use all available CPUs for fetching 2025-08-14T21:23:29.2681068Z # Use all available CPUs for fetching 2025-08-14T21:23:29.2681314Z cd "${GITHUB_WORKSPACE}" 2025-08-14T21:23:29.2681594Z git config --global fetch.parallel 0 2025-08-14T21:23:29.2681843Z git config --global submodule.fetchJobs 0 2025-08-14T21:23:29.2682056Z  2025-08-14T21:23:29.2682280Z # Clean workspace. The default checkout action should also do this, but 2025-08-14T21:23:29.2682570Z # do it here as well just in case 2025-08-14T21:23:29.2682777Z if [[ -d .git ]]; then 2025-08-14T21:23:29.2682969Z  if [ -z "${NO_SUDO}" ]; then 2025-08-14T21:23:29.2683177Z  sudo git clean -ffdx 2025-08-14T21:23:29.2683367Z  else 2025-08-14T21:23:29.2683526Z  git clean -ffdx 2025-08-14T21:23:29.2683703Z  fi 2025-08-14T21:23:29.2683853Z fi 2025-08-14T21:23:29.2688601Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:23:29.2688979Z env: 2025-08-14T21:23:29.2689220Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:23:29.2689418Z NO_SUDO: true 2025-08-14T21:23:29.2689572Z ##[endgroup] 2025-08-14T21:23:29.2790851Z ##[group]Run actions/checkout@v4 2025-08-14T21:23:29.2791092Z with: 2025-08-14T21:23:29.2791288Z ref: 1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:23:29.2791514Z fetch-depth: 0 2025-08-14T21:23:29.2791699Z submodules: recursive 2025-08-14T21:23:29.2791890Z show-progress: false 2025-08-14T21:23:29.2792089Z repository: pytorch/pytorch 2025-08-14T21:23:29.2792390Z token: *** 2025-08-14T21:23:29.2792560Z ssh-strict: true 2025-08-14T21:23:29.2792733Z ssh-user: git 2025-08-14T21:23:29.2792910Z persist-credentials: true 2025-08-14T21:23:29.2793113Z clean: true 2025-08-14T21:23:29.2793313Z sparse-checkout-cone-mode: true 2025-08-14T21:23:29.2793523Z fetch-tags: false 2025-08-14T21:23:29.2793698Z lfs: false 2025-08-14T21:23:29.2793878Z set-safe-directory: true 2025-08-14T21:23:29.2794093Z env: 2025-08-14T21:23:29.2794264Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:23:29.2794460Z ##[endgroup] 2025-08-14T21:23:29.3729680Z Syncing repository: pytorch/pytorch 2025-08-14T21:23:29.3730815Z ##[group]Getting Git version info 2025-08-14T21:23:29.3731157Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-08-14T21:23:29.3731620Z [command]/usr/bin/git version 2025-08-14T21:23:29.3942018Z git version 2.47.1 2025-08-14T21:23:29.3963009Z ##[endgroup] 2025-08-14T21:23:29.3975248Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/7b2a53c4-0ad2-4aa5-b8fb-fadb40f3bdb1/.gitconfig' 2025-08-14T21:23:29.4004339Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/7b2a53c4-0ad2-4aa5-b8fb-fadb40f3bdb1' before making global git config changes 2025-08-14T21:23:29.4005210Z Adding repository directory to the temporary git global config as a safe directory 2025-08-14T21:23:29.4014233Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-08-14T21:23:29.4076076Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch' 2025-08-14T21:23:29.4081702Z ##[group]Initializing the repository 2025-08-14T21:23:29.4083351Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-08-14T21:23:29.4136644Z hint: Using 'master' as the name for the initial branch. This default branch name 2025-08-14T21:23:29.4137129Z hint: is subject to change. To configure the initial branch name to use in all 2025-08-14T21:23:29.4137516Z hint: of your new repositories, which will suppress this warning, call: 2025-08-14T21:23:29.4138049Z hint: 2025-08-14T21:23:29.4138284Z hint: git config --global init.defaultBranch 2025-08-14T21:23:29.4138505Z hint: 2025-08-14T21:23:29.4138728Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and 2025-08-14T21:23:29.4139096Z hint: 'development'. The just-created branch can be renamed via this command: 2025-08-14T21:23:29.4139367Z hint: 2025-08-14T21:23:29.4139529Z hint: git branch -m 2025-08-14T21:23:29.4162990Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/ 2025-08-14T21:23:29.4166134Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch 2025-08-14T21:23:29.4212245Z ##[endgroup] 2025-08-14T21:23:29.4212599Z ##[group]Disabling automatic garbage collection 2025-08-14T21:23:29.4213703Z [command]/usr/bin/git config --local gc.auto 0 2025-08-14T21:23:29.4246928Z ##[endgroup] 2025-08-14T21:23:29.4247355Z ##[group]Setting up auth 2025-08-14T21:23:29.4257106Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-08-14T21:23:29.4282711Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-08-14T21:23:29.4675522Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-08-14T21:23:29.4705994Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-08-14T21:23:29.5031585Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-08-14T21:23:29.5112660Z ##[endgroup] 2025-08-14T21:23:29.5113052Z ##[group]Fetching the repository 2025-08-14T21:23:29.5121709Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/* 2025-08-14T21:24:12.5517141Z From https://github.com/pytorch/pytorch 2025-08-14T21:24:12.5519261Z * [new branch] 2.6.0.dev20241004+ -> origin/2.6.0.dev20241004+ 2025-08-14T21:24:12.5524427Z * [new branch] 5addvllmbuild -> origin/5addvllmbuild 2025-08-14T21:24:12.5526204Z * [new branch] AaronWang04_addmmfusion_perftest -> origin/AaronWang04_addmmfusion_perftest 2025-08-14T21:24:12.5526768Z * [new branch] HDCharles-2.6.0-release-notes -> origin/HDCharles-2.6.0-release-notes 2025-08-14T21:24:12.5530947Z * [new branch] JackCaoG/dynamo_make_fx_non_core_aten_ops -> origin/JackCaoG/dynamo_make_fx_non_core_aten_ops 2025-08-14T21:24:12.5531432Z * [new branch] PR-AOTInductorNoneBug -> origin/PR-AOTInductorNoneBug 2025-08-14T21:24:12.5531854Z * [new branch] PR-AOTInductorNoneBugFix -> origin/PR-AOTInductorNoneBugFix 2025-08-14T21:24:12.5532253Z * [new branch] PR-FixConfigsIssue -> origin/PR-FixConfigsIssue 2025-08-14T21:24:12.5532625Z * [new branch] PR-NoneBugFix-viable -> origin/PR-NoneBugFix-viable 2025-08-14T21:24:12.5532988Z * [new branch] PR-ResetToZero -> origin/PR-ResetToZero 2025-08-14T21:24:12.5539707Z * [new branch] Update-Flash-Packaging -> origin/Update-Flash-Packaging 2025-08-14T21:24:12.5540270Z * [new branch] add-missing-args-normalization -> origin/add-missing-args-normalization 2025-08-14T21:24:12.5540770Z * [new branch] add-user-guide-structure -> origin/add-user-guide-structure 2025-08-14T21:24:12.5541147Z * [new branch] addVllmPin -> origin/addVllmPin 2025-08-14T21:24:12.5541515Z * [new branch] add_windows_testing_back -> origin/add_windows_testing_back 2025-08-14T21:24:12.5541867Z * [new branch] addbuildvllm -> origin/addbuildvllm 2025-08-14T21:24:12.5542203Z * [new branch] addmm-heuristic -> origin/addmm-heuristic 2025-08-14T21:24:12.5542535Z * [new branch] addsimde -> origin/addsimde 2025-08-14T21:24:12.5542866Z * [new branch] addvllpinnedfile -> origin/addvllpinnedfile 2025-08-14T21:24:12.5543224Z * [new branch] adi/acl_upgrade -> origin/adi/acl_upgrade 2025-08-14T21:24:12.5543543Z * [new branch] adi/skip_slow_tests -> origin/adi/skip_slow_tests 2025-08-14T21:24:12.5543850Z * [new branch] adi/test -> origin/adi/test 2025-08-14T21:24:12.5544137Z * [new branch] adi/test_bgemm -> origin/adi/test_bgemm 2025-08-14T21:24:12.5544439Z * [new branch] adi/test_fusions -> origin/adi/test_fusions 2025-08-14T21:24:12.5544751Z * [new branch] adi/test_onednn_v3.9 -> origin/adi/test_onednn_v3.9 2025-08-14T21:24:12.5545082Z * [new branch] adi/test_presve_change -> origin/adi/test_presve_change 2025-08-14T21:24:12.5545400Z * [new branch] adi/test_timm -> origin/adi/test_timm 2025-08-14T21:24:12.5545714Z * [new branch] adi/testpresve_change -> origin/adi/testpresve_change 2025-08-14T21:24:12.5546174Z * [new branch] aditew01/test/vec_bf16 -> origin/aditew01/test/vec_bf16 2025-08-14T21:24:12.5546650Z * [new branch] ah-globalfeedback-hook -> origin/ah-globalfeedback-hook 2025-08-14T21:24:12.5547012Z * [new branch] albanD-patch-1 -> origin/albanD-patch-1 2025-08-14T21:24:12.5547343Z * [new branch] alt-disable -> origin/alt-disable 2025-08-14T21:24:12.5547730Z * [new branch] angelayi/aoti_additional_files -> origin/angelayi/aoti_additional_files 2025-08-14T21:24:12.5548151Z * [new branch] angelayi/aoti_inductor_fx -> origin/angelayi/aoti_inductor_fx 2025-08-14T21:24:12.5548607Z * [new branch] angelayi/assert_tensor_metadata_device -> origin/angelayi/assert_tensor_metadata_device 2025-08-14T21:24:12.5548999Z * [new branch] angelayi/benchmark -> origin/angelayi/benchmark 2025-08-14T21:24:12.5549338Z * [new branch] angelayi/benchmark2 -> origin/angelayi/benchmark2 2025-08-14T21:24:12.5549738Z * [new branch] angelayi/change_pytree_serialization -> origin/angelayi/change_pytree_serialization 2025-08-14T21:24:12.5550137Z * [new branch] angelayi/cpp_loader -> origin/angelayi/cpp_loader 2025-08-14T21:24:12.5550485Z * [new branch] angelayi/custom_op_subgraph -> origin/angelayi/custom_op_subgraph 2025-08-14T21:24:12.5550857Z * [new branch] angelayi/customop -> origin/angelayi/customop 2025-08-14T21:24:12.5551176Z * [new branch] angelayi/del_lib -> origin/angelayi/del_lib 2025-08-14T21:24:12.5551485Z * [new branch] angelayi/docs -> origin/angelayi/docs 2025-08-14T21:24:12.5551780Z * [new branch] angelayi/docs2 -> origin/angelayi/docs2 2025-08-14T21:24:12.5552104Z * [new branch] angelayi/fix_pt2 -> origin/angelayi/fix_pt2 2025-08-14T21:24:12.5552520Z * [new branch] angelayi/logging.bak -> origin/angelayi/logging.bak 2025-08-14T21:24:12.5552889Z * [new branch] angelayi/logging2 -> origin/angelayi/logging2 2025-08-14T21:24:12.5553235Z * [new branch] angelayi/no_so_weight -> origin/angelayi/no_so_weight 2025-08-14T21:24:12.5553560Z * [new branch] angelayi/pytree -> origin/angelayi/pytree 2025-08-14T21:24:12.5553875Z * [new branch] angelayi/save_error -> origin/angelayi/save_error 2025-08-14T21:24:12.5554193Z * [new branch] angelayi/scan_layers -> origin/angelayi/scan_layers 2025-08-14T21:24:12.5554541Z * [new branch] angelayi/symint_input -> origin/angelayi/symint_input 2025-08-14T21:24:12.5554953Z * [new branch] angelayi/tensor_nn_module_meta -> origin/angelayi/tensor_nn_module_meta 2025-08-14T21:24:12.5555317Z * [new branch] angelayi/torch_size -> origin/angelayi/torch_size 2025-08-14T21:24:12.5555632Z * [new branch] aoti-cuda-alloc -> origin/aoti-cuda-alloc 2025-08-14T21:24:12.5555948Z * [new branch] aoti_weight_sharing -> origin/aoti_weight_sharing 2025-08-14T21:24:12.5556290Z * [new branch] arsh/symint_mm_ind_decomp -> origin/arsh/symint_mm_ind_decomp 2025-08-14T21:24:12.5556663Z * [new branch] atalman-inductor-perf-cu124 -> origin/atalman-inductor-perf-cu124 2025-08-14T21:24:12.5557058Z * [new branch] atalman-inductor-perf-cu124.1 -> origin/atalman-inductor-perf-cu124.1 2025-08-14T21:24:12.5557420Z * [new branch] atalman-patch-1 -> origin/atalman-patch-1 2025-08-14T21:24:12.5559407Z * [new branch] atalman-patch-2 -> origin/atalman-patch-2 2025-08-14T21:24:12.5559874Z * [new branch] atalman-patch-3 -> origin/atalman-patch-3 2025-08-14T21:24:12.5560294Z * [new branch] atalman-patch-6 -> origin/atalman-patch-6 2025-08-14T21:24:12.5560814Z * [new branch] atalman-patch-7 -> origin/atalman-patch-7 2025-08-14T21:24:12.5561762Z * [new branch] atalman-patch-8 -> origin/atalman-patch-8 2025-08-14T21:24:12.5562425Z * [new branch] atalman_inductor_2.3.0 -> origin/atalman_inductor_2.3.0 2025-08-14T21:24:12.5562830Z * [new branch] atalman_inductor_2.3.1 -> origin/atalman_inductor_2.3.1 2025-08-14T21:24:12.5563217Z * [new branch] atalman_inductor_2.4.0 -> origin/atalman_inductor_2.4.0 2025-08-14T21:24:12.5563570Z * [new branch] atalman_inductor_2.4.x -> origin/atalman_inductor_2.4.x 2025-08-14T21:24:12.5564039Z * [new branch] autoupdate-transformers-pin-via-pr -> origin/autoupdate-transformers-pin-via-pr 2025-08-14T21:24:12.5564478Z * [new branch] backupvllm -> origin/backupvllm 2025-08-14T21:24:12.5564844Z * [new branch] base/1.5 -> origin/base/1.5 2025-08-14T21:24:12.5565248Z * [new branch] batching_sdpa_efficient_attention -> origin/batching_sdpa_efficient_attention 2025-08-14T21:24:12.5565942Z * [new branch] benchmark-updates -> origin/benchmark-updates 2025-08-14T21:24:12.5566321Z * [new branch] benchmarking-script -> origin/benchmarking-script 2025-08-14T21:24:12.5571398Z * [new branch] benjaminglass1/mark-large-tensor-tests-serial -> origin/benjaminglass1/mark-large-tensor-tests-serial 2025-08-14T21:24:12.5576777Z * [new branch] bertmaher/pinbump26 -> origin/bertmaher/pinbump26 2025-08-14T21:24:12.5580026Z * [new branch] bertrand/cutlass -> origin/bertrand/cutlass 2025-08-14T21:24:12.5584852Z * [new branch] bf/cg-log -> origin/bf/cg-log 2025-08-14T21:24:12.5589774Z * [new branch] bf/cg-remove-check -> origin/bf/cg-remove-check 2025-08-14T21:24:12.5595694Z * [new branch] bf/cg-skip-1-kernel -> origin/bf/cg-skip-1-kernel 2025-08-14T21:24:12.5600329Z * [new branch] bf/cudagraph -> origin/bf/cudagraph 2025-08-14T21:24:12.5600899Z * [new branch] bf/cudagraph-disable-input-mutation -> origin/bf/cudagraph-disable-input-mutation 2025-08-14T21:24:12.5601634Z * [new branch] bf/cudagraph-enable-input-mutation-support-benchmark -> origin/bf/cudagraph-enable-input-mutation-support-benchmark 2025-08-14T21:24:12.5602234Z * [new branch] bf/cudagraph-partition -> origin/bf/cudagraph-partition 2025-08-14T21:24:12.5602658Z * [new branch] bf/default-recompile-reason -> origin/bf/default-recompile-reason 2025-08-14T21:24:12.5603096Z * [new branch] bf/donated-buffer-bench -> origin/bf/donated-buffer-bench 2025-08-14T21:24:12.5603510Z * [new branch] bf/improve-kernel-bench -> origin/bf/improve-kernel-bench 2025-08-14T21:24:12.5603921Z * [new branch] bf/kernel-benchmark -> origin/bf/kernel-benchmark 2025-08-14T21:24:12.5604301Z * [new branch] bf/partition-doc -> origin/bf/partition-doc 2025-08-14T21:24:12.5604674Z * [new branch] bf/partition-move-cpu -> origin/bf/partition-move-cpu 2025-08-14T21:24:12.5605050Z * [new branch] bf/partition-turn-on -> origin/bf/partition-turn-on 2025-08-14T21:24:12.5605461Z * [new branch] bf/remove-check-55b0c39d -> origin/bf/remove-check-55b0c39d 2025-08-14T21:24:12.5606040Z * [new branch] bf/skip-asserts -> origin/bf/skip-asserts 2025-08-14T21:24:12.5606380Z * [new branch] bf16adamw -> origin/bf16adamw 2025-08-14T21:24:12.5606749Z * [new branch] bisect_perf_hf_T5_3acc6eac492 -> origin/bisect_perf_hf_T5_3acc6eac492 2025-08-14T21:24:12.5607169Z * [new branch] bisect_perf_hf_T5_3fcf66f61fb -> origin/bisect_perf_hf_T5_3fcf66f61fb 2025-08-14T21:24:12.5607721Z * [new branch] bisect_perf_hf_T5_4009d154129 -> origin/bisect_perf_hf_T5_4009d154129 2025-08-14T21:24:12.5608097Z * [new branch] bisect_perf_hf_T5_40d0740e73d -> origin/bisect_perf_hf_T5_40d0740e73d 2025-08-14T21:24:12.5608462Z * [new branch] bisect_perf_hf_T5_5268754e -> origin/bisect_perf_hf_T5_5268754e 2025-08-14T21:24:12.5608838Z * [new branch] bisect_perf_hf_T5_7d89a8d385c -> origin/bisect_perf_hf_T5_7d89a8d385c 2025-08-14T21:24:12.5609215Z * [new branch] bisect_perf_hf_T5_b7a25c1ee7c -> origin/bisect_perf_hf_T5_b7a25c1ee7c 2025-08-14T21:24:12.5609596Z * [new branch] bisect_perf_hf_T5_c25b201583f -> origin/bisect_perf_hf_T5_c25b201583f 2025-08-14T21:24:12.5609959Z * [new branch] bisect_perf_hf_T5_c93e57efac0 -> origin/bisect_perf_hf_T5_c93e57efac0 2025-08-14T21:24:12.5610336Z * [new branch] bisect_perf_hf_T5_ca9813ea149 -> origin/bisect_perf_hf_T5_ca9813ea149 2025-08-14T21:24:12.5610707Z * [new branch] bisect_perf_hf_T5_d65f194a -> origin/bisect_perf_hf_T5_d65f194a 2025-08-14T21:24:12.5611062Z * [new branch] bisect_perf_hf_T5_da94ab0b -> origin/bisect_perf_hf_T5_da94ab0b 2025-08-14T21:24:12.5611434Z * [new branch] bisect_perf_hf_T5_da94ab0b_new -> origin/bisect_perf_hf_T5_da94ab0b_new 2025-08-14T21:24:12.5611816Z * [new branch] bisect_perf_hf_T5_db4e8a1d8a8 -> origin/bisect_perf_hf_T5_db4e8a1d8a8 2025-08-14T21:24:12.5612184Z * [new branch] bisect_perf_hf_T5_e0d97e936a2 -> origin/bisect_perf_hf_T5_e0d97e936a2 2025-08-14T21:24:12.5612545Z * [new branch] bisect_perf_hf_T5_f23621ec563 -> origin/bisect_perf_hf_T5_f23621ec563 2025-08-14T21:24:12.5612921Z * [new branch] bowbao/bench_updates_stage -> origin/bowbao/bench_updates_stage 2025-08-14T21:24:12.5613289Z * [new branch] bowbao/dort_rewriter -> origin/bowbao/dort_rewriter 2025-08-14T21:24:12.5613683Z * [new branch] bowbao/wip_prs -> origin/bowbao/wip_prs 2025-08-14T21:24:12.5614057Z * [new branch] bowenbao/partial_min_max_reduce -> origin/bowenbao/partial_min_max_reduce 2025-08-14T21:24:12.5614470Z * [new branch] brister/always_wrapper_ir -> origin/brister/always_wrapper_ir 2025-08-14T21:24:12.5614844Z * [new branch] brister/flatten_contig -> origin/brister/flatten_contig 2025-08-14T21:24:12.5615226Z * [new branch] brister/test_block_ptr_same -> origin/brister/test_block_ptr_same 2025-08-14T21:24:12.5615646Z * [new branch] brister/tiled_reduction_no_numel_check -> origin/brister/tiled_reduction_no_numel_check 2025-08-14T21:24:12.5616039Z * [new branch] c57382a49 -> origin/c57382a49 2025-08-14T21:24:12.5616341Z * [new branch] ca_0431d47eaa -> origin/ca_0431d47eaa 2025-08-14T21:24:12.5616659Z * [new branch] ca_fix_0431d47eaa -> origin/ca_fix_0431d47eaa 2025-08-14T21:24:12.5617269Z * [new branch] camyll/revert-94bc900da97ad7f3c35b3b819bb53b23c74b581a-for-release-2.8 -> origin/camyll/revert-94bc900da97ad7f3c35b3b819bb53b23c74b581a-for-release-2.8 2025-08-14T21:24:12.5617949Z * [new branch] camyll/test_precommit_hooks_lintrunner -> origin/camyll/test_precommit_hooks_lintrunner 2025-08-14T21:24:12.5618458Z * [new branch] camyllh/cherrypick-151547-for-release28 -> origin/camyllh/cherrypick-151547-for-release28 2025-08-14T21:24:12.5618922Z * [new branch] camyllh/test_setup_hooks_push -> origin/camyllh/test_setup_hooks_push 2025-08-14T21:24:12.5619431Z * [new branch] cherry-pick-149654-by-pytorch_bot_bot_ -> origin/cherry-pick-149654-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5619912Z * [new branch] cherry-pick-151939-by-pytorch_bot_bot_ -> origin/cherry-pick-151939-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5620386Z * [new branch] cherry-pick-154174-by-pytorch_bot_bot_ -> origin/cherry-pick-154174-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5620994Z * [new branch] cherry-pick-155896-by-pytorch_bot_bot_ -> origin/cherry-pick-155896-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5621471Z * [new branch] cherry-pick-156260-by-pytorch_bot_bot_ -> origin/cherry-pick-156260-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5621955Z * [new branch] cherry-pick-156719-by-pytorch_bot_bot_ -> origin/cherry-pick-156719-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5622435Z * [new branch] cherry-pick-156876-by-pytorch_bot_bot_ -> origin/cherry-pick-156876-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5622915Z * [new branch] cherry-pick-156888-by-pytorch_bot_bot_ -> origin/cherry-pick-156888-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5623389Z * [new branch] cherry-pick-157014-by-pytorch_bot_bot_ -> origin/cherry-pick-157014-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5623873Z * [new branch] cherry-pick-157179-by-pytorch_bot_bot_ -> origin/cherry-pick-157179-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5624401Z * [new branch] cherry-pick-157453-by-pytorch_bot_bot_ -> origin/cherry-pick-157453-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5624890Z * [new branch] cherry-pick-157513-by-pytorch_bot_bot_ -> origin/cherry-pick-157513-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5625361Z * [new branch] cherry-pick-157558-by-pytorch_bot_bot_ -> origin/cherry-pick-157558-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5625842Z * [new branch] cherry-pick-157598-by-pytorch_bot_bot_ -> origin/cherry-pick-157598-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5626318Z * [new branch] cherry-pick-157600-by-pytorch_bot_bot_ -> origin/cherry-pick-157600-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5626802Z * [new branch] cherry-pick-157630-by-pytorch_bot_bot_ -> origin/cherry-pick-157630-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5627327Z * [new branch] cherry-pick-157695-by-pytorch_bot_bot_ -> origin/cherry-pick-157695-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5627812Z * [new branch] cherry-pick-157732-by-pytorch_bot_bot_ -> origin/cherry-pick-157732-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5628292Z * [new branch] cherry-pick-157733-by-pytorch_bot_bot_ -> origin/cherry-pick-157733-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5628768Z * [new branch] cherry-pick-157985-by-pytorch_bot_bot_ -> origin/cherry-pick-157985-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5629245Z * [new branch] cherry-pick-157993-by-pytorch_bot_bot_ -> origin/cherry-pick-157993-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5629786Z * [new branch] cherry-pick-158064-by-pytorch_bot_bot_ -> origin/cherry-pick-158064-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5630275Z * [new branch] cherry-pick-158152-by-pytorch_bot_bot_ -> origin/cherry-pick-158152-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5630769Z * [new branch] cherry-pick-158295-by-pytorch_bot_bot_ -> origin/cherry-pick-158295-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5631278Z * [new branch] cherry-pick-158301-by-pytorch_bot_bot_ -> origin/cherry-pick-158301-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5631773Z * [new branch] cherry-pick-158537-by-pytorch_bot_bot_ -> origin/cherry-pick-158537-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5632269Z * [new branch] cherry-pick-158572-by-pytorch_bot_bot_ -> origin/cherry-pick-158572-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5632706Z * [new branch] cherry-pick-158595 -> origin/cherry-pick-158595 2025-08-14T21:24:12.5633137Z * [new branch] cherry-pick-159181-by-pytorch_bot_bot_ -> origin/cherry-pick-159181-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5633635Z * [new branch] cherry-pick-159969-by-pytorch_bot_bot_ -> origin/cherry-pick-159969-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5634131Z * [new branch] cherry-pick-160586-by-pytorch_bot_bot_ -> origin/cherry-pick-160586-by-pytorch_bot_bot_ 2025-08-14T21:24:12.5634617Z * [new branch] cherry-pick-PR-158746 -> origin/cherry-pick-PR-158746 2025-08-14T21:24:12.5635119Z * [new branch] cherrypick-e4e2701429c17078c3c475382a8b1fa4c8a8cefc -> origin/cherrypick-e4e2701429c17078c3c475382a8b1fa4c8a8cefc 2025-08-14T21:24:12.5635600Z * [new branch] chilli/flex_vllm -> origin/chilli/flex_vllm 2025-08-14T21:24:12.5635936Z * [new branch] ckluk2-compileThread-1 -> origin/ckluk2-compileThread-1 2025-08-14T21:24:12.5636292Z * [new branch] ckluk2-compileThread-2 -> origin/ckluk2-compileThread-2 2025-08-14T21:24:12.5636640Z * [new branch] ckluk2-compileThread-64 -> origin/ckluk2-compileThread-64 2025-08-14T21:24:12.5637119Z * [new branch] ckluk2-test-1 -> origin/ckluk2-test-1 2025-08-14T21:24:12.5637444Z * [new branch] cleantest1 -> origin/cleantest1 2025-08-14T21:24:12.5638044Z * [new branch] codex-testing -> origin/codex-testing 2025-08-14T21:24:12.5638548Z * [new branch] codex/create-test-for-tensor-memory-leak-in-cudagraph -> origin/codex/create-test-for-tensor-memory-leak-in-cudagraph 2025-08-14T21:24:12.5639112Z * [new branch] codex/fix-issue-121219-in-pytorch -> origin/codex/fix-issue-121219-in-pytorch 2025-08-14T21:24:12.5639537Z * [new branch] codex/fix-issue-160415-in-pytorch -> origin/codex/fix-issue-160415-in-pytorch 2025-08-14T21:24:12.5640051Z * [new branch] codex/fix-noqengine-quantized-engine-support -> origin/codex/fix-noqengine-quantized-engine-support 2025-08-14T21:24:12.5640558Z * [new branch] codex/fix-pin_memory-error-handling -> origin/codex/fix-pin_memory-error-handling 2025-08-14T21:24:12.5641025Z * [new branch] codex/propose-fix-for-issue-160332 -> origin/codex/propose-fix-for-issue-160332 2025-08-14T21:24:12.5641717Z * [new branch] codex/refactor-lintrunner-config-to-use-uv-run -> origin/codex/refactor-lintrunner-config-to-use-uv-run 2025-08-14T21:24:12.5642325Z * [new branch] codex/verify-torch-output-and-log-results -> origin/codex/verify-torch-output-and-log-results 2025-08-14T21:24:12.5642840Z * [new branch] compile_fsdp2_disable_stream_and_event -> origin/compile_fsdp2_disable_stream_and_event 2025-08-14T21:24:12.5643279Z * [new branch] comply-with-setuptools -> origin/comply-with-setuptools 2025-08-14T21:24:12.5643652Z * [new branch] context_test -> origin/context_test 2025-08-14T21:24:12.5644002Z * [new branch] copilot/fix-157446 -> origin/copilot/fix-157446 2025-08-14T21:24:12.5644353Z * [new branch] copilot/fix-159257 -> origin/copilot/fix-159257 2025-08-14T21:24:12.5644696Z * [new branch] copy_graph -> origin/copy_graph 2025-08-14T21:24:12.5645054Z * [new branch] cpio/fix_new_ami_tests -> origin/cpio/fix_new_ami_tests 2025-08-14T21:24:12.5645435Z * [new branch] csl/3_proc_sm -> origin/csl/3_proc_sm 2025-08-14T21:24:12.5645871Z * [new branch] csl/add_file_merge_conflict_csv -> origin/csl/add_file_merge_conflict_csv 2025-08-14T21:24:12.5646276Z * [new branch] csl/always_produce_xml -> origin/csl/always_produce_xml 2025-08-14T21:24:12.5646865Z * [new branch] csl/build_test_more_procs -> origin/csl/build_test_more_procs 2025-08-14T21:24:12.5652792Z * [new branch] csl/build_test_more_procs2 -> origin/csl/build_test_more_procs2 2025-08-14T21:24:12.5658830Z * [new branch] csl/disable_flaky_cpp_test -> origin/csl/disable_flaky_cpp_test 2025-08-14T21:24:12.5664324Z * [new branch] csl/disable_periodic_test -> origin/csl/disable_periodic_test 2025-08-14T21:24:12.5669474Z * [new branch] csl/executorch_docker_fail -> origin/csl/executorch_docker_fail 2025-08-14T21:24:12.5670436Z * [new branch] csl/fix_check_alerts -> origin/csl/fix_check_alerts 2025-08-14T21:24:12.5670932Z * [new branch] csl/katex -> origin/csl/katex 2025-08-14T21:24:12.5671416Z * [new branch] csl/larger_runner -> origin/csl/larger_runner 2025-08-14T21:24:12.5672010Z * [new branch] csl/lintrunner_changed_files_removed -> origin/csl/lintrunner_changed_files_removed 2025-08-14T21:24:12.5672515Z * [new branch] csl/lintrunner_changed_files_removed_test -> origin/csl/lintrunner_changed_files_removed_test 2025-08-14T21:24:12.5674970Z * [new branch] csl/lintrunner_stuff -> origin/csl/lintrunner_stuff 2025-08-14T21:24:12.5675429Z * [new branch] csl/mps_sharding -> origin/csl/mps_sharding 2025-08-14T21:24:12.5675816Z * [new branch] csl/multistage_docker -> origin/csl/multistage_docker 2025-08-14T21:24:12.5676180Z * [new branch] csl/no_keep_goin_rocm -> origin/csl/no_keep_goin_rocm 2025-08-14T21:24:12.5676541Z * [new branch] csl/not_600_timeout -> origin/csl/not_600_timeout 2025-08-14T21:24:12.5676974Z * [new branch] csl/remove_unused_docker_images -> origin/csl/remove_unused_docker_images 2025-08-14T21:24:12.5677353Z * [new branch] csl/revert_open -> origin/csl/revert_open 2025-08-14T21:24:12.5677765Z * [new branch] csl/rocm_upload_artifacts_while_running -> origin/csl/rocm_upload_artifacts_while_running 2025-08-14T21:24:12.5678190Z * [new branch] csl/skip_build -> origin/csl/skip_build 2025-08-14T21:24:12.5678549Z * [new branch] csl/td_dynamo -> origin/csl/td_dynamo 2025-08-14T21:24:12.5678921Z * [new branch] csl/test_cuda_build_large_runner -> origin/csl/test_cuda_build_large_runner 2025-08-14T21:24:12.5679447Z * [new branch] csl/unused_docker -> origin/csl/unused_docker 2025-08-14T21:24:12.5679888Z * [new branch] csl/win_sccache -> origin/csl/win_sccache 2025-08-14T21:24:12.5680216Z * [new branch] cublasltrelax2 -> origin/cublasltrelax2 2025-08-14T21:24:12.5680550Z * [new branch] cublasrelax2 -> origin/cublasrelax2 2025-08-14T21:24:12.5680910Z * [new branch] cudnnsdparefactor -> origin/cudnnsdparefactor 2025-08-14T21:24:12.5681291Z * [new branch] custom_lowering_dict -> origin/custom_lowering_dict 2025-08-14T21:24:12.5681623Z * [new branch] czhuge_muon_dev -> origin/czhuge_muon_dev 2025-08-14T21:24:12.5681936Z * [new branch] d4l3k/delete_hook -> origin/d4l3k/delete_hook 2025-08-14T21:24:12.5682260Z * [new branch] d4l3k/dist_queue -> origin/d4l3k/dist_queue 2025-08-14T21:24:12.5682582Z * [new branch] d4l3k/wait_stream -> origin/d4l3k/wait_stream 2025-08-14T21:24:12.5682946Z * [new branch] dcp-safetensor-test-fix -> origin/dcp-safetensor-test-fix 2025-08-14T21:24:12.5683285Z * [new branch] dcp_zoc -> origin/dcp_zoc 2025-08-14T21:24:12.5683600Z * [new branch] delete-quant-docs -> origin/delete-quant-docs 2025-08-14T21:24:12.5684063Z * [new branch] dependabot/pip/dot-ci/docker/protobuf-5.29.5 -> origin/dependabot/pip/dot-ci/docker/protobuf-5.29.5 2025-08-14T21:24:12.5684541Z * [new branch] desertfire/test_cpp_wrapper -> origin/desertfire/test_cpp_wrapper 2025-08-14T21:24:12.5684982Z * [new branch] desertfire/triton-cpu-for-aarch64 -> origin/desertfire/triton-cpu-for-aarch64 2025-08-14T21:24:12.5685412Z * [new branch] dev/joona/MPSNDArrayAdd -> origin/dev/joona/MPSNDArrayAdd 2025-08-14T21:24:12.5685855Z * [new branch] dev/joona/Unranked -> origin/dev/joona/Unranked 2025-08-14T21:24:12.5686272Z * [new branch] dev/joona/cat -> origin/dev/joona/cat 2025-08-14T21:24:12.5686647Z * [new branch] dev/joona/cat_remove_graph -> origin/dev/joona/cat_remove_graph 2025-08-14T21:24:12.5687044Z * [new branch] dev/joona/embeddingbag -> origin/dev/joona/embeddingbag 2025-08-14T21:24:12.5687431Z * [new branch] dev/joona/getTensorsString -> origin/dev/joona/getTensorsString 2025-08-14T21:24:12.5687898Z * [new branch] dev/joona/maxpool2dwithindices_errmsg -> origin/dev/joona/maxpool2dwithindices_errmsg 2025-08-14T21:24:12.5688367Z * [new branch] dev/joona/mps_linear_macos14 -> origin/dev/joona/mps_linear_macos14 2025-08-14T21:24:12.5688732Z * [new branch] dev/joona/sdpa -> origin/dev/joona/sdpa 2025-08-14T21:24:12.5689131Z * [new branch] dev/joona/synchronize_benchmark -> origin/dev/joona/synchronize_benchmark 2025-08-14T21:24:12.5689541Z * [new branch] dev/joona/topk_newapi -> origin/dev/joona/topk_newapi 2025-08-14T21:24:12.5689901Z * [new branch] dev/joona/type_inf -> origin/dev/joona/type_inf 2025-08-14T21:24:12.5690248Z * [new branch] dev/joona/upsize3d -> origin/dev/joona/upsize3d 2025-08-14T21:24:12.5690580Z * [new branch] disable -> origin/disable 2025-08-14T21:24:12.5691004Z * [new branch] divyanshk-log-api-usage-datapipes-1 -> origin/divyanshk-log-api-usage-datapipes-1 2025-08-14T21:24:12.5691424Z * [new branch] e2e-baseline -> origin/e2e-baseline 2025-08-14T21:24:12.5691773Z * [new branch] embg/test_inductor_ci_128B -> origin/embg/test_inductor_ci_128B 2025-08-14T21:24:12.5692148Z * [new branch] embg/test_inductor_ci_base -> origin/embg/test_inductor_ci_base 2025-08-14T21:24:12.5692583Z * [new branch] embg/test_inductor_ci_control -> origin/embg/test_inductor_ci_control 2025-08-14T21:24:12.5692984Z * [new branch] embg/triton_l2_prefetch_128B -> origin/embg/triton_l2_prefetch_128B 2025-08-14T21:24:12.5693364Z * [new branch] embg/triton_l2_prefetch_256B -> origin/embg/triton_l2_prefetch_256B 2025-08-14T21:24:12.5693734Z * [new branch] enable-b200-benchmark -> origin/enable-b200-benchmark 2025-08-14T21:24:12.5694074Z * [new branch] eqy-patch-1 -> origin/eqy-patch-1 2025-08-14T21:24:12.5694390Z * [new branch] eqy-patch-10 -> origin/eqy-patch-10 2025-08-14T21:24:12.5695720Z * [new branch] eqy-patch-2 -> origin/eqy-patch-2 2025-08-14T21:24:12.5696185Z * [new branch] example-convert-torch.nn -> origin/example-convert-torch.nn 2025-08-14T21:24:12.5701352Z * [new branch] exclamaforte/amd-ma -> origin/exclamaforte/amd-ma 2025-08-14T21:24:12.5703725Z * [new branch] exclamaforte/bump-transformer-version -> origin/exclamaforte/bump-transformer-version 2025-08-14T21:24:12.5704320Z * [new branch] exclamaforte/combo-kernels-perf-run -> origin/exclamaforte/combo-kernels-perf-run 2025-08-14T21:24:12.5704825Z * [new branch] exclamaforte/debug-autotuner-profile -> origin/exclamaforte/debug-autotuner-profile 2025-08-14T21:24:12.5705301Z * [new branch] exclamaforte/do_bench_refactor -> origin/exclamaforte/do_bench_refactor 2025-08-14T21:24:12.5705745Z * [new branch] exclamaforte/enable-mem-dep-fusion -> origin/exclamaforte/enable-mem-dep-fusion 2025-08-14T21:24:12.5706247Z * [new branch] exclamaforte/fix-exhaustive-autotuning -> origin/exclamaforte/fix-exhaustive-autotuning 2025-08-14T21:24:12.5706761Z * [new branch] exclamaforte/fix-trace-parsing-fx-svg -> origin/exclamaforte/fix-trace-parsing-fx-svg 2025-08-14T21:24:12.5707296Z * [new branch] exclamaforte/force-pointwise-cat-perf-run -> origin/exclamaforte/force-pointwise-cat-perf-run 2025-08-14T21:24:12.5707937Z * [new branch] exclamaforte/fusion-data -> origin/exclamaforte/fusion-data 2025-08-14T21:24:12.5708357Z * [new branch] exclamaforte/gemm-benchmark-run -> origin/exclamaforte/gemm-benchmark-run 2025-08-14T21:24:12.5708798Z * [new branch] exclamaforte/gemm-export-model -> origin/exclamaforte/gemm-export-model 2025-08-14T21:24:12.5709207Z * [new branch] exclamaforte/gemm-model -> origin/exclamaforte/gemm-model 2025-08-14T21:24:12.5709676Z * [new branch] exclamaforte/gemm-model-all-data-collection -> origin/exclamaforte/gemm-model-all-data-collection 2025-08-14T21:24:12.5710150Z * [new branch] exclamaforte/gemm-to-amd -> origin/exclamaforte/gemm-to-amd 2025-08-14T21:24:12.5710543Z * [new branch] exclamaforte/just-gemm-model -> origin/exclamaforte/just-gemm-model 2025-08-14T21:24:12.5711014Z * [new branch] exclamaforte/just-gemm-model-no-refactor -> origin/exclamaforte/just-gemm-model-no-refactor 2025-08-14T21:24:12.5711486Z * [new branch] exclamaforte/memory-counter -> origin/exclamaforte/memory-counter 2025-08-14T21:24:12.5711907Z * [new branch] exclamaforte/scheduler-refactor -> origin/exclamaforte/scheduler-refactor 2025-08-14T21:24:12.5712356Z * [new branch] exclamaforte/test_cpp_wrapper_mode -> origin/exclamaforte/test_cpp_wrapper_mode 2025-08-14T21:24:12.5712826Z * [new branch] exclamaforte/update-autotune-configs -> origin/exclamaforte/update-autotune-configs 2025-08-14T21:24:12.5713327Z * [new branch] exclamaforte/update-autotune-configs-2 -> origin/exclamaforte/update-autotune-configs-2 2025-08-14T21:24:12.5713841Z * [new branch] exclamaforte/update-pandas-numpy-ci -> origin/exclamaforte/update-pandas-numpy-ci 2025-08-14T21:24:12.5714337Z * [new branch] exclamforte/gemm-model-final -> origin/exclamforte/gemm-model-final 2025-08-14T21:24:12.5714707Z * [new branch] exec -> origin/exec 2025-08-14T21:24:12.5715033Z * [new branch] experimental-mosaic -> origin/experimental-mosaic 2025-08-14T21:24:12.5715393Z * [new branch] export-D58091437 -> origin/export-D58091437 2025-08-14T21:24:12.5715724Z * [new branch] export-D61047529 -> origin/export-D61047529 2025-08-14T21:24:12.5716049Z * [new branch] export-D68846308 -> origin/export-D68846308 2025-08-14T21:24:12.5716363Z * [new branch] export-D70112642 -> origin/export-D70112642 2025-08-14T21:24:12.5716683Z * [new branch] export-D71412006 -> origin/export-D71412006 2025-08-14T21:24:12.5717004Z * [new branch] export-D72483950 -> origin/export-D72483950 2025-08-14T21:24:12.5717322Z * [new branch] export-D73042989 -> origin/export-D73042989 2025-08-14T21:24:12.5717647Z * [new branch] export-D73287751 -> origin/export-D73287751 2025-08-14T21:24:12.5717967Z * [new branch] export-D75183591 -> origin/export-D75183591 2025-08-14T21:24:12.5718484Z * [new branch] export-D75605373 -> origin/export-D75605373 2025-08-14T21:24:12.5718814Z * [new branch] export-D75617432 -> origin/export-D75617432 2025-08-14T21:24:12.5719360Z * [new branch] export-D75659965 -> origin/export-D75659965 2025-08-14T21:24:12.5721056Z * [new branch] export-D76080931 -> origin/export-D76080931 2025-08-14T21:24:12.5721487Z * [new branch] export-D76463347 -> origin/export-D76463347 2025-08-14T21:24:12.5721833Z * [new branch] export-D76797250 -> origin/export-D76797250 2025-08-14T21:24:12.5722264Z * [new branch] export-D76885271 -> origin/export-D76885271 2025-08-14T21:24:12.5722992Z * [new branch] export-D76885620 -> origin/export-D76885620 2025-08-14T21:24:12.5723779Z * [new branch] export-D76936623 -> origin/export-D76936623 2025-08-14T21:24:12.5724495Z * [new branch] export-D76958268 -> origin/export-D76958268 2025-08-14T21:24:12.5725236Z * [new branch] export-D78047846 -> origin/export-D78047846 2025-08-14T21:24:12.5726100Z * [new branch] export-D78308105 -> origin/export-D78308105 2025-08-14T21:24:12.5726808Z * [new branch] export-D78363609 -> origin/export-D78363609 2025-08-14T21:24:12.5727599Z * [new branch] export-D78375400 -> origin/export-D78375400 2025-08-14T21:24:12.5728241Z * [new branch] export-D78431075 -> origin/export-D78431075 2025-08-14T21:24:12.5728861Z * [new branch] export-D78431305 -> origin/export-D78431305 2025-08-14T21:24:12.5730069Z * [new branch] export-D78458745 -> origin/export-D78458745 2025-08-14T21:24:12.5730614Z * [new branch] export-D78524147 -> origin/export-D78524147 2025-08-14T21:24:12.5731274Z * [new branch] export-D78580107 -> origin/export-D78580107 2025-08-14T21:24:12.5732031Z * [new branch] export-D78588406 -> origin/export-D78588406 2025-08-14T21:24:12.5732833Z * [new branch] export-D78691422 -> origin/export-D78691422 2025-08-14T21:24:12.5733507Z * [new branch] export-D78758466 -> origin/export-D78758466 2025-08-14T21:24:12.5734180Z * [new branch] export-D78822171 -> origin/export-D78822171 2025-08-14T21:24:12.5734965Z * [new branch] export-D78822351 -> origin/export-D78822351 2025-08-14T21:24:12.5735512Z * [new branch] export-D78822507 -> origin/export-D78822507 2025-08-14T21:24:12.5736330Z * [new branch] export-D78826994 -> origin/export-D78826994 2025-08-14T21:24:12.5736862Z * [new branch] export-D78894142 -> origin/export-D78894142 2025-08-14T21:24:12.5737556Z * [new branch] export-D78894324 -> origin/export-D78894324 2025-08-14T21:24:12.5738410Z * [new branch] export-D78907485 -> origin/export-D78907485 2025-08-14T21:24:12.5742588Z * [new branch] export-D78929245 -> origin/export-D78929245 2025-08-14T21:24:12.5742986Z * [new branch] export-D78934925 -> origin/export-D78934925 2025-08-14T21:24:12.5743304Z * [new branch] export-D78953203 -> origin/export-D78953203 2025-08-14T21:24:12.5743615Z * [new branch] export-D78953229 -> origin/export-D78953229 2025-08-14T21:24:12.5743908Z * [new branch] export-D78957093 -> origin/export-D78957093 2025-08-14T21:24:12.5744223Z * [new branch] export-D78957389 -> origin/export-D78957389 2025-08-14T21:24:12.5745135Z * [new branch] export-D78957974 -> origin/export-D78957974 2025-08-14T21:24:12.5745592Z * [new branch] export-D78979812 -> origin/export-D78979812 2025-08-14T21:24:12.5745955Z * [new branch] export-D78996107 -> origin/export-D78996107 2025-08-14T21:24:12.5746309Z * [new branch] export-D79026433 -> origin/export-D79026433 2025-08-14T21:24:12.5746651Z * [new branch] export-D79230339 -> origin/export-D79230339 2025-08-14T21:24:12.5747075Z * [new branch] export-D79319835 -> origin/export-D79319835 2025-08-14T21:24:12.5747697Z * [new branch] export-D79328456 -> origin/export-D79328456 2025-08-14T21:24:12.5749333Z * [new branch] export-D79534608 -> origin/export-D79534608 2025-08-14T21:24:12.5749893Z * [new branch] export-D79647167 -> origin/export-D79647167 2025-08-14T21:24:12.5750460Z * [new branch] export-D79751098 -> origin/export-D79751098 2025-08-14T21:24:12.5752163Z * [new branch] export-D79785974 -> origin/export-D79785974 2025-08-14T21:24:12.5752755Z * [new branch] export-D80025417 -> origin/export-D80025417 2025-08-14T21:24:12.5753243Z * [new branch] export-D80120333 -> origin/export-D80120333 2025-08-14T21:24:12.5753708Z * [new branch] export-D80214882 -> origin/export-D80214882 2025-08-14T21:24:12.5754297Z * [new branch] exported-model-train-idempotent -> origin/exported-model-train-idempotent 2025-08-14T21:24:12.5755447Z * [new branch] ezyang/wip-aot-descriptors -> origin/ezyang/wip-aot-descriptors 2025-08-14T21:24:12.5755915Z * [new branch] fa_u8_brgemm -> origin/fa_u8_brgemm 2025-08-14T21:24:12.5758107Z * [new branch] fastmath_baseline -> origin/fastmath_baseline 2025-08-14T21:24:12.5758646Z * [new branch] fbcode/warm -> origin/fbcode/warm 2025-08-14T21:24:12.5759058Z * [new branch] fca -> origin/fca 2025-08-14T21:24:12.5759441Z * [new branch] fca2_ca5984c -> origin/fca2_ca5984c 2025-08-14T21:24:12.5759826Z * [new branch] fca5 -> origin/fca5 2025-08-14T21:24:12.5761281Z * [new branch] feature/function-numa-binding -> origin/feature/function-numa-binding 2025-08-14T21:24:12.5762007Z * [new branch] fengyuan/external-proj -> origin/fengyuan/external-proj 2025-08-14T21:24:12.5762768Z * [new branch] fengyuan/out-of-tree-xpu-ops-improve-test -> origin/fengyuan/out-of-tree-xpu-ops-improve-test 2025-08-14T21:24:12.5763674Z * [new branch] fengyuan/out-of-tree-xpu-ops-remove-dtype -> origin/fengyuan/out-of-tree-xpu-ops-remove-dtype 2025-08-14T21:24:12.5764619Z * [new branch] fengyuan/test-xpu -> origin/fengyuan/test-xpu 2025-08-14T21:24:12.5765248Z * [new branch] ffast_math_baseline -> origin/ffast_math_baseline 2025-08-14T21:24:12.5766026Z * [new branch] ffast_math_target -> origin/ffast_math_target 2025-08-14T21:24:12.5769590Z * [new branch] findhao/base_commit -> origin/findhao/base_commit 2025-08-14T21:24:12.5770036Z * [new branch] findhao/base_commit1 -> origin/findhao/base_commit1 2025-08-14T21:24:12.5770449Z * [new branch] findhao/fix-indirect-access -> origin/findhao/fix-indirect-access 2025-08-14T21:24:12.5770848Z * [new branch] findhao/multistream2 -> origin/findhao/multistream2 2025-08-14T21:24:12.5771219Z * [new branch] findhao/multistream5 -> origin/findhao/multistream5 2025-08-14T21:24:12.5771776Z * [new branch] findhao/multistream6 -> origin/findhao/multistream6 2025-08-14T21:24:12.5772305Z * [new branch] findhao/operatorbench3 -> origin/findhao/operatorbench3 2025-08-14T21:24:12.5772692Z * [new branch] findhao/operatorbench5 -> origin/findhao/operatorbench5 2025-08-14T21:24:12.5773390Z * [new branch] findhao/tritonparse -> origin/findhao/tritonparse 2025-08-14T21:24:12.5774140Z * [new branch] fix -> origin/fix 2025-08-14T21:24:12.5774961Z * [new branch] fix-ck-gemm-template-format -> origin/fix-ck-gemm-template-format 2025-08-14T21:24:12.5775448Z * [new branch] fix-config-ignore -> origin/fix-config-ignore 2025-08-14T21:24:12.5776059Z * [new branch] fix-dict-guard -> origin/fix-dict-guard 2025-08-14T21:24:12.5779228Z * [new branch] fix-distributed-warning -> origin/fix-distributed-warning 2025-08-14T21:24:12.5779860Z * [new branch] fix-inductor-periodic-0528 -> origin/fix-inductor-periodic-0528 2025-08-14T21:24:12.5780614Z * [new branch] fix-rlease-feature-template -> origin/fix-rlease-feature-template 2025-08-14T21:24:12.5780964Z * [new branch] fix_153389 -> origin/fix_153389 2025-08-14T21:24:12.5781298Z * [new branch] fixes-triage -> origin/fixes-triage 2025-08-14T21:24:12.5781641Z * [new branch] flash_decoding_cpu -> origin/flash_decoding_cpu 2025-08-14T21:24:12.5781958Z * [new branch] flex-flash -> origin/flex-flash 2025-08-14T21:24:12.5782727Z * [new branch] flex-lowering -> origin/flex-lowering 2025-08-14T21:24:12.5783156Z * [new branch] flex-warning -> origin/flex-warning 2025-08-14T21:24:12.5783522Z * [new branch] flex_attention_functorch_grad -> origin/flex_attention_functorch_grad 2025-08-14T21:24:12.5783936Z * [new branch] flex_flash -> origin/flex_flash 2025-08-14T21:24:12.5785534Z * [new branch] fmassa/fix_memeff_sharding_rule -> origin/fmassa/fix_memeff_sharding_rule 2025-08-14T21:24:12.5786165Z * [new branch] fmassa/try_fix_ac_tag_propagation -> origin/fmassa/try_fix_ac_tag_propagation 2025-08-14T21:24:12.5786680Z * [new branch] fsdp2_trace_rules -> origin/fsdp2_trace_rules 2025-08-14T21:24:12.5787128Z * [new branch] fsdpv2_3d -> origin/fsdpv2_3d 2025-08-14T21:24:12.5787643Z * [new branch] fsdpv2_3d_m1 -> origin/fsdpv2_3d_m1 2025-08-14T21:24:12.5789314Z * [new branch] fx_cpp -> origin/fx_cpp 2025-08-14T21:24:12.5789843Z * [new branch] fy/fix-win -> origin/fy/fix-win 2025-08-14T21:24:12.5792296Z * [new branch] gh/AlnisM/1/base -> origin/gh/AlnisM/1/base 2025-08-14T21:24:12.5793047Z * [new branch] gh/AlnisM/1/head -> origin/gh/AlnisM/1/head 2025-08-14T21:24:12.5793545Z * [new branch] gh/CaoE/2/base -> origin/gh/CaoE/2/base 2025-08-14T21:24:12.5793879Z * [new branch] gh/CaoE/2/head -> origin/gh/CaoE/2/head 2025-08-14T21:24:12.5794671Z * [new branch] gh/CaoE/2/orig -> origin/gh/CaoE/2/orig 2025-08-14T21:24:12.5799716Z * [new branch] gh/ColinPeppler/72/base -> origin/gh/ColinPeppler/72/base 2025-08-14T21:24:12.5800170Z * [new branch] gh/ColinPeppler/72/head -> origin/gh/ColinPeppler/72/head 2025-08-14T21:24:12.5800568Z * [new branch] gh/ColinPeppler/72/orig -> origin/gh/ColinPeppler/72/orig 2025-08-14T21:24:12.5800953Z * [new branch] gh/ColinPeppler/77/base -> origin/gh/ColinPeppler/77/base 2025-08-14T21:24:12.5801361Z * [new branch] gh/ColinPeppler/77/head -> origin/gh/ColinPeppler/77/head 2025-08-14T21:24:12.5801764Z * [new branch] gh/ColinPeppler/77/orig -> origin/gh/ColinPeppler/77/orig 2025-08-14T21:24:12.5802176Z * [new branch] gh/ColinPeppler/78/base -> origin/gh/ColinPeppler/78/base 2025-08-14T21:24:12.5802790Z * [new branch] gh/ColinPeppler/78/head -> origin/gh/ColinPeppler/78/head 2025-08-14T21:24:12.5803475Z * [new branch] gh/ColinPeppler/78/orig -> origin/gh/ColinPeppler/78/orig 2025-08-14T21:24:12.5803866Z * [new branch] gh/EikanWang/67/base -> origin/gh/EikanWang/67/base 2025-08-14T21:24:12.5804577Z * [new branch] gh/EikanWang/67/head -> origin/gh/EikanWang/67/head 2025-08-14T21:24:12.5806579Z * [new branch] gh/EikanWang/80/base -> origin/gh/EikanWang/80/base 2025-08-14T21:24:12.5806958Z * [new branch] gh/EikanWang/80/head -> origin/gh/EikanWang/80/head 2025-08-14T21:24:12.5807331Z * [new branch] gh/EikanWang/80/orig -> origin/gh/EikanWang/80/orig 2025-08-14T21:24:12.5807932Z * [new branch] gh/EikanWang/81/base -> origin/gh/EikanWang/81/base 2025-08-14T21:24:12.5809945Z * [new branch] gh/EikanWang/81/head -> origin/gh/EikanWang/81/head 2025-08-14T21:24:12.5810409Z * [new branch] gh/EikanWang/81/orig -> origin/gh/EikanWang/81/orig 2025-08-14T21:24:12.5811210Z * [new branch] gh/Gasoonjia/1/base -> origin/gh/Gasoonjia/1/base 2025-08-14T21:24:12.5811846Z * [new branch] gh/Gasoonjia/1/head -> origin/gh/Gasoonjia/1/head 2025-08-14T21:24:12.5816424Z * [new branch] gh/H-Huang/131/base -> origin/gh/H-Huang/131/base 2025-08-14T21:24:12.5816952Z * [new branch] gh/H-Huang/131/head -> origin/gh/H-Huang/131/head 2025-08-14T21:24:12.5823527Z * [new branch] gh/H-Huang/131/orig -> origin/gh/H-Huang/131/orig 2025-08-14T21:24:12.5828779Z * [new branch] gh/H-Huang/132/base -> origin/gh/H-Huang/132/base 2025-08-14T21:24:12.5833191Z * [new branch] gh/H-Huang/132/head -> origin/gh/H-Huang/132/head 2025-08-14T21:24:12.5837470Z * [new branch] gh/H-Huang/132/orig -> origin/gh/H-Huang/132/orig 2025-08-14T21:24:12.5839752Z * [new branch] gh/H-Huang/180/base -> origin/gh/H-Huang/180/base 2025-08-14T21:24:12.5840094Z * [new branch] gh/H-Huang/180/head -> origin/gh/H-Huang/180/head 2025-08-14T21:24:12.5840424Z * [new branch] gh/H-Huang/180/orig -> origin/gh/H-Huang/180/orig 2025-08-14T21:24:12.5840737Z * [new branch] gh/H-Huang/182/base -> origin/gh/H-Huang/182/base 2025-08-14T21:24:12.5841058Z * [new branch] gh/H-Huang/182/head -> origin/gh/H-Huang/182/head 2025-08-14T21:24:12.5841375Z * [new branch] gh/H-Huang/182/orig -> origin/gh/H-Huang/182/orig 2025-08-14T21:24:12.5841923Z * [new branch] gh/H-Huang/183/base -> origin/gh/H-Huang/183/base 2025-08-14T21:24:12.5842289Z * [new branch] gh/H-Huang/183/head -> origin/gh/H-Huang/183/head 2025-08-14T21:24:12.5842630Z * [new branch] gh/H-Huang/183/orig -> origin/gh/H-Huang/183/orig 2025-08-14T21:24:12.5842975Z * [new branch] gh/H-Huang/187/base -> origin/gh/H-Huang/187/base 2025-08-14T21:24:12.5843310Z * [new branch] gh/H-Huang/187/head -> origin/gh/H-Huang/187/head 2025-08-14T21:24:12.5843651Z * [new branch] gh/H-Huang/187/orig -> origin/gh/H-Huang/187/orig 2025-08-14T21:24:12.5843982Z * [new branch] gh/H-Huang/192/base -> origin/gh/H-Huang/192/base 2025-08-14T21:24:12.5844303Z * [new branch] gh/H-Huang/192/head -> origin/gh/H-Huang/192/head 2025-08-14T21:24:12.5844634Z * [new branch] gh/H-Huang/192/orig -> origin/gh/H-Huang/192/orig 2025-08-14T21:24:12.5844966Z * [new branch] gh/H-Huang/195/base -> origin/gh/H-Huang/195/base 2025-08-14T21:24:12.5845294Z * [new branch] gh/H-Huang/195/head -> origin/gh/H-Huang/195/head 2025-08-14T21:24:12.5845743Z * [new branch] gh/H-Huang/195/orig -> origin/gh/H-Huang/195/orig 2025-08-14T21:24:12.5846084Z * [new branch] gh/H-Huang/196/base -> origin/gh/H-Huang/196/base 2025-08-14T21:24:12.5846422Z * [new branch] gh/H-Huang/196/head -> origin/gh/H-Huang/196/head 2025-08-14T21:24:12.5846757Z * [new branch] gh/H-Huang/196/orig -> origin/gh/H-Huang/196/orig 2025-08-14T21:24:12.5847084Z * [new branch] gh/H-Huang/197/base -> origin/gh/H-Huang/197/base 2025-08-14T21:24:12.5847416Z * [new branch] gh/H-Huang/197/head -> origin/gh/H-Huang/197/head 2025-08-14T21:24:12.5847725Z * [new branch] gh/H-Huang/197/orig -> origin/gh/H-Huang/197/orig 2025-08-14T21:24:12.5848136Z * [new branch] gh/H-Huang/198/base -> origin/gh/H-Huang/198/base 2025-08-14T21:24:12.5848468Z * [new branch] gh/H-Huang/198/head -> origin/gh/H-Huang/198/head 2025-08-14T21:24:12.5848801Z * [new branch] gh/H-Huang/198/orig -> origin/gh/H-Huang/198/orig 2025-08-14T21:24:12.5849135Z * [new branch] gh/H-Huang/199/base -> origin/gh/H-Huang/199/base 2025-08-14T21:24:12.5849458Z * [new branch] gh/H-Huang/199/head -> origin/gh/H-Huang/199/head 2025-08-14T21:24:12.5849793Z * [new branch] gh/H-Huang/199/orig -> origin/gh/H-Huang/199/orig 2025-08-14T21:24:12.5850128Z * [new branch] gh/H-Huang/200/base -> origin/gh/H-Huang/200/base 2025-08-14T21:24:12.5850451Z * [new branch] gh/H-Huang/200/head -> origin/gh/H-Huang/200/head 2025-08-14T21:24:12.5850787Z * [new branch] gh/H-Huang/200/orig -> origin/gh/H-Huang/200/orig 2025-08-14T21:24:12.5851126Z * [new branch] gh/H-Huang/201/base -> origin/gh/H-Huang/201/base 2025-08-14T21:24:12.5851471Z * [new branch] gh/H-Huang/201/head -> origin/gh/H-Huang/201/head 2025-08-14T21:24:12.5851859Z * [new branch] gh/H-Huang/201/orig -> origin/gh/H-Huang/201/orig 2025-08-14T21:24:12.5852212Z * [new branch] gh/H-Huang/202/base -> origin/gh/H-Huang/202/base 2025-08-14T21:24:12.5852552Z * [new branch] gh/H-Huang/202/head -> origin/gh/H-Huang/202/head 2025-08-14T21:24:12.5852891Z * [new branch] gh/H-Huang/202/orig -> origin/gh/H-Huang/202/orig 2025-08-14T21:24:12.5853224Z * [new branch] gh/H-Huang/203/base -> origin/gh/H-Huang/203/base 2025-08-14T21:24:12.5853567Z * [new branch] gh/H-Huang/203/head -> origin/gh/H-Huang/203/head 2025-08-14T21:24:12.5854055Z * [new branch] gh/H-Huang/203/orig -> origin/gh/H-Huang/203/orig 2025-08-14T21:24:12.5854396Z * [new branch] gh/H-Huang/204/base -> origin/gh/H-Huang/204/base 2025-08-14T21:24:12.5854728Z * [new branch] gh/H-Huang/204/head -> origin/gh/H-Huang/204/head 2025-08-14T21:24:12.5855062Z * [new branch] gh/H-Huang/204/orig -> origin/gh/H-Huang/204/orig 2025-08-14T21:24:12.5855408Z * [new branch] gh/H-Huang/205/base -> origin/gh/H-Huang/205/base 2025-08-14T21:24:12.5855752Z * [new branch] gh/H-Huang/205/head -> origin/gh/H-Huang/205/head 2025-08-14T21:24:12.5856092Z * [new branch] gh/H-Huang/205/orig -> origin/gh/H-Huang/205/orig 2025-08-14T21:24:12.5856429Z * [new branch] gh/H-Huang/206/base -> origin/gh/H-Huang/206/base 2025-08-14T21:24:12.5856762Z * [new branch] gh/H-Huang/206/head -> origin/gh/H-Huang/206/head 2025-08-14T21:24:12.5857111Z * [new branch] gh/H-Huang/206/orig -> origin/gh/H-Huang/206/orig 2025-08-14T21:24:12.5857795Z * [new branch] gh/H-Huang/207/base -> origin/gh/H-Huang/207/base 2025-08-14T21:24:12.5858158Z * [new branch] gh/H-Huang/207/head -> origin/gh/H-Huang/207/head 2025-08-14T21:24:12.5858506Z * [new branch] gh/H-Huang/207/orig -> origin/gh/H-Huang/207/orig 2025-08-14T21:24:12.5858848Z * [new branch] gh/H-Huang/208/base -> origin/gh/H-Huang/208/base 2025-08-14T21:24:12.5859315Z * [new branch] gh/H-Huang/208/head -> origin/gh/H-Huang/208/head 2025-08-14T21:24:12.5859802Z * [new branch] gh/H-Huang/208/orig -> origin/gh/H-Huang/208/orig 2025-08-14T21:24:12.5860361Z * [new branch] gh/H-Huang/209/base -> origin/gh/H-Huang/209/base 2025-08-14T21:24:12.5862754Z * [new branch] gh/H-Huang/209/head -> origin/gh/H-Huang/209/head 2025-08-14T21:24:12.5863343Z * [new branch] gh/H-Huang/209/orig -> origin/gh/H-Huang/209/orig 2025-08-14T21:24:12.5868477Z * [new branch] gh/IvanKobzarev/107/base -> origin/gh/IvanKobzarev/107/base 2025-08-14T21:24:12.5869083Z * [new branch] gh/IvanKobzarev/107/head -> origin/gh/IvanKobzarev/107/head 2025-08-14T21:24:12.5869595Z * [new branch] gh/IvanKobzarev/107/orig -> origin/gh/IvanKobzarev/107/orig 2025-08-14T21:24:12.5870417Z * [new branch] gh/IvanKobzarev/110/base -> origin/gh/IvanKobzarev/110/base 2025-08-14T21:24:12.5870859Z * [new branch] gh/IvanKobzarev/110/head -> origin/gh/IvanKobzarev/110/head 2025-08-14T21:24:12.5871230Z * [new branch] gh/IvanKobzarev/110/orig -> origin/gh/IvanKobzarev/110/orig 2025-08-14T21:24:12.5871607Z * [new branch] gh/IvanKobzarev/111/base -> origin/gh/IvanKobzarev/111/base 2025-08-14T21:24:12.5871987Z * [new branch] gh/IvanKobzarev/111/head -> origin/gh/IvanKobzarev/111/head 2025-08-14T21:24:12.5872350Z * [new branch] gh/IvanKobzarev/111/orig -> origin/gh/IvanKobzarev/111/orig 2025-08-14T21:24:12.5872714Z * [new branch] gh/IvanKobzarev/112/base -> origin/gh/IvanKobzarev/112/base 2025-08-14T21:24:12.5873071Z * [new branch] gh/IvanKobzarev/112/head -> origin/gh/IvanKobzarev/112/head 2025-08-14T21:24:12.5877534Z * [new branch] gh/IvanKobzarev/112/orig -> origin/gh/IvanKobzarev/112/orig 2025-08-14T21:24:12.5878115Z * [new branch] gh/IvanKobzarev/115/base -> origin/gh/IvanKobzarev/115/base 2025-08-14T21:24:12.5878632Z * [new branch] gh/IvanKobzarev/115/head -> origin/gh/IvanKobzarev/115/head 2025-08-14T21:24:12.5879171Z * [new branch] gh/IvanKobzarev/115/orig -> origin/gh/IvanKobzarev/115/orig 2025-08-14T21:24:12.5879558Z * [new branch] gh/IvanKobzarev/116/base -> origin/gh/IvanKobzarev/116/base 2025-08-14T21:24:12.5880126Z * [new branch] gh/IvanKobzarev/116/head -> origin/gh/IvanKobzarev/116/head 2025-08-14T21:24:12.5880525Z * [new branch] gh/IvanKobzarev/116/orig -> origin/gh/IvanKobzarev/116/orig 2025-08-14T21:24:12.5880915Z * [new branch] gh/IvanKobzarev/118/base -> origin/gh/IvanKobzarev/118/base 2025-08-14T21:24:12.5881300Z * [new branch] gh/IvanKobzarev/118/head -> origin/gh/IvanKobzarev/118/head 2025-08-14T21:24:12.5881686Z * [new branch] gh/IvanKobzarev/118/orig -> origin/gh/IvanKobzarev/118/orig 2025-08-14T21:24:12.5882074Z * [new branch] gh/IvanKobzarev/124/base -> origin/gh/IvanKobzarev/124/base 2025-08-14T21:24:12.5882463Z * [new branch] gh/IvanKobzarev/124/head -> origin/gh/IvanKobzarev/124/head 2025-08-14T21:24:12.5882855Z * [new branch] gh/IvanKobzarev/124/orig -> origin/gh/IvanKobzarev/124/orig 2025-08-14T21:24:12.5884444Z * [new branch] gh/IvanKobzarev/126/base -> origin/gh/IvanKobzarev/126/base 2025-08-14T21:24:12.5884868Z * [new branch] gh/IvanKobzarev/126/head -> origin/gh/IvanKobzarev/126/head 2025-08-14T21:24:12.5885269Z * [new branch] gh/IvanKobzarev/126/orig -> origin/gh/IvanKobzarev/126/orig 2025-08-14T21:24:12.5885735Z * [new branch] gh/IvanKobzarev/127/base -> origin/gh/IvanKobzarev/127/base 2025-08-14T21:24:12.5886137Z * [new branch] gh/IvanKobzarev/127/head -> origin/gh/IvanKobzarev/127/head 2025-08-14T21:24:12.5886565Z * [new branch] gh/IvanKobzarev/127/orig -> origin/gh/IvanKobzarev/127/orig 2025-08-14T21:24:12.5886963Z * [new branch] gh/IvanKobzarev/128/base -> origin/gh/IvanKobzarev/128/base 2025-08-14T21:24:12.5891520Z * [new branch] gh/IvanKobzarev/128/head -> origin/gh/IvanKobzarev/128/head 2025-08-14T21:24:12.5891986Z * [new branch] gh/IvanKobzarev/128/orig -> origin/gh/IvanKobzarev/128/orig 2025-08-14T21:24:12.5892566Z * [new branch] gh/IvanKobzarev/129/base -> origin/gh/IvanKobzarev/129/base 2025-08-14T21:24:12.5892970Z * [new branch] gh/IvanKobzarev/129/head -> origin/gh/IvanKobzarev/129/head 2025-08-14T21:24:12.5893378Z * [new branch] gh/IvanKobzarev/129/orig -> origin/gh/IvanKobzarev/129/orig 2025-08-14T21:24:12.5893960Z * [new branch] gh/IvanKobzarev/130/base -> origin/gh/IvanKobzarev/130/base 2025-08-14T21:24:12.5894381Z * [new branch] gh/IvanKobzarev/130/head -> origin/gh/IvanKobzarev/130/head 2025-08-14T21:24:12.5894785Z * [new branch] gh/IvanKobzarev/130/orig -> origin/gh/IvanKobzarev/130/orig 2025-08-14T21:24:12.5895175Z * [new branch] gh/IvanKobzarev/131/base -> origin/gh/IvanKobzarev/131/base 2025-08-14T21:24:12.5895593Z * [new branch] gh/IvanKobzarev/131/head -> origin/gh/IvanKobzarev/131/head 2025-08-14T21:24:12.5896042Z * [new branch] gh/IvanKobzarev/131/orig -> origin/gh/IvanKobzarev/131/orig 2025-08-14T21:24:12.5901137Z * [new branch] gh/IvanKobzarev/132/base -> origin/gh/IvanKobzarev/132/base 2025-08-14T21:24:12.5901767Z * [new branch] gh/IvanKobzarev/132/head -> origin/gh/IvanKobzarev/132/head 2025-08-14T21:24:12.5902273Z * [new branch] gh/IvanKobzarev/132/orig -> origin/gh/IvanKobzarev/132/orig 2025-08-14T21:24:12.5903149Z * [new branch] gh/IvanKobzarev/133/base -> origin/gh/IvanKobzarev/133/base 2025-08-14T21:24:12.5903670Z * [new branch] gh/IvanKobzarev/133/head -> origin/gh/IvanKobzarev/133/head 2025-08-14T21:24:12.5904043Z * [new branch] gh/IvanKobzarev/133/orig -> origin/gh/IvanKobzarev/133/orig 2025-08-14T21:24:12.5904576Z * [new branch] gh/IvanKobzarev/134/base -> origin/gh/IvanKobzarev/134/base 2025-08-14T21:24:12.5905266Z * [new branch] gh/IvanKobzarev/134/head -> origin/gh/IvanKobzarev/134/head 2025-08-14T21:24:12.5905803Z * [new branch] gh/IvanKobzarev/134/orig -> origin/gh/IvanKobzarev/134/orig 2025-08-14T21:24:12.5906307Z * [new branch] gh/IvanKobzarev/135/base -> origin/gh/IvanKobzarev/135/base 2025-08-14T21:24:12.5906676Z * [new branch] gh/IvanKobzarev/135/head -> origin/gh/IvanKobzarev/135/head 2025-08-14T21:24:12.5907045Z * [new branch] gh/IvanKobzarev/135/orig -> origin/gh/IvanKobzarev/135/orig 2025-08-14T21:24:12.5911916Z * [new branch] gh/NikhilAPatel/1/base -> origin/gh/NikhilAPatel/1/base 2025-08-14T21:24:12.5912371Z * [new branch] gh/NikhilAPatel/1/head -> origin/gh/NikhilAPatel/1/head 2025-08-14T21:24:12.5912748Z * [new branch] gh/NikhilAPatel/16/base -> origin/gh/NikhilAPatel/16/base 2025-08-14T21:24:12.5913127Z * [new branch] gh/NikhilAPatel/16/head -> origin/gh/NikhilAPatel/16/head 2025-08-14T21:24:12.5913511Z * [new branch] gh/NikhilAPatel/16/orig -> origin/gh/NikhilAPatel/16/orig 2025-08-14T21:24:12.5913878Z * [new branch] gh/NikhilAPatel/18/base -> origin/gh/NikhilAPatel/18/base 2025-08-14T21:24:12.5914232Z * [new branch] gh/NikhilAPatel/18/head -> origin/gh/NikhilAPatel/18/head 2025-08-14T21:24:12.5914591Z * [new branch] gh/NikhilAPatel/18/orig -> origin/gh/NikhilAPatel/18/orig 2025-08-14T21:24:12.5914950Z * [new branch] gh/NikhilAPatel/19/base -> origin/gh/NikhilAPatel/19/base 2025-08-14T21:24:12.5915312Z * [new branch] gh/NikhilAPatel/19/head -> origin/gh/NikhilAPatel/19/head 2025-08-14T21:24:12.5915670Z * [new branch] gh/NikhilAPatel/19/orig -> origin/gh/NikhilAPatel/19/orig 2025-08-14T21:24:12.5916054Z * [new branch] gh/NikhilAPatel/2/base -> origin/gh/NikhilAPatel/2/base 2025-08-14T21:24:12.5916409Z * [new branch] gh/NikhilAPatel/2/head -> origin/gh/NikhilAPatel/2/head 2025-08-14T21:24:12.5916900Z * [new branch] gh/NikhilAPatel/4/base -> origin/gh/NikhilAPatel/4/base 2025-08-14T21:24:12.5917236Z * [new branch] gh/NikhilAPatel/4/head -> origin/gh/NikhilAPatel/4/head 2025-08-14T21:24:12.5917576Z * [new branch] gh/NikhilAPatel/8/base -> origin/gh/NikhilAPatel/8/base 2025-08-14T21:24:12.5917917Z * [new branch] gh/NikhilAPatel/8/head -> origin/gh/NikhilAPatel/8/head 2025-08-14T21:24:12.5919678Z * [new branch] gh/NikhilAPatel/8/orig -> origin/gh/NikhilAPatel/8/orig 2025-08-14T21:24:12.5920038Z * [new branch] gh/NikhilAPatel/9/base -> origin/gh/NikhilAPatel/9/base 2025-08-14T21:24:12.5920382Z * [new branch] gh/NikhilAPatel/9/head -> origin/gh/NikhilAPatel/9/head 2025-08-14T21:24:12.5920723Z * [new branch] gh/NikhilAPatel/9/orig -> origin/gh/NikhilAPatel/9/orig 2025-08-14T21:24:12.5921092Z * [new branch] gh/PaliC/1/base -> origin/gh/PaliC/1/base 2025-08-14T21:24:12.5921427Z * [new branch] gh/PaliC/1/head -> origin/gh/PaliC/1/head 2025-08-14T21:24:12.5921749Z * [new branch] gh/PaliC/1/orig -> origin/gh/PaliC/1/orig 2025-08-14T21:24:12.5922100Z * [new branch] gh/PaliC/12/base -> origin/gh/PaliC/12/base 2025-08-14T21:24:12.5922935Z * [new branch] gh/PaliC/12/head -> origin/gh/PaliC/12/head 2025-08-14T21:24:12.5923715Z * [new branch] gh/PaliC/12/orig -> origin/gh/PaliC/12/orig 2025-08-14T21:24:12.5928328Z * [new branch] gh/PaliC/13/base -> origin/gh/PaliC/13/base 2025-08-14T21:24:12.5928717Z * [new branch] gh/PaliC/13/head -> origin/gh/PaliC/13/head 2025-08-14T21:24:12.5929034Z * [new branch] gh/PaliC/13/orig -> origin/gh/PaliC/13/orig 2025-08-14T21:24:12.5929483Z * [new branch] gh/PaliC/14/base -> origin/gh/PaliC/14/base 2025-08-14T21:24:12.5929810Z * [new branch] gh/PaliC/14/head -> origin/gh/PaliC/14/head 2025-08-14T21:24:12.5930113Z * [new branch] gh/PaliC/14/orig -> origin/gh/PaliC/14/orig 2025-08-14T21:24:12.5930424Z * [new branch] gh/PaliC/15/base -> origin/gh/PaliC/15/base 2025-08-14T21:24:12.5930721Z * [new branch] gh/PaliC/15/head -> origin/gh/PaliC/15/head 2025-08-14T21:24:12.5931070Z * [new branch] gh/PaliC/15/orig -> origin/gh/PaliC/15/orig 2025-08-14T21:24:12.5932393Z * [new branch] gh/PaliC/16/base -> origin/gh/PaliC/16/base 2025-08-14T21:24:12.5932739Z * [new branch] gh/PaliC/16/head -> origin/gh/PaliC/16/head 2025-08-14T21:24:12.5933117Z * [new branch] gh/PaliC/16/orig -> origin/gh/PaliC/16/orig 2025-08-14T21:24:12.5938478Z * [new branch] gh/PaliC/17/base -> origin/gh/PaliC/17/base 2025-08-14T21:24:12.5938887Z * [new branch] gh/PaliC/17/head -> origin/gh/PaliC/17/head 2025-08-14T21:24:12.5939230Z * [new branch] gh/PaliC/17/orig -> origin/gh/PaliC/17/orig 2025-08-14T21:24:12.5939566Z * [new branch] gh/PaliC/18/base -> origin/gh/PaliC/18/base 2025-08-14T21:24:12.5939893Z * [new branch] gh/PaliC/18/head -> origin/gh/PaliC/18/head 2025-08-14T21:24:12.5940222Z * [new branch] gh/PaliC/18/orig -> origin/gh/PaliC/18/orig 2025-08-14T21:24:12.5940537Z * [new branch] gh/PaliC/19/base -> origin/gh/PaliC/19/base 2025-08-14T21:24:12.5941282Z * [new branch] gh/PaliC/19/head -> origin/gh/PaliC/19/head 2025-08-14T21:24:12.5946956Z * [new branch] gh/PaliC/19/orig -> origin/gh/PaliC/19/orig 2025-08-14T21:24:12.5949323Z * [new branch] gh/PaliC/2/base -> origin/gh/PaliC/2/base 2025-08-14T21:24:12.5955293Z * [new branch] gh/PaliC/2/head -> origin/gh/PaliC/2/head 2025-08-14T21:24:12.5955719Z * [new branch] gh/PaliC/2/orig -> origin/gh/PaliC/2/orig 2025-08-14T21:24:12.5956077Z * [new branch] gh/PaliC/20/base -> origin/gh/PaliC/20/base 2025-08-14T21:24:12.5956414Z * [new branch] gh/PaliC/20/head -> origin/gh/PaliC/20/head 2025-08-14T21:24:12.5956741Z * [new branch] gh/PaliC/20/orig -> origin/gh/PaliC/20/orig 2025-08-14T21:24:12.5957066Z * [new branch] gh/PaliC/21/base -> origin/gh/PaliC/21/base 2025-08-14T21:24:12.5957378Z * [new branch] gh/PaliC/21/head -> origin/gh/PaliC/21/head 2025-08-14T21:24:12.5957699Z * [new branch] gh/PaliC/21/orig -> origin/gh/PaliC/21/orig 2025-08-14T21:24:12.5958031Z * [new branch] gh/PaliC/22/base -> origin/gh/PaliC/22/base 2025-08-14T21:24:12.5958360Z * [new branch] gh/PaliC/22/head -> origin/gh/PaliC/22/head 2025-08-14T21:24:12.5958672Z * [new branch] gh/PaliC/22/orig -> origin/gh/PaliC/22/orig 2025-08-14T21:24:12.5958992Z * [new branch] gh/PaliC/23/base -> origin/gh/PaliC/23/base 2025-08-14T21:24:12.5959310Z * [new branch] gh/PaliC/23/head -> origin/gh/PaliC/23/head 2025-08-14T21:24:12.5959627Z * [new branch] gh/PaliC/23/orig -> origin/gh/PaliC/23/orig 2025-08-14T21:24:12.5959939Z * [new branch] gh/PaliC/24/base -> origin/gh/PaliC/24/base 2025-08-14T21:24:12.5960257Z * [new branch] gh/PaliC/24/head -> origin/gh/PaliC/24/head 2025-08-14T21:24:12.5960574Z * [new branch] gh/PaliC/24/orig -> origin/gh/PaliC/24/orig 2025-08-14T21:24:12.5961173Z * [new branch] gh/PaulZhang12/17/base -> origin/gh/PaulZhang12/17/base 2025-08-14T21:24:12.5961564Z * [new branch] gh/PaulZhang12/17/head -> origin/gh/PaulZhang12/17/head 2025-08-14T21:24:12.5961945Z * [new branch] gh/PaulZhang12/18/base -> origin/gh/PaulZhang12/18/base 2025-08-14T21:24:12.5962305Z * [new branch] gh/PaulZhang12/18/head -> origin/gh/PaulZhang12/18/head 2025-08-14T21:24:12.5962657Z * [new branch] gh/PaulZhang12/18/orig -> origin/gh/PaulZhang12/18/orig 2025-08-14T21:24:12.5963014Z * [new branch] gh/PaulZhang12/19/base -> origin/gh/PaulZhang12/19/base 2025-08-14T21:24:12.5963373Z * [new branch] gh/PaulZhang12/19/head -> origin/gh/PaulZhang12/19/head 2025-08-14T21:24:12.5963726Z * [new branch] gh/PaulZhang12/19/orig -> origin/gh/PaulZhang12/19/orig 2025-08-14T21:24:12.5964073Z * [new branch] gh/PaulZhang12/20/base -> origin/gh/PaulZhang12/20/base 2025-08-14T21:24:12.5964425Z * [new branch] gh/PaulZhang12/20/head -> origin/gh/PaulZhang12/20/head 2025-08-14T21:24:12.5964782Z * [new branch] gh/PaulZhang12/20/orig -> origin/gh/PaulZhang12/20/orig 2025-08-14T21:24:12.5965146Z * [new branch] gh/PaulZhang12/21/base -> origin/gh/PaulZhang12/21/base 2025-08-14T21:24:12.5965501Z * [new branch] gh/PaulZhang12/21/head -> origin/gh/PaulZhang12/21/head 2025-08-14T21:24:12.5966065Z * [new branch] gh/PaulZhang12/21/orig -> origin/gh/PaulZhang12/21/orig 2025-08-14T21:24:12.5966436Z * [new branch] gh/PaulZhang12/22/base -> origin/gh/PaulZhang12/22/base 2025-08-14T21:24:12.5966812Z * [new branch] gh/PaulZhang12/22/head -> origin/gh/PaulZhang12/22/head 2025-08-14T21:24:12.5967163Z * [new branch] gh/PaulZhang12/22/orig -> origin/gh/PaulZhang12/22/orig 2025-08-14T21:24:12.5967530Z * [new branch] gh/SamGinzburg/11/base -> origin/gh/SamGinzburg/11/base 2025-08-14T21:24:12.5967945Z * [new branch] gh/SamGinzburg/11/head -> origin/gh/SamGinzburg/11/head 2025-08-14T21:24:12.5972545Z * [new branch] gh/Sidharth123-cpu/24/base -> origin/gh/Sidharth123-cpu/24/base 2025-08-14T21:24:12.5972990Z * [new branch] gh/Sidharth123-cpu/25/base -> origin/gh/Sidharth123-cpu/25/base 2025-08-14T21:24:12.5973365Z * [new branch] gh/Sidharth123-cpu/26/base -> origin/gh/Sidharth123-cpu/26/base 2025-08-14T21:24:12.5973724Z * [new branch] gh/Sidharth123-cpu/27/base -> origin/gh/Sidharth123-cpu/27/base 2025-08-14T21:24:12.5974083Z * [new branch] gh/Sidharth123-cpu/42/base -> origin/gh/Sidharth123-cpu/42/base 2025-08-14T21:24:12.5974433Z * [new branch] gh/Sidharth123-cpu/42/head -> origin/gh/Sidharth123-cpu/42/head 2025-08-14T21:24:12.5979196Z * [new branch] gh/Sidharth123-cpu/42/orig -> origin/gh/Sidharth123-cpu/42/orig 2025-08-14T21:24:12.5984674Z * [new branch] gh/Sidharth123-cpu/43/base -> origin/gh/Sidharth123-cpu/43/base 2025-08-14T21:24:12.5988865Z * [new branch] gh/Sidharth123-cpu/43/head -> origin/gh/Sidharth123-cpu/43/head 2025-08-14T21:24:12.5994973Z * [new branch] gh/Sidharth123-cpu/43/orig -> origin/gh/Sidharth123-cpu/43/orig 2025-08-14T21:24:12.5995427Z * [new branch] gh/Sidharth123-cpu/44/base -> origin/gh/Sidharth123-cpu/44/base 2025-08-14T21:24:12.5995808Z * [new branch] gh/Sidharth123-cpu/44/head -> origin/gh/Sidharth123-cpu/44/head 2025-08-14T21:24:12.5996187Z * [new branch] gh/Sidharth123-cpu/44/orig -> origin/gh/Sidharth123-cpu/44/orig 2025-08-14T21:24:12.5996560Z * [new branch] gh/Sidharth123-cpu/45/base -> origin/gh/Sidharth123-cpu/45/base 2025-08-14T21:24:12.5996925Z * [new branch] gh/Sidharth123-cpu/45/head -> origin/gh/Sidharth123-cpu/45/head 2025-08-14T21:24:12.5997448Z * [new branch] gh/Sidharth123-cpu/45/orig -> origin/gh/Sidharth123-cpu/45/orig 2025-08-14T21:24:12.5997852Z * [new branch] gh/StrongerXi/1/base -> origin/gh/StrongerXi/1/base 2025-08-14T21:24:12.5998211Z * [new branch] gh/StrongerXi/1/head -> origin/gh/StrongerXi/1/head 2025-08-14T21:24:12.5998576Z * [new branch] gh/StrongerXi/103/base -> origin/gh/StrongerXi/103/base 2025-08-14T21:24:12.5998939Z * [new branch] gh/StrongerXi/103/head -> origin/gh/StrongerXi/103/head 2025-08-14T21:24:12.5999296Z * [new branch] gh/StrongerXi/103/orig -> origin/gh/StrongerXi/103/orig 2025-08-14T21:24:12.5999649Z * [new branch] gh/StrongerXi/133/base -> origin/gh/StrongerXi/133/base 2025-08-14T21:24:12.6000007Z * [new branch] gh/StrongerXi/133/head -> origin/gh/StrongerXi/133/head 2025-08-14T21:24:12.6000366Z * [new branch] gh/StrongerXi/133/orig -> origin/gh/StrongerXi/133/orig 2025-08-14T21:24:12.6000720Z * [new branch] gh/StrongerXi/134/base -> origin/gh/StrongerXi/134/base 2025-08-14T21:24:12.6001068Z * [new branch] gh/StrongerXi/134/head -> origin/gh/StrongerXi/134/head 2025-08-14T21:24:12.6001420Z * [new branch] gh/StrongerXi/134/orig -> origin/gh/StrongerXi/134/orig 2025-08-14T21:24:12.6001783Z * [new branch] gh/StrongerXi/135/base -> origin/gh/StrongerXi/135/base 2025-08-14T21:24:12.6002143Z * [new branch] gh/StrongerXi/135/head -> origin/gh/StrongerXi/135/head 2025-08-14T21:24:12.6002495Z * [new branch] gh/StrongerXi/135/orig -> origin/gh/StrongerXi/135/orig 2025-08-14T21:24:12.6002856Z * [new branch] gh/StrongerXi/136/base -> origin/gh/StrongerXi/136/base 2025-08-14T21:24:12.6003216Z * [new branch] gh/StrongerXi/136/head -> origin/gh/StrongerXi/136/head 2025-08-14T21:24:12.6003567Z * [new branch] gh/StrongerXi/136/orig -> origin/gh/StrongerXi/136/orig 2025-08-14T21:24:12.6003980Z * [new branch] gh/StrongerXi/137/base -> origin/gh/StrongerXi/137/base 2025-08-14T21:24:12.6004328Z * [new branch] gh/StrongerXi/137/head -> origin/gh/StrongerXi/137/head 2025-08-14T21:24:12.6004691Z * [new branch] gh/StrongerXi/137/orig -> origin/gh/StrongerXi/137/orig 2025-08-14T21:24:12.6005043Z * [new branch] gh/StrongerXi/138/base -> origin/gh/StrongerXi/138/base 2025-08-14T21:24:12.6005403Z * [new branch] gh/StrongerXi/138/head -> origin/gh/StrongerXi/138/head 2025-08-14T21:24:12.6005933Z * [new branch] gh/StrongerXi/138/orig -> origin/gh/StrongerXi/138/orig 2025-08-14T21:24:12.6006303Z * [new branch] gh/StrongerXi/71/base -> origin/gh/StrongerXi/71/base 2025-08-14T21:24:12.6006651Z * [new branch] gh/StrongerXi/71/head -> origin/gh/StrongerXi/71/head 2025-08-14T21:24:12.6007012Z * [new branch] gh/StrongerXi/72/base -> origin/gh/StrongerXi/72/base 2025-08-14T21:24:12.6007347Z * [new branch] gh/StrongerXi/72/head -> origin/gh/StrongerXi/72/head 2025-08-14T21:24:12.6007682Z * [new branch] gh/XilunWu/131/base -> origin/gh/XilunWu/131/base 2025-08-14T21:24:12.6008011Z * [new branch] gh/XilunWu/131/head -> origin/gh/XilunWu/131/head 2025-08-14T21:24:12.6008332Z * [new branch] gh/XilunWu/131/orig -> origin/gh/XilunWu/131/orig 2025-08-14T21:24:12.6008649Z * [new branch] gh/XilunWu/133/base -> origin/gh/XilunWu/133/base 2025-08-14T21:24:12.6008964Z * [new branch] gh/XilunWu/133/head -> origin/gh/XilunWu/133/head 2025-08-14T21:24:12.6009273Z * [new branch] gh/XilunWu/133/orig -> origin/gh/XilunWu/133/orig 2025-08-14T21:24:12.6009607Z * [new branch] gh/XilunWu/136/base -> origin/gh/XilunWu/136/base 2025-08-14T21:24:12.6009983Z * [new branch] gh/XilunWu/136/head -> origin/gh/XilunWu/136/head 2025-08-14T21:24:12.6010662Z * [new branch] gh/XilunWu/136/orig -> origin/gh/XilunWu/136/orig 2025-08-14T21:24:12.6011017Z * [new branch] gh/XilunWu/139/base -> origin/gh/XilunWu/139/base 2025-08-14T21:24:12.6011669Z * [new branch] gh/XilunWu/139/head -> origin/gh/XilunWu/139/head 2025-08-14T21:24:12.6012417Z * [new branch] gh/XilunWu/139/orig -> origin/gh/XilunWu/139/orig 2025-08-14T21:24:12.6013518Z * [new branch] gh/XilunWu/143/base -> origin/gh/XilunWu/143/base 2025-08-14T21:24:12.6014072Z * [new branch] gh/XilunWu/143/head -> origin/gh/XilunWu/143/head 2025-08-14T21:24:12.6014739Z * [new branch] gh/XilunWu/143/orig -> origin/gh/XilunWu/143/orig 2025-08-14T21:24:12.6016519Z * [new branch] gh/XilunWu/144/base -> origin/gh/XilunWu/144/base 2025-08-14T21:24:12.6017127Z * [new branch] gh/XilunWu/144/head -> origin/gh/XilunWu/144/head 2025-08-14T21:24:12.6017623Z * [new branch] gh/XilunWu/144/orig -> origin/gh/XilunWu/144/orig 2025-08-14T21:24:12.6018155Z * [new branch] gh/XilunWu/145/base -> origin/gh/XilunWu/145/base 2025-08-14T21:24:12.6018770Z * [new branch] gh/XilunWu/145/head -> origin/gh/XilunWu/145/head 2025-08-14T21:24:12.6019409Z * [new branch] gh/XilunWu/145/orig -> origin/gh/XilunWu/145/orig 2025-08-14T21:24:12.6023954Z * [new branch] gh/XilunWu/146/base -> origin/gh/XilunWu/146/base 2025-08-14T21:24:12.6024378Z * [new branch] gh/XilunWu/146/head -> origin/gh/XilunWu/146/head 2025-08-14T21:24:12.6024718Z * [new branch] gh/XilunWu/146/orig -> origin/gh/XilunWu/146/orig 2025-08-14T21:24:12.6025064Z * [new branch] gh/XilunWu/147/base -> origin/gh/XilunWu/147/base 2025-08-14T21:24:12.6025422Z * [new branch] gh/XilunWu/147/head -> origin/gh/XilunWu/147/head 2025-08-14T21:24:12.6025918Z * [new branch] gh/XilunWu/147/orig -> origin/gh/XilunWu/147/orig 2025-08-14T21:24:12.6026251Z * [new branch] gh/XilunWu/148/base -> origin/gh/XilunWu/148/base 2025-08-14T21:24:12.6026588Z * [new branch] gh/XilunWu/148/head -> origin/gh/XilunWu/148/head 2025-08-14T21:24:12.6026918Z * [new branch] gh/XilunWu/148/orig -> origin/gh/XilunWu/148/orig 2025-08-14T21:24:12.6027253Z * [new branch] gh/XilunWu/149/base -> origin/gh/XilunWu/149/base 2025-08-14T21:24:12.6027613Z * [new branch] gh/XilunWu/149/head -> origin/gh/XilunWu/149/head 2025-08-14T21:24:12.6028406Z * [new branch] gh/XilunWu/149/orig -> origin/gh/XilunWu/149/orig 2025-08-14T21:24:12.6029215Z * [new branch] gh/XilunWu/150/base -> origin/gh/XilunWu/150/base 2025-08-14T21:24:12.6029872Z * [new branch] gh/XilunWu/150/head -> origin/gh/XilunWu/150/head 2025-08-14T21:24:12.6030535Z * [new branch] gh/XilunWu/150/orig -> origin/gh/XilunWu/150/orig 2025-08-14T21:24:12.6032694Z * [new branch] gh/XilunWu/151/base -> origin/gh/XilunWu/151/base 2025-08-14T21:24:12.6033064Z * [new branch] gh/XilunWu/151/head -> origin/gh/XilunWu/151/head 2025-08-14T21:24:12.6033412Z * [new branch] gh/XilunWu/151/orig -> origin/gh/XilunWu/151/orig 2025-08-14T21:24:12.6033794Z * [new branch] gh/XilunWu/152/base -> origin/gh/XilunWu/152/base 2025-08-14T21:24:12.6034517Z * [new branch] gh/XilunWu/152/head -> origin/gh/XilunWu/152/head 2025-08-14T21:24:12.6035193Z * [new branch] gh/XilunWu/152/orig -> origin/gh/XilunWu/152/orig 2025-08-14T21:24:12.6037197Z * [new branch] gh/XilunWu/153/base -> origin/gh/XilunWu/153/base 2025-08-14T21:24:12.6037810Z * [new branch] gh/XilunWu/153/head -> origin/gh/XilunWu/153/head 2025-08-14T21:24:12.6038172Z * [new branch] gh/XilunWu/153/orig -> origin/gh/XilunWu/153/orig 2025-08-14T21:24:12.6041128Z * [new branch] gh/XilunWu/154/base -> origin/gh/XilunWu/154/base 2025-08-14T21:24:12.6041510Z * [new branch] gh/XilunWu/154/head -> origin/gh/XilunWu/154/head 2025-08-14T21:24:12.6041900Z * [new branch] gh/XilunWu/154/orig -> origin/gh/XilunWu/154/orig 2025-08-14T21:24:12.6042258Z * [new branch] gh/XilunWu/156/base -> origin/gh/XilunWu/156/base 2025-08-14T21:24:12.6042654Z * [new branch] gh/XilunWu/156/head -> origin/gh/XilunWu/156/head 2025-08-14T21:24:12.6047604Z * [new branch] gh/XilunWu/156/orig -> origin/gh/XilunWu/156/orig 2025-08-14T21:24:12.6048130Z * [new branch] gh/XilunWu/157/base -> origin/gh/XilunWu/157/base 2025-08-14T21:24:12.6048596Z * [new branch] gh/XilunWu/157/head -> origin/gh/XilunWu/157/head 2025-08-14T21:24:12.6049066Z * [new branch] gh/XilunWu/157/orig -> origin/gh/XilunWu/157/orig 2025-08-14T21:24:12.6049926Z * [new branch] gh/XilunWu/158/base -> origin/gh/XilunWu/158/base 2025-08-14T21:24:12.6050339Z * [new branch] gh/XilunWu/158/head -> origin/gh/XilunWu/158/head 2025-08-14T21:24:12.6050693Z * [new branch] gh/XilunWu/158/orig -> origin/gh/XilunWu/158/orig 2025-08-14T21:24:12.6055766Z * [new branch] gh/XilunWu/159/base -> origin/gh/XilunWu/159/base 2025-08-14T21:24:12.6056176Z * [new branch] gh/XilunWu/159/head -> origin/gh/XilunWu/159/head 2025-08-14T21:24:12.6056552Z * [new branch] gh/XilunWu/159/orig -> origin/gh/XilunWu/159/orig 2025-08-14T21:24:12.6056934Z * [new branch] gh/XilunWu/160/base -> origin/gh/XilunWu/160/base 2025-08-14T21:24:12.6057532Z * [new branch] gh/XilunWu/160/head -> origin/gh/XilunWu/160/head 2025-08-14T21:24:12.6057886Z * [new branch] gh/XilunWu/160/orig -> origin/gh/XilunWu/160/orig 2025-08-14T21:24:12.6058227Z * [new branch] gh/XilunWu/161/base -> origin/gh/XilunWu/161/base 2025-08-14T21:24:12.6058584Z * [new branch] gh/XilunWu/161/head -> origin/gh/XilunWu/161/head 2025-08-14T21:24:12.6058926Z * [new branch] gh/XilunWu/161/orig -> origin/gh/XilunWu/161/orig 2025-08-14T21:24:12.6059269Z * [new branch] gh/XilunWu/162/base -> origin/gh/XilunWu/162/base 2025-08-14T21:24:12.6059581Z * [new branch] gh/XilunWu/162/head -> origin/gh/XilunWu/162/head 2025-08-14T21:24:12.6059885Z * [new branch] gh/XilunWu/162/orig -> origin/gh/XilunWu/162/orig 2025-08-14T21:24:12.6060207Z * [new branch] gh/XilunWu/163/base -> origin/gh/XilunWu/163/base 2025-08-14T21:24:12.6064943Z * [new branch] gh/XilunWu/163/head -> origin/gh/XilunWu/163/head 2025-08-14T21:24:12.6065342Z * [new branch] gh/XilunWu/163/orig -> origin/gh/XilunWu/163/orig 2025-08-14T21:24:12.6065689Z * [new branch] gh/XuehaiPan/14/base -> origin/gh/XuehaiPan/14/base 2025-08-14T21:24:12.6066032Z * [new branch] gh/XuehaiPan/14/head -> origin/gh/XuehaiPan/14/head 2025-08-14T21:24:12.6066359Z * [new branch] gh/XuehaiPan/14/orig -> origin/gh/XuehaiPan/14/orig 2025-08-14T21:24:12.6066696Z * [new branch] gh/XuehaiPan/179/base -> origin/gh/XuehaiPan/179/base 2025-08-14T21:24:12.6067502Z * [new branch] gh/XuehaiPan/179/head -> origin/gh/XuehaiPan/179/head 2025-08-14T21:24:12.6067981Z * [new branch] gh/XuehaiPan/179/orig -> origin/gh/XuehaiPan/179/orig 2025-08-14T21:24:12.6068630Z * [new branch] gh/XuehaiPan/189/base -> origin/gh/XuehaiPan/189/base 2025-08-14T21:24:12.6068985Z * [new branch] gh/XuehaiPan/189/head -> origin/gh/XuehaiPan/189/head 2025-08-14T21:24:12.6069353Z * [new branch] gh/XuehaiPan/189/orig -> origin/gh/XuehaiPan/189/orig 2025-08-14T21:24:12.6069683Z * [new branch] gh/XuehaiPan/227/base -> origin/gh/XuehaiPan/227/base 2025-08-14T21:24:12.6070053Z * [new branch] gh/XuehaiPan/227/head -> origin/gh/XuehaiPan/227/head 2025-08-14T21:24:12.6070435Z * [new branch] gh/XuehaiPan/227/orig -> origin/gh/XuehaiPan/227/orig 2025-08-14T21:24:12.6070977Z * [new branch] gh/XuehaiPan/231/base -> origin/gh/XuehaiPan/231/base 2025-08-14T21:24:12.6071472Z * [new branch] gh/XuehaiPan/231/head -> origin/gh/XuehaiPan/231/head 2025-08-14T21:24:12.6072649Z * [new branch] gh/XuehaiPan/231/orig -> origin/gh/XuehaiPan/231/orig 2025-08-14T21:24:12.6073158Z * [new branch] gh/XuehaiPan/232/base -> origin/gh/XuehaiPan/232/base 2025-08-14T21:24:12.6073608Z * [new branch] gh/XuehaiPan/232/head -> origin/gh/XuehaiPan/232/head 2025-08-14T21:24:12.6074057Z * [new branch] gh/XuehaiPan/232/orig -> origin/gh/XuehaiPan/232/orig 2025-08-14T21:24:12.6074513Z * [new branch] gh/XuehaiPan/249/base -> origin/gh/XuehaiPan/249/base 2025-08-14T21:24:12.6075009Z * [new branch] gh/XuehaiPan/249/head -> origin/gh/XuehaiPan/249/head 2025-08-14T21:24:12.6079893Z * [new branch] gh/XuehaiPan/249/orig -> origin/gh/XuehaiPan/249/orig 2025-08-14T21:24:12.6080440Z * [new branch] gh/XuehaiPan/253/base -> origin/gh/XuehaiPan/253/base 2025-08-14T21:24:12.6080900Z * [new branch] gh/XuehaiPan/253/head -> origin/gh/XuehaiPan/253/head 2025-08-14T21:24:12.6081739Z * [new branch] gh/XuehaiPan/253/orig -> origin/gh/XuehaiPan/253/orig 2025-08-14T21:24:12.6082353Z * [new branch] gh/XuehaiPan/254/base -> origin/gh/XuehaiPan/254/base 2025-08-14T21:24:12.6082688Z * [new branch] gh/XuehaiPan/254/head -> origin/gh/XuehaiPan/254/head 2025-08-14T21:24:12.6083024Z * [new branch] gh/XuehaiPan/254/orig -> origin/gh/XuehaiPan/254/orig 2025-08-14T21:24:12.6083360Z * [new branch] gh/XuehaiPan/255/base -> origin/gh/XuehaiPan/255/base 2025-08-14T21:24:12.6083693Z * [new branch] gh/XuehaiPan/255/head -> origin/gh/XuehaiPan/255/head 2025-08-14T21:24:12.6084018Z * [new branch] gh/XuehaiPan/255/orig -> origin/gh/XuehaiPan/255/orig 2025-08-14T21:24:12.6084374Z * [new branch] gh/XuehaiPan/257/base -> origin/gh/XuehaiPan/257/base 2025-08-14T21:24:12.6084735Z * [new branch] gh/XuehaiPan/257/head -> origin/gh/XuehaiPan/257/head 2025-08-14T21:24:12.6085101Z * [new branch] gh/XuehaiPan/257/orig -> origin/gh/XuehaiPan/257/orig 2025-08-14T21:24:12.6085471Z * [new branch] gh/XuehaiPan/271/base -> origin/gh/XuehaiPan/271/base 2025-08-14T21:24:12.6086080Z * [new branch] gh/XuehaiPan/271/head -> origin/gh/XuehaiPan/271/head 2025-08-14T21:24:12.6086455Z * [new branch] gh/XuehaiPan/271/orig -> origin/gh/XuehaiPan/271/orig 2025-08-14T21:24:12.6086820Z * [new branch] gh/XuehaiPan/283/base -> origin/gh/XuehaiPan/283/base 2025-08-14T21:24:12.6087197Z * [new branch] gh/XuehaiPan/283/head -> origin/gh/XuehaiPan/283/head 2025-08-14T21:24:12.6087547Z * [new branch] gh/XuehaiPan/283/orig -> origin/gh/XuehaiPan/283/orig 2025-08-14T21:24:12.6087925Z * [new branch] gh/XuehaiPan/290/base -> origin/gh/XuehaiPan/290/base 2025-08-14T21:24:12.6088511Z * [new branch] gh/XuehaiPan/290/head -> origin/gh/XuehaiPan/290/head 2025-08-14T21:24:12.6089106Z * [new branch] gh/XuehaiPan/290/orig -> origin/gh/XuehaiPan/290/orig 2025-08-14T21:24:12.6090044Z * [new branch] gh/XuehaiPan/328/base -> origin/gh/XuehaiPan/328/base 2025-08-14T21:24:12.6090617Z * [new branch] gh/XuehaiPan/328/head -> origin/gh/XuehaiPan/328/head 2025-08-14T21:24:12.6091279Z * [new branch] gh/XuehaiPan/328/orig -> origin/gh/XuehaiPan/328/orig 2025-08-14T21:24:12.6092630Z * [new branch] gh/XuehaiPan/339/base -> origin/gh/XuehaiPan/339/base 2025-08-14T21:24:12.6092992Z * [new branch] gh/XuehaiPan/339/head -> origin/gh/XuehaiPan/339/head 2025-08-14T21:24:12.6093587Z * [new branch] gh/XuehaiPan/339/orig -> origin/gh/XuehaiPan/339/orig 2025-08-14T21:24:12.6095068Z * [new branch] gh/XuehaiPan/343/base -> origin/gh/XuehaiPan/343/base 2025-08-14T21:24:12.6095426Z * [new branch] gh/XuehaiPan/343/head -> origin/gh/XuehaiPan/343/head 2025-08-14T21:24:12.6095777Z * [new branch] gh/XuehaiPan/343/orig -> origin/gh/XuehaiPan/343/orig 2025-08-14T21:24:12.6097169Z * [new branch] gh/XuehaiPan/344/base -> origin/gh/XuehaiPan/344/base 2025-08-14T21:24:12.6097650Z * [new branch] gh/XuehaiPan/344/head -> origin/gh/XuehaiPan/344/head 2025-08-14T21:24:12.6098317Z * [new branch] gh/XuehaiPan/344/orig -> origin/gh/XuehaiPan/344/orig 2025-08-14T21:24:12.6099357Z * [new branch] gh/XuehaiPan/345/base -> origin/gh/XuehaiPan/345/base 2025-08-14T21:24:12.6099854Z * [new branch] gh/XuehaiPan/345/head -> origin/gh/XuehaiPan/345/head 2025-08-14T21:24:12.6100560Z * [new branch] gh/XuehaiPan/345/orig -> origin/gh/XuehaiPan/345/orig 2025-08-14T21:24:12.6101591Z * [new branch] gh/XuehaiPan/346/base -> origin/gh/XuehaiPan/346/base 2025-08-14T21:24:12.6102361Z * [new branch] gh/XuehaiPan/346/head -> origin/gh/XuehaiPan/346/head 2025-08-14T21:24:12.6103051Z * [new branch] gh/XuehaiPan/346/orig -> origin/gh/XuehaiPan/346/orig 2025-08-14T21:24:12.6104201Z * [new branch] gh/XuehaiPan/347/base -> origin/gh/XuehaiPan/347/base 2025-08-14T21:24:12.6106535Z * [new branch] gh/XuehaiPan/347/head -> origin/gh/XuehaiPan/347/head 2025-08-14T21:24:12.6106901Z * [new branch] gh/XuehaiPan/347/orig -> origin/gh/XuehaiPan/347/orig 2025-08-14T21:24:12.6107326Z * [new branch] gh/XuehaiPan/348/base -> origin/gh/XuehaiPan/348/base 2025-08-14T21:24:12.6107689Z * [new branch] gh/XuehaiPan/348/head -> origin/gh/XuehaiPan/348/head 2025-08-14T21:24:12.6108170Z * [new branch] gh/XuehaiPan/348/orig -> origin/gh/XuehaiPan/348/orig 2025-08-14T21:24:12.6108562Z * [new branch] gh/XuehaiPan/350/base -> origin/gh/XuehaiPan/350/base 2025-08-14T21:24:12.6109396Z * [new branch] gh/XuehaiPan/350/head -> origin/gh/XuehaiPan/350/head 2025-08-14T21:24:12.6110519Z * [new branch] gh/XuehaiPan/350/orig -> origin/gh/XuehaiPan/350/orig 2025-08-14T21:24:12.6111271Z * [new branch] gh/XuehaiPan/352/base -> origin/gh/XuehaiPan/352/base 2025-08-14T21:24:12.6111877Z * [new branch] gh/XuehaiPan/352/head -> origin/gh/XuehaiPan/352/head 2025-08-14T21:24:12.6112261Z * [new branch] gh/XuehaiPan/352/orig -> origin/gh/XuehaiPan/352/orig 2025-08-14T21:24:12.6113545Z * [new branch] gh/XuehaiPan/356/base -> origin/gh/XuehaiPan/356/base 2025-08-14T21:24:12.6113919Z * [new branch] gh/XuehaiPan/356/head -> origin/gh/XuehaiPan/356/head 2025-08-14T21:24:12.6114997Z * [new branch] gh/XuehaiPan/356/orig -> origin/gh/XuehaiPan/356/orig 2025-08-14T21:24:12.6115999Z * [new branch] gh/XuehaiPan/357/base -> origin/gh/XuehaiPan/357/base 2025-08-14T21:24:12.6116715Z * [new branch] gh/XuehaiPan/357/head -> origin/gh/XuehaiPan/357/head 2025-08-14T21:24:12.6117240Z * [new branch] gh/XuehaiPan/357/orig -> origin/gh/XuehaiPan/357/orig 2025-08-14T21:24:12.6118175Z * [new branch] gh/XuehaiPan/358/base -> origin/gh/XuehaiPan/358/base 2025-08-14T21:24:12.6118781Z * [new branch] gh/XuehaiPan/358/head -> origin/gh/XuehaiPan/358/head 2025-08-14T21:24:12.6119536Z * [new branch] gh/XuehaiPan/358/orig -> origin/gh/XuehaiPan/358/orig 2025-08-14T21:24:12.6120596Z * [new branch] gh/XuehaiPan/359/base -> origin/gh/XuehaiPan/359/base 2025-08-14T21:24:12.6121256Z * [new branch] gh/XuehaiPan/359/head -> origin/gh/XuehaiPan/359/head 2025-08-14T21:24:12.6121831Z * [new branch] gh/XuehaiPan/359/orig -> origin/gh/XuehaiPan/359/orig 2025-08-14T21:24:12.6122815Z * [new branch] gh/XuehaiPan/360/base -> origin/gh/XuehaiPan/360/base 2025-08-14T21:24:12.6124222Z * [new branch] gh/XuehaiPan/360/head -> origin/gh/XuehaiPan/360/head 2025-08-14T21:24:12.6124593Z * [new branch] gh/XuehaiPan/360/orig -> origin/gh/XuehaiPan/360/orig 2025-08-14T21:24:12.6126011Z * [new branch] gh/XuehaiPan/365/base -> origin/gh/XuehaiPan/365/base 2025-08-14T21:24:12.6126607Z * [new branch] gh/XuehaiPan/365/head -> origin/gh/XuehaiPan/365/head 2025-08-14T21:24:12.6127343Z * [new branch] gh/XuehaiPan/365/orig -> origin/gh/XuehaiPan/365/orig 2025-08-14T21:24:12.6128434Z * [new branch] gh/XuehaiPan/366/base -> origin/gh/XuehaiPan/366/base 2025-08-14T21:24:12.6128848Z * [new branch] gh/XuehaiPan/366/head -> origin/gh/XuehaiPan/366/head 2025-08-14T21:24:12.6129925Z * [new branch] gh/XuehaiPan/368/base -> origin/gh/XuehaiPan/368/base 2025-08-14T21:24:12.6130460Z * [new branch] gh/XuehaiPan/368/head -> origin/gh/XuehaiPan/368/head 2025-08-14T21:24:12.6130985Z * [new branch] gh/XuehaiPan/368/orig -> origin/gh/XuehaiPan/368/orig 2025-08-14T21:24:12.6132704Z * [new branch] gh/XuehaiPan/369/base -> origin/gh/XuehaiPan/369/base 2025-08-14T21:24:12.6133161Z * [new branch] gh/XuehaiPan/369/head -> origin/gh/XuehaiPan/369/head 2025-08-14T21:24:12.6133570Z * [new branch] gh/XuehaiPan/369/orig -> origin/gh/XuehaiPan/369/orig 2025-08-14T21:24:12.6134659Z * [new branch] gh/XuehaiPan/370/base -> origin/gh/XuehaiPan/370/base 2025-08-14T21:24:12.6135160Z * [new branch] gh/XuehaiPan/370/head -> origin/gh/XuehaiPan/370/head 2025-08-14T21:24:12.6135876Z * [new branch] gh/XuehaiPan/370/orig -> origin/gh/XuehaiPan/370/orig 2025-08-14T21:24:12.6136834Z * [new branch] gh/XuehaiPan/371/base -> origin/gh/XuehaiPan/371/base 2025-08-14T21:24:12.6137360Z * [new branch] gh/XuehaiPan/371/head -> origin/gh/XuehaiPan/371/head 2025-08-14T21:24:12.6138053Z * [new branch] gh/XuehaiPan/371/orig -> origin/gh/XuehaiPan/371/orig 2025-08-14T21:24:12.6141395Z * [new branch] gh/XuehaiPan/372/base -> origin/gh/XuehaiPan/372/base 2025-08-14T21:24:12.6141849Z * [new branch] gh/XuehaiPan/372/head -> origin/gh/XuehaiPan/372/head 2025-08-14T21:24:12.6142498Z * [new branch] gh/XuehaiPan/372/orig -> origin/gh/XuehaiPan/372/orig 2025-08-14T21:24:12.6144243Z * [new branch] gh/XuehaiPan/373/base -> origin/gh/XuehaiPan/373/base 2025-08-14T21:24:12.6144675Z * [new branch] gh/XuehaiPan/373/head -> origin/gh/XuehaiPan/373/head 2025-08-14T21:24:12.6145038Z * [new branch] gh/XuehaiPan/373/orig -> origin/gh/XuehaiPan/373/orig 2025-08-14T21:24:12.6146003Z * [new branch] gh/XuehaiPan/374/base -> origin/gh/XuehaiPan/374/base 2025-08-14T21:24:12.6146414Z * [new branch] gh/XuehaiPan/374/head -> origin/gh/XuehaiPan/374/head 2025-08-14T21:24:12.6147088Z * [new branch] gh/XuehaiPan/374/orig -> origin/gh/XuehaiPan/374/orig 2025-08-14T21:24:12.6148068Z * [new branch] gh/XuehaiPan/375/base -> origin/gh/XuehaiPan/375/base 2025-08-14T21:24:12.6148586Z * [new branch] gh/XuehaiPan/375/head -> origin/gh/XuehaiPan/375/head 2025-08-14T21:24:12.6149227Z * [new branch] gh/XuehaiPan/375/orig -> origin/gh/XuehaiPan/375/orig 2025-08-14T21:24:12.6150336Z * [new branch] gh/XuehaiPan/376/base -> origin/gh/XuehaiPan/376/base 2025-08-14T21:24:12.6150821Z * [new branch] gh/XuehaiPan/376/head -> origin/gh/XuehaiPan/376/head 2025-08-14T21:24:12.6151363Z * [new branch] gh/XuehaiPan/376/orig -> origin/gh/XuehaiPan/376/orig 2025-08-14T21:24:12.6155282Z * [new branch] gh/XuehaiPan/377/base -> origin/gh/XuehaiPan/377/base 2025-08-14T21:24:12.6155691Z * [new branch] gh/XuehaiPan/377/head -> origin/gh/XuehaiPan/377/head 2025-08-14T21:24:12.6156039Z * [new branch] gh/XuehaiPan/377/orig -> origin/gh/XuehaiPan/377/orig 2025-08-14T21:24:12.6156380Z * [new branch] gh/XuehaiPan/378/base -> origin/gh/XuehaiPan/378/base 2025-08-14T21:24:12.6156713Z * [new branch] gh/XuehaiPan/378/head -> origin/gh/XuehaiPan/378/head 2025-08-14T21:24:12.6157581Z * [new branch] gh/XuehaiPan/378/orig -> origin/gh/XuehaiPan/378/orig 2025-08-14T21:24:12.6158062Z * [new branch] gh/XuehaiPan/379/base -> origin/gh/XuehaiPan/379/base 2025-08-14T21:24:12.6158463Z * [new branch] gh/XuehaiPan/379/head -> origin/gh/XuehaiPan/379/head 2025-08-14T21:24:12.6158882Z * [new branch] gh/XuehaiPan/379/orig -> origin/gh/XuehaiPan/379/orig 2025-08-14T21:24:12.6159678Z * [new branch] gh/ZhiweiYan-96/39/base -> origin/gh/ZhiweiYan-96/39/base 2025-08-14T21:24:12.6160181Z * [new branch] gh/ZhiweiYan-96/39/head -> origin/gh/ZhiweiYan-96/39/head 2025-08-14T21:24:12.6160661Z * [new branch] gh/ZhiweiYan-96/39/orig -> origin/gh/ZhiweiYan-96/39/orig 2025-08-14T21:24:12.6161427Z * [new branch] gh/ZhiweiYan-96/44/base -> origin/gh/ZhiweiYan-96/44/base 2025-08-14T21:24:12.6162171Z * [new branch] gh/ZhiweiYan-96/44/head -> origin/gh/ZhiweiYan-96/44/head 2025-08-14T21:24:12.6163342Z * [new branch] gh/ZhiweiYan-96/45/base -> origin/gh/ZhiweiYan-96/45/base 2025-08-14T21:24:12.6163923Z * [new branch] gh/ZhiweiYan-96/45/head -> origin/gh/ZhiweiYan-96/45/head 2025-08-14T21:24:12.6166386Z * [new branch] gh/ZhiweiYan-96/49/base -> origin/gh/ZhiweiYan-96/49/base 2025-08-14T21:24:12.6166861Z * [new branch] gh/ZhiweiYan-96/49/head -> origin/gh/ZhiweiYan-96/49/head 2025-08-14T21:24:12.6167239Z * [new branch] gh/ZhiweiYan-96/62/base -> origin/gh/ZhiweiYan-96/62/base 2025-08-14T21:24:12.6167753Z * [new branch] gh/ZhiweiYan-96/62/head -> origin/gh/ZhiweiYan-96/62/head 2025-08-14T21:24:12.6168229Z * [new branch] gh/ZhiweiYan-96/64/base -> origin/gh/ZhiweiYan-96/64/base 2025-08-14T21:24:12.6168907Z * [new branch] gh/ZhiweiYan-96/64/head -> origin/gh/ZhiweiYan-96/64/head 2025-08-14T21:24:12.6170596Z * [new branch] gh/ZhiweiYan-96/64/orig -> origin/gh/ZhiweiYan-96/64/orig 2025-08-14T21:24:12.6171187Z * [new branch] gh/ZhiweiYan-96/65/base -> origin/gh/ZhiweiYan-96/65/base 2025-08-14T21:24:12.6171695Z * [new branch] gh/ZhiweiYan-96/65/head -> origin/gh/ZhiweiYan-96/65/head 2025-08-14T21:24:12.6172447Z * [new branch] gh/ZhiweiYan-96/65/orig -> origin/gh/ZhiweiYan-96/65/orig 2025-08-14T21:24:12.6173121Z * [new branch] gh/ZhiweiYan-96/66/base -> origin/gh/ZhiweiYan-96/66/base 2025-08-14T21:24:12.6173746Z * [new branch] gh/ZhiweiYan-96/66/head -> origin/gh/ZhiweiYan-96/66/head 2025-08-14T21:24:12.6177336Z * [new branch] gh/ZhiweiYan-96/67/base -> origin/gh/ZhiweiYan-96/67/base 2025-08-14T21:24:12.6177905Z * [new branch] gh/ZhiweiYan-96/67/head -> origin/gh/ZhiweiYan-96/67/head 2025-08-14T21:24:12.6178389Z * [new branch] gh/ZhiweiYan-96/68/base -> origin/gh/ZhiweiYan-96/68/base 2025-08-14T21:24:12.6178735Z * [new branch] gh/ZhiweiYan-96/68/head -> origin/gh/ZhiweiYan-96/68/head 2025-08-14T21:24:12.6179073Z * [new branch] gh/ZhiweiYan-96/68/orig -> origin/gh/ZhiweiYan-96/68/orig 2025-08-14T21:24:12.6179415Z * [new branch] gh/aakhundov/1/base -> origin/gh/aakhundov/1/base 2025-08-14T21:24:12.6179890Z * [new branch] gh/aakhundov/1/head -> origin/gh/aakhundov/1/head 2025-08-14T21:24:12.6180355Z * [new branch] gh/aakhundov/2/base -> origin/gh/aakhundov/2/base 2025-08-14T21:24:12.6180996Z * [new branch] gh/aakhundov/2/head -> origin/gh/aakhundov/2/head 2025-08-14T21:24:12.6182118Z * [new branch] gh/aditew01/openblas -> origin/gh/aditew01/openblas 2025-08-14T21:24:12.6182545Z * [new branch] gh/aditew01/sbgemm -> origin/gh/aditew01/sbgemm 2025-08-14T21:24:12.6183179Z * [new branch] gh/aditew01/vecbf16 -> origin/gh/aditew01/vecbf16 2025-08-14T21:24:12.6185203Z * [new branch] gh/alexbrauckmann/paddedtensor_faketensor_init -> origin/gh/alexbrauckmann/paddedtensor_faketensor_init 2025-08-14T21:24:12.6185945Z * [new branch] gh/alexbrauckmann/paddedtensor_init -> origin/gh/alexbrauckmann/paddedtensor_init 2025-08-14T21:24:12.6186596Z * [new branch] gh/alexbrauckmann/paddedtensor_meta_init -> origin/gh/alexbrauckmann/paddedtensor_meta_init 2025-08-14T21:24:12.6187487Z * [new branch] gh/alexsamardzic/7/base -> origin/gh/alexsamardzic/7/base 2025-08-14T21:24:12.6187858Z * [new branch] gh/alexsamardzic/7/head -> origin/gh/alexsamardzic/7/head 2025-08-14T21:24:12.6188246Z * [new branch] gh/alexsamardzic/7/orig -> origin/gh/alexsamardzic/7/orig 2025-08-14T21:24:12.6192000Z * [new branch] gh/alexsamardzic/8/base -> origin/gh/alexsamardzic/8/base 2025-08-14T21:24:12.6192464Z * [new branch] gh/alexsamardzic/8/head -> origin/gh/alexsamardzic/8/head 2025-08-14T21:24:12.6192855Z * [new branch] gh/alexsamardzic/8/orig -> origin/gh/alexsamardzic/8/orig 2025-08-14T21:24:12.6193240Z * [new branch] gh/amjames/18/base -> origin/gh/amjames/18/base 2025-08-14T21:24:12.6193615Z * [new branch] gh/amjames/18/head -> origin/gh/amjames/18/head 2025-08-14T21:24:12.6193967Z * [new branch] gh/amjames/18/orig -> origin/gh/amjames/18/orig 2025-08-14T21:24:12.6194523Z * [new branch] gh/andrewor14/35/base -> origin/gh/andrewor14/35/base 2025-08-14T21:24:12.6195205Z * [new branch] gh/andrewor14/35/head -> origin/gh/andrewor14/35/head 2025-08-14T21:24:12.6195926Z * [new branch] gh/andrewor14/35/orig -> origin/gh/andrewor14/35/orig 2025-08-14T21:24:12.6197293Z * [new branch] gh/andrewor14/50/base -> origin/gh/andrewor14/50/base 2025-08-14T21:24:12.6197673Z * [new branch] gh/andrewor14/50/head -> origin/gh/andrewor14/50/head 2025-08-14T21:24:12.6211181Z * [new branch] gh/andrewor14/50/orig -> origin/gh/andrewor14/50/orig 2025-08-14T21:24:12.6211630Z * [new branch] gh/andyanwang/1/base -> origin/gh/andyanwang/1/base 2025-08-14T21:24:12.6212155Z * [new branch] gh/andyanwang/1/head -> origin/gh/andyanwang/1/head 2025-08-14T21:24:12.6212528Z * [new branch] gh/andyanwang/1/orig -> origin/gh/andyanwang/1/orig 2025-08-14T21:24:12.6212888Z * [new branch] gh/andyanwang/13/base -> origin/gh/andyanwang/13/base 2025-08-14T21:24:12.6213240Z * [new branch] gh/andyanwang/13/head -> origin/gh/andyanwang/13/head 2025-08-14T21:24:12.6213596Z * [new branch] gh/andyanwang/13/orig -> origin/gh/andyanwang/13/orig 2025-08-14T21:24:12.6213946Z * [new branch] gh/andyanwang/2/base -> origin/gh/andyanwang/2/base 2025-08-14T21:24:12.6214283Z * [new branch] gh/andyanwang/2/head -> origin/gh/andyanwang/2/head 2025-08-14T21:24:12.6214627Z * [new branch] gh/andyanwang/2/orig -> origin/gh/andyanwang/2/orig 2025-08-14T21:24:12.6214972Z * [new branch] gh/andyanwang/28/base -> origin/gh/andyanwang/28/base 2025-08-14T21:24:12.6215323Z * [new branch] gh/andyanwang/28/head -> origin/gh/andyanwang/28/head 2025-08-14T21:24:12.6215667Z * [new branch] gh/andyanwang/28/orig -> origin/gh/andyanwang/28/orig 2025-08-14T21:24:12.6216012Z * [new branch] gh/andyanwang/3/base -> origin/gh/andyanwang/3/base 2025-08-14T21:24:12.6216379Z * [new branch] gh/andyanwang/3/head -> origin/gh/andyanwang/3/head 2025-08-14T21:24:12.6216718Z * [new branch] gh/andyanwang/3/orig -> origin/gh/andyanwang/3/orig 2025-08-14T21:24:12.6217034Z * [new branch] gh/andyanwang/30/base -> origin/gh/andyanwang/30/base 2025-08-14T21:24:12.6217359Z * [new branch] gh/andyanwang/30/orig -> origin/gh/andyanwang/30/orig 2025-08-14T21:24:12.6217695Z * [new branch] gh/andyanwang/31/base -> origin/gh/andyanwang/31/base 2025-08-14T21:24:12.6218044Z * [new branch] gh/andyanwang/31/orig -> origin/gh/andyanwang/31/orig 2025-08-14T21:24:12.6218437Z * [new branch] gh/andyanwang/32/base -> origin/gh/andyanwang/32/base 2025-08-14T21:24:12.6218784Z * [new branch] gh/andyanwang/32/head -> origin/gh/andyanwang/32/head 2025-08-14T21:24:12.6219128Z * [new branch] gh/andyanwang/32/orig -> origin/gh/andyanwang/32/orig 2025-08-14T21:24:12.6219463Z * [new branch] gh/andyanwang/33/base -> origin/gh/andyanwang/33/base 2025-08-14T21:24:12.6220703Z * [new branch] gh/andyanwang/33/head -> origin/gh/andyanwang/33/head 2025-08-14T21:24:12.6221063Z * [new branch] gh/andyanwang/33/orig -> origin/gh/andyanwang/33/orig 2025-08-14T21:24:12.6221423Z * [new branch] gh/andyanwang/34/base -> origin/gh/andyanwang/34/base 2025-08-14T21:24:12.6221773Z * [new branch] gh/andyanwang/34/head -> origin/gh/andyanwang/34/head 2025-08-14T21:24:12.6222126Z * [new branch] gh/andyanwang/34/orig -> origin/gh/andyanwang/34/orig 2025-08-14T21:24:12.6222492Z * [new branch] gh/andyanwang/35/base -> origin/gh/andyanwang/35/base 2025-08-14T21:24:12.6222858Z * [new branch] gh/andyanwang/35/head -> origin/gh/andyanwang/35/head 2025-08-14T21:24:12.6223211Z * [new branch] gh/andyanwang/35/orig -> origin/gh/andyanwang/35/orig 2025-08-14T21:24:12.6228551Z * [new branch] gh/andyanwang/36/base -> origin/gh/andyanwang/36/base 2025-08-14T21:24:12.6228981Z * [new branch] gh/andyanwang/36/head -> origin/gh/andyanwang/36/head 2025-08-14T21:24:12.6229355Z * [new branch] gh/andyanwang/36/orig -> origin/gh/andyanwang/36/orig 2025-08-14T21:24:12.6229716Z * [new branch] gh/andyanwang/37/base -> origin/gh/andyanwang/37/base 2025-08-14T21:24:12.6230082Z * [new branch] gh/andyanwang/37/head -> origin/gh/andyanwang/37/head 2025-08-14T21:24:12.6230633Z * [new branch] gh/andyanwang/37/orig -> origin/gh/andyanwang/37/orig 2025-08-14T21:24:12.6231025Z * [new branch] gh/andyanwang/38/base -> origin/gh/andyanwang/38/base 2025-08-14T21:24:12.6231379Z * [new branch] gh/andyanwang/38/head -> origin/gh/andyanwang/38/head 2025-08-14T21:24:12.6231735Z * [new branch] gh/andyanwang/38/orig -> origin/gh/andyanwang/38/orig 2025-08-14T21:24:12.6232099Z * [new branch] gh/andyanwang/39/base -> origin/gh/andyanwang/39/base 2025-08-14T21:24:12.6232474Z * [new branch] gh/andyanwang/39/head -> origin/gh/andyanwang/39/head 2025-08-14T21:24:12.6232837Z * [new branch] gh/andyanwang/39/orig -> origin/gh/andyanwang/39/orig 2025-08-14T21:24:12.6238125Z * [new branch] gh/andyanwang/4/base -> origin/gh/andyanwang/4/base 2025-08-14T21:24:12.6238572Z * [new branch] gh/andyanwang/4/head -> origin/gh/andyanwang/4/head 2025-08-14T21:24:12.6238938Z * [new branch] gh/andyanwang/4/orig -> origin/gh/andyanwang/4/orig 2025-08-14T21:24:12.6239295Z * [new branch] gh/andyanwang/40/base -> origin/gh/andyanwang/40/base 2025-08-14T21:24:12.6239657Z * [new branch] gh/andyanwang/40/head -> origin/gh/andyanwang/40/head 2025-08-14T21:24:12.6240008Z * [new branch] gh/andyanwang/40/orig -> origin/gh/andyanwang/40/orig 2025-08-14T21:24:12.6240360Z * [new branch] gh/angelayi/102/base -> origin/gh/angelayi/102/base 2025-08-14T21:24:12.6240696Z * [new branch] gh/angelayi/102/head -> origin/gh/angelayi/102/head 2025-08-14T21:24:12.6241043Z * [new branch] gh/angelayi/102/orig -> origin/gh/angelayi/102/orig 2025-08-14T21:24:12.6241392Z * [new branch] gh/angelayi/103/base -> origin/gh/angelayi/103/base 2025-08-14T21:24:12.6241741Z * [new branch] gh/angelayi/103/head -> origin/gh/angelayi/103/head 2025-08-14T21:24:12.6242315Z * [new branch] gh/angelayi/103/orig -> origin/gh/angelayi/103/orig 2025-08-14T21:24:12.6242664Z * [new branch] gh/angelayi/104/base -> origin/gh/angelayi/104/base 2025-08-14T21:24:12.6243212Z * [new branch] gh/angelayi/104/head -> origin/gh/angelayi/104/head 2025-08-14T21:24:12.6243644Z * [new branch] gh/angelayi/104/orig -> origin/gh/angelayi/104/orig 2025-08-14T21:24:12.6245717Z * [new branch] gh/angelayi/105/base -> origin/gh/angelayi/105/base 2025-08-14T21:24:12.6246328Z * [new branch] gh/angelayi/105/head -> origin/gh/angelayi/105/head 2025-08-14T21:24:12.6246840Z * [new branch] gh/angelayi/105/orig -> origin/gh/angelayi/105/orig 2025-08-14T21:24:12.6247338Z * [new branch] gh/angelayi/106/base -> origin/gh/angelayi/106/base 2025-08-14T21:24:12.6247898Z * [new branch] gh/angelayi/106/head -> origin/gh/angelayi/106/head 2025-08-14T21:24:12.6248602Z * [new branch] gh/angelayi/106/orig -> origin/gh/angelayi/106/orig 2025-08-14T21:24:12.6249298Z * [new branch] gh/angelayi/107/base -> origin/gh/angelayi/107/base 2025-08-14T21:24:12.6249888Z * [new branch] gh/angelayi/107/head -> origin/gh/angelayi/107/head 2025-08-14T21:24:12.6253178Z * [new branch] gh/angelayi/108/base -> origin/gh/angelayi/108/base 2025-08-14T21:24:12.6253754Z * [new branch] gh/angelayi/108/head -> origin/gh/angelayi/108/head 2025-08-14T21:24:12.6254246Z * [new branch] gh/angelayi/108/orig -> origin/gh/angelayi/108/orig 2025-08-14T21:24:12.6255036Z * [new branch] gh/angelayi/109/base -> origin/gh/angelayi/109/base 2025-08-14T21:24:12.6255452Z * [new branch] gh/angelayi/109/head -> origin/gh/angelayi/109/head 2025-08-14T21:24:12.6256024Z * [new branch] gh/angelayi/109/orig -> origin/gh/angelayi/109/orig 2025-08-14T21:24:12.6256792Z * [new branch] gh/angelayi/110/base -> origin/gh/angelayi/110/base 2025-08-14T21:24:12.6257230Z * [new branch] gh/angelayi/110/head -> origin/gh/angelayi/110/head 2025-08-14T21:24:12.6257637Z * [new branch] gh/angelayi/110/orig -> origin/gh/angelayi/110/orig 2025-08-14T21:24:12.6258101Z * [new branch] gh/angelayi/97/base -> origin/gh/angelayi/97/base 2025-08-14T21:24:12.6258592Z * [new branch] gh/angelayi/97/head -> origin/gh/angelayi/97/head 2025-08-14T21:24:12.6259479Z * [new branch] gh/angelayi/97/orig -> origin/gh/angelayi/97/orig 2025-08-14T21:24:12.6260151Z * [new branch] gh/ani300/1/base -> origin/gh/ani300/1/base 2025-08-14T21:24:12.6260736Z * [new branch] gh/ani300/1/head -> origin/gh/ani300/1/head 2025-08-14T21:24:12.6261423Z * [new branch] gh/ani300/1/orig -> origin/gh/ani300/1/orig 2025-08-14T21:24:12.6263567Z * [new branch] gh/anijain2305/753/base -> origin/gh/anijain2305/753/base 2025-08-14T21:24:12.6264185Z * [new branch] gh/anijain2305/753/head -> origin/gh/anijain2305/753/head 2025-08-14T21:24:12.6264701Z * [new branch] gh/anijain2305/753/orig -> origin/gh/anijain2305/753/orig 2025-08-14T21:24:12.6265462Z * [new branch] gh/anijain2305/766/base -> origin/gh/anijain2305/766/base 2025-08-14T21:24:12.6270190Z * [new branch] gh/anijain2305/766/head -> origin/gh/anijain2305/766/head 2025-08-14T21:24:12.6270622Z * [new branch] gh/anijain2305/766/orig -> origin/gh/anijain2305/766/orig 2025-08-14T21:24:12.6271017Z * [new branch] gh/anijain2305/790/base -> origin/gh/anijain2305/790/base 2025-08-14T21:24:12.6271406Z * [new branch] gh/anijain2305/790/head -> origin/gh/anijain2305/790/head 2025-08-14T21:24:12.6271980Z * [new branch] gh/anijain2305/790/orig -> origin/gh/anijain2305/790/orig 2025-08-14T21:24:12.6272419Z * [new branch] gh/anijain2305/792/base -> origin/gh/anijain2305/792/base 2025-08-14T21:24:12.6272778Z * [new branch] gh/anijain2305/792/head -> origin/gh/anijain2305/792/head 2025-08-14T21:24:12.6273150Z * [new branch] gh/anijain2305/792/orig -> origin/gh/anijain2305/792/orig 2025-08-14T21:24:12.6273507Z * [new branch] gh/anijain2305/803/base -> origin/gh/anijain2305/803/base 2025-08-14T21:24:12.6273882Z * [new branch] gh/anijain2305/803/head -> origin/gh/anijain2305/803/head 2025-08-14T21:24:12.6274395Z * [new branch] gh/anijain2305/803/orig -> origin/gh/anijain2305/803/orig 2025-08-14T21:24:12.6279718Z * [new branch] gh/anijain2305/804/base -> origin/gh/anijain2305/804/base 2025-08-14T21:24:12.6280395Z * [new branch] gh/anijain2305/804/head -> origin/gh/anijain2305/804/head 2025-08-14T21:24:12.6280943Z * [new branch] gh/anijain2305/804/orig -> origin/gh/anijain2305/804/orig 2025-08-14T21:24:12.6281338Z * [new branch] gh/anijain2305/805/base -> origin/gh/anijain2305/805/base 2025-08-14T21:24:12.6281728Z * [new branch] gh/anijain2305/805/head -> origin/gh/anijain2305/805/head 2025-08-14T21:24:12.6282108Z * [new branch] gh/anijain2305/805/orig -> origin/gh/anijain2305/805/orig 2025-08-14T21:24:12.6282492Z * [new branch] gh/anijain2305/810/base -> origin/gh/anijain2305/810/base 2025-08-14T21:24:12.6282872Z * [new branch] gh/anijain2305/810/head -> origin/gh/anijain2305/810/head 2025-08-14T21:24:12.6283237Z * [new branch] gh/anijain2305/810/orig -> origin/gh/anijain2305/810/orig 2025-08-14T21:24:12.6283792Z * [new branch] gh/anijain2305/811/base -> origin/gh/anijain2305/811/base 2025-08-14T21:24:12.6284204Z * [new branch] gh/anijain2305/811/head -> origin/gh/anijain2305/811/head 2025-08-14T21:24:12.6284592Z * [new branch] gh/anijain2305/811/orig -> origin/gh/anijain2305/811/orig 2025-08-14T21:24:12.6284973Z * [new branch] gh/anijain2305/812/base -> origin/gh/anijain2305/812/base 2025-08-14T21:24:12.6285365Z * [new branch] gh/anijain2305/812/head -> origin/gh/anijain2305/812/head 2025-08-14T21:24:12.6285961Z * [new branch] gh/anijain2305/812/orig -> origin/gh/anijain2305/812/orig 2025-08-14T21:24:12.6287220Z * [new branch] gh/anijain2305/813/base -> origin/gh/anijain2305/813/base 2025-08-14T21:24:12.6288041Z * [new branch] gh/anijain2305/813/head -> origin/gh/anijain2305/813/head 2025-08-14T21:24:12.6288732Z * [new branch] gh/anijain2305/813/orig -> origin/gh/anijain2305/813/orig 2025-08-14T21:24:12.6289296Z * [new branch] gh/anijain2305/814/base -> origin/gh/anijain2305/814/base 2025-08-14T21:24:12.6289953Z * [new branch] gh/anijain2305/814/head -> origin/gh/anijain2305/814/head 2025-08-14T21:24:12.6293609Z * [new branch] gh/anijain2305/814/orig -> origin/gh/anijain2305/814/orig 2025-08-14T21:24:12.6294037Z * [new branch] gh/anijain2305/815/base -> origin/gh/anijain2305/815/base 2025-08-14T21:24:12.6294406Z * [new branch] gh/anijain2305/815/head -> origin/gh/anijain2305/815/head 2025-08-14T21:24:12.6294769Z * [new branch] gh/anijain2305/815/orig -> origin/gh/anijain2305/815/orig 2025-08-14T21:24:12.6295124Z * [new branch] gh/anijain2305/816/base -> origin/gh/anijain2305/816/base 2025-08-14T21:24:12.6295482Z * [new branch] gh/anijain2305/816/head -> origin/gh/anijain2305/816/head 2025-08-14T21:24:12.6295863Z * [new branch] gh/anijain2305/817/base -> origin/gh/anijain2305/817/base 2025-08-14T21:24:12.6296413Z * [new branch] gh/anijain2305/817/head -> origin/gh/anijain2305/817/head 2025-08-14T21:24:12.6297074Z * [new branch] gh/anijain2305/817/orig -> origin/gh/anijain2305/817/orig 2025-08-14T21:24:12.6297998Z * [new branch] gh/anijain2305/818/base -> origin/gh/anijain2305/818/base 2025-08-14T21:24:12.6298960Z * [new branch] gh/anijain2305/818/head -> origin/gh/anijain2305/818/head 2025-08-14T21:24:12.6299262Z * [new branch] gh/anijain2305/818/orig -> origin/gh/anijain2305/818/orig 2025-08-14T21:24:12.6300825Z * [new branch] gh/anijain2305/819/base -> origin/gh/anijain2305/819/base 2025-08-14T21:24:12.6301312Z * [new branch] gh/anijain2305/819/head -> origin/gh/anijain2305/819/head 2025-08-14T21:24:12.6302892Z * [new branch] gh/anijain2305/819/orig -> origin/gh/anijain2305/819/orig 2025-08-14T21:24:12.6303135Z * [new branch] gh/anijain2305/820/base -> origin/gh/anijain2305/820/base 2025-08-14T21:24:12.6304225Z * [new branch] gh/anijain2305/820/head -> origin/gh/anijain2305/820/head 2025-08-14T21:24:12.6304488Z * [new branch] gh/anijain2305/820/orig -> origin/gh/anijain2305/820/orig 2025-08-14T21:24:12.6308590Z * [new branch] gh/anijain2305/821/base -> origin/gh/anijain2305/821/base 2025-08-14T21:24:12.6308782Z * [new branch] gh/anijain2305/821/head -> origin/gh/anijain2305/821/head 2025-08-14T21:24:12.6308943Z * [new branch] gh/anijain2305/821/orig -> origin/gh/anijain2305/821/orig 2025-08-14T21:24:12.6309092Z * [new branch] gh/anijain2305/822/base -> origin/gh/anijain2305/822/base 2025-08-14T21:24:12.6309249Z * [new branch] gh/anijain2305/822/head -> origin/gh/anijain2305/822/head 2025-08-14T21:24:12.6309713Z * [new branch] gh/anijain2305/822/orig -> origin/gh/anijain2305/822/orig 2025-08-14T21:24:12.6310174Z * [new branch] gh/anijain2305/823/base -> origin/gh/anijain2305/823/base 2025-08-14T21:24:12.6311195Z * [new branch] gh/anijain2305/823/head -> origin/gh/anijain2305/823/head 2025-08-14T21:24:12.6313524Z * [new branch] gh/anijain2305/823/orig -> origin/gh/anijain2305/823/orig 2025-08-14T21:24:12.6313864Z * [new branch] gh/anijain2305/824/base -> origin/gh/anijain2305/824/base 2025-08-14T21:24:12.6314127Z * [new branch] gh/anijain2305/824/head -> origin/gh/anijain2305/824/head 2025-08-14T21:24:12.6316858Z * [new branch] gh/anijain2305/824/orig -> origin/gh/anijain2305/824/orig 2025-08-14T21:24:12.6317194Z * [new branch] gh/anijain2305/825/base -> origin/gh/anijain2305/825/base 2025-08-14T21:24:12.6317444Z * [new branch] gh/anijain2305/825/head -> origin/gh/anijain2305/825/head 2025-08-14T21:24:12.6317738Z * [new branch] gh/anijain2305/825/orig -> origin/gh/anijain2305/825/orig 2025-08-14T21:24:12.6318020Z * [new branch] gh/anijain2305/826/base -> origin/gh/anijain2305/826/base 2025-08-14T21:24:12.6321411Z * [new branch] gh/anijain2305/826/head -> origin/gh/anijain2305/826/head 2025-08-14T21:24:12.6321987Z * [new branch] gh/anijain2305/826/orig -> origin/gh/anijain2305/826/orig 2025-08-14T21:24:12.6322180Z * [new branch] gh/anijain2305/827/base -> origin/gh/anijain2305/827/base 2025-08-14T21:24:12.6322329Z * [new branch] gh/anijain2305/827/head -> origin/gh/anijain2305/827/head 2025-08-14T21:24:12.6322482Z * [new branch] gh/anijain2305/827/orig -> origin/gh/anijain2305/827/orig 2025-08-14T21:24:12.6322630Z * [new branch] gh/anijain2305/828/base -> origin/gh/anijain2305/828/base 2025-08-14T21:24:12.6322817Z * [new branch] gh/anijain2305/828/head -> origin/gh/anijain2305/828/head 2025-08-14T21:24:12.6323138Z * [new branch] gh/anijain2305/828/orig -> origin/gh/anijain2305/828/orig 2025-08-14T21:24:12.6323521Z * [new branch] gh/anijain2305/829/base -> origin/gh/anijain2305/829/base 2025-08-14T21:24:12.6323774Z * [new branch] gh/anijain2305/829/head -> origin/gh/anijain2305/829/head 2025-08-14T21:24:12.6324838Z * [new branch] gh/anijain2305/829/orig -> origin/gh/anijain2305/829/orig 2025-08-14T21:24:12.6325896Z * [new branch] gh/anijain2305/830/base -> origin/gh/anijain2305/830/base 2025-08-14T21:24:12.6326222Z * [new branch] gh/anijain2305/830/head -> origin/gh/anijain2305/830/head 2025-08-14T21:24:12.6332999Z * [new branch] gh/anijain2305/830/orig -> origin/gh/anijain2305/830/orig 2025-08-14T21:24:12.6333206Z * [new branch] gh/anijain2305/831/base -> origin/gh/anijain2305/831/base 2025-08-14T21:24:12.6333374Z * [new branch] gh/anijain2305/831/head -> origin/gh/anijain2305/831/head 2025-08-14T21:24:12.6333533Z * [new branch] gh/anijain2305/831/orig -> origin/gh/anijain2305/831/orig 2025-08-14T21:24:12.6333690Z * [new branch] gh/anijain2305/832/base -> origin/gh/anijain2305/832/base 2025-08-14T21:24:12.6333834Z * [new branch] gh/anijain2305/832/head -> origin/gh/anijain2305/832/head 2025-08-14T21:24:12.6333987Z * [new branch] gh/anijain2305/832/orig -> origin/gh/anijain2305/832/orig 2025-08-14T21:24:12.6338515Z * [new branch] gh/anijain2305/833/base -> origin/gh/anijain2305/833/base 2025-08-14T21:24:12.6338695Z * [new branch] gh/anijain2305/833/head -> origin/gh/anijain2305/833/head 2025-08-14T21:24:12.6338846Z * [new branch] gh/anijain2305/833/orig -> origin/gh/anijain2305/833/orig 2025-08-14T21:24:12.6339211Z * [new branch] gh/anijain2305/834/base -> origin/gh/anijain2305/834/base 2025-08-14T21:24:12.6339376Z * [new branch] gh/anijain2305/834/head -> origin/gh/anijain2305/834/head 2025-08-14T21:24:12.6339514Z * [new branch] gh/anijain2305/834/orig -> origin/gh/anijain2305/834/orig 2025-08-14T21:24:12.6339651Z * [new branch] gh/anijain2305/835/base -> origin/gh/anijain2305/835/base 2025-08-14T21:24:12.6339833Z * [new branch] gh/anijain2305/835/head -> origin/gh/anijain2305/835/head 2025-08-14T21:24:12.6339969Z * [new branch] gh/anijain2305/835/orig -> origin/gh/anijain2305/835/orig 2025-08-14T21:24:12.6340115Z * [new branch] gh/anijain2305/836/base -> origin/gh/anijain2305/836/base 2025-08-14T21:24:12.6340252Z * [new branch] gh/anijain2305/836/head -> origin/gh/anijain2305/836/head 2025-08-14T21:24:12.6340395Z * [new branch] gh/anijain2305/836/orig -> origin/gh/anijain2305/836/orig 2025-08-14T21:24:12.6340541Z * [new branch] gh/anijain2305/837/base -> origin/gh/anijain2305/837/base 2025-08-14T21:24:12.6340691Z * [new branch] gh/anijain2305/837/head -> origin/gh/anijain2305/837/head 2025-08-14T21:24:12.6341518Z * [new branch] gh/anijain2305/837/orig -> origin/gh/anijain2305/837/orig 2025-08-14T21:24:12.6342497Z * [new branch] gh/anijain2305/838/base -> origin/gh/anijain2305/838/base 2025-08-14T21:24:12.6342725Z * [new branch] gh/anijain2305/838/head -> origin/gh/anijain2305/838/head 2025-08-14T21:24:12.6347025Z * [new branch] gh/anijain2305/838/orig -> origin/gh/anijain2305/838/orig 2025-08-14T21:24:12.6347213Z * [new branch] gh/anijain2305/839/base -> origin/gh/anijain2305/839/base 2025-08-14T21:24:12.6347372Z * [new branch] gh/anijain2305/839/head -> origin/gh/anijain2305/839/head 2025-08-14T21:24:12.6347518Z * [new branch] gh/anijain2305/839/orig -> origin/gh/anijain2305/839/orig 2025-08-14T21:24:12.6347884Z * [new branch] gh/anijain2305/840/base -> origin/gh/anijain2305/840/base 2025-08-14T21:24:12.6348040Z * [new branch] gh/anijain2305/840/head -> origin/gh/anijain2305/840/head 2025-08-14T21:24:12.6348439Z * [new branch] gh/anijain2305/840/orig -> origin/gh/anijain2305/840/orig 2025-08-14T21:24:12.6353379Z * [new branch] gh/anijain2305/841/base -> origin/gh/anijain2305/841/base 2025-08-14T21:24:12.6353569Z * [new branch] gh/anijain2305/841/head -> origin/gh/anijain2305/841/head 2025-08-14T21:24:12.6353727Z * [new branch] gh/anijain2305/841/orig -> origin/gh/anijain2305/841/orig 2025-08-14T21:24:12.6353872Z * [new branch] gh/anijain2305/842/base -> origin/gh/anijain2305/842/base 2025-08-14T21:24:12.6354014Z * [new branch] gh/anijain2305/842/head -> origin/gh/anijain2305/842/head 2025-08-14T21:24:12.6354180Z * [new branch] gh/anijain2305/842/orig -> origin/gh/anijain2305/842/orig 2025-08-14T21:24:12.6354335Z * [new branch] gh/anijain2305/843/base -> origin/gh/anijain2305/843/base 2025-08-14T21:24:12.6354602Z * [new branch] gh/anijain2305/843/head -> origin/gh/anijain2305/843/head 2025-08-14T21:24:12.6356015Z * [new branch] gh/anijain2305/843/orig -> origin/gh/anijain2305/843/orig 2025-08-14T21:24:12.6356370Z * [new branch] gh/anijain2305/844/base -> origin/gh/anijain2305/844/base 2025-08-14T21:24:12.6361348Z * [new branch] gh/anijain2305/844/head -> origin/gh/anijain2305/844/head 2025-08-14T21:24:12.6361542Z * [new branch] gh/anijain2305/844/orig -> origin/gh/anijain2305/844/orig 2025-08-14T21:24:12.6361707Z * [new branch] gh/anijain2305/845/base -> origin/gh/anijain2305/845/base 2025-08-14T21:24:12.6361856Z * [new branch] gh/anijain2305/845/head -> origin/gh/anijain2305/845/head 2025-08-14T21:24:12.6362163Z * [new branch] gh/anijain2305/845/orig -> origin/gh/anijain2305/845/orig 2025-08-14T21:24:12.6362318Z * [new branch] gh/anijain2305/846/base -> origin/gh/anijain2305/846/base 2025-08-14T21:24:12.6362460Z * [new branch] gh/anijain2305/846/head -> origin/gh/anijain2305/846/head 2025-08-14T21:24:12.6362618Z * [new branch] gh/anijain2305/846/orig -> origin/gh/anijain2305/846/orig 2025-08-14T21:24:12.6363557Z * [new branch] gh/anijain2305/847/base -> origin/gh/anijain2305/847/base 2025-08-14T21:24:12.6363994Z * [new branch] gh/anijain2305/847/head -> origin/gh/anijain2305/847/head 2025-08-14T21:24:12.6365076Z * [new branch] gh/anijain2305/847/orig -> origin/gh/anijain2305/847/orig 2025-08-14T21:24:12.6366085Z * [new branch] gh/anijain2305/848/base -> origin/gh/anijain2305/848/base 2025-08-14T21:24:12.6366567Z * [new branch] gh/anijain2305/848/head -> origin/gh/anijain2305/848/head 2025-08-14T21:24:12.6367342Z * [new branch] gh/anijain2305/848/orig -> origin/gh/anijain2305/848/orig 2025-08-14T21:24:12.6371504Z * [new branch] gh/anjali411/216/base -> origin/gh/anjali411/216/base 2025-08-14T21:24:12.6371681Z * [new branch] gh/anjali411/216/head -> origin/gh/anjali411/216/head 2025-08-14T21:24:12.6371830Z * [new branch] gh/anjali411/216/orig -> origin/gh/anjali411/216/orig 2025-08-14T21:24:12.6371990Z * [new branch] gh/ankitageorge/10/base -> origin/gh/ankitageorge/10/base 2025-08-14T21:24:12.6372146Z * [new branch] gh/ankitageorge/10/head -> origin/gh/ankitageorge/10/head 2025-08-14T21:24:12.6372847Z * [new branch] gh/ankitageorge/10/orig -> origin/gh/ankitageorge/10/orig 2025-08-14T21:24:12.6373899Z * [new branch] gh/ankitageorge/12/base -> origin/gh/ankitageorge/12/base 2025-08-14T21:24:12.6374271Z * [new branch] gh/ankitageorge/12/head -> origin/gh/ankitageorge/12/head 2025-08-14T21:24:12.6374979Z * [new branch] gh/ankitageorge/12/orig -> origin/gh/ankitageorge/12/orig 2025-08-14T21:24:12.6378491Z * [new branch] gh/ankitageorge/13/base -> origin/gh/ankitageorge/13/base 2025-08-14T21:24:12.6378682Z * [new branch] gh/ankitageorge/13/head -> origin/gh/ankitageorge/13/head 2025-08-14T21:24:12.6378834Z * [new branch] gh/ankitageorge/13/orig -> origin/gh/ankitageorge/13/orig 2025-08-14T21:24:12.6378991Z * [new branch] gh/ankitageorge/14/base -> origin/gh/ankitageorge/14/base 2025-08-14T21:24:12.6379149Z * [new branch] gh/ankitageorge/14/head -> origin/gh/ankitageorge/14/head 2025-08-14T21:24:12.6380132Z * [new branch] gh/ankitageorge/14/orig -> origin/gh/ankitageorge/14/orig 2025-08-14T21:24:12.6381119Z * [new branch] gh/ankitageorge/15/base -> origin/gh/ankitageorge/15/base 2025-08-14T21:24:12.6381684Z * [new branch] gh/ankitageorge/15/head -> origin/gh/ankitageorge/15/head 2025-08-14T21:24:12.6382572Z * [new branch] gh/ankitageorge/15/orig -> origin/gh/ankitageorge/15/orig 2025-08-14T21:24:12.6386483Z * [new branch] gh/ankitageorge/16/base -> origin/gh/ankitageorge/16/base 2025-08-14T21:24:12.6386681Z * [new branch] gh/ankitageorge/16/head -> origin/gh/ankitageorge/16/head 2025-08-14T21:24:12.6386835Z * [new branch] gh/ankitageorge/16/orig -> origin/gh/ankitageorge/16/orig 2025-08-14T21:24:12.6386992Z * [new branch] gh/ankitageorge/17/base -> origin/gh/ankitageorge/17/base 2025-08-14T21:24:12.6387140Z * [new branch] gh/ankitageorge/17/head -> origin/gh/ankitageorge/17/head 2025-08-14T21:24:12.6387319Z * [new branch] gh/ankitageorge/17/orig -> origin/gh/ankitageorge/17/orig 2025-08-14T21:24:12.6391819Z * [new branch] gh/ankitageorge/18/base -> origin/gh/ankitageorge/18/base 2025-08-14T21:24:12.6392024Z * [new branch] gh/ankitageorge/18/head -> origin/gh/ankitageorge/18/head 2025-08-14T21:24:12.6392186Z * [new branch] gh/ankitageorge/18/orig -> origin/gh/ankitageorge/18/orig 2025-08-14T21:24:12.6392335Z * [new branch] gh/ankitageorge/19/base -> origin/gh/ankitageorge/19/base 2025-08-14T21:24:12.6392483Z * [new branch] gh/ankitageorge/19/head -> origin/gh/ankitageorge/19/head 2025-08-14T21:24:12.6392651Z * [new branch] gh/ankitageorge/19/orig -> origin/gh/ankitageorge/19/orig 2025-08-14T21:24:12.6393591Z * [new branch] gh/ankitageorge/20/base -> origin/gh/ankitageorge/20/base 2025-08-14T21:24:12.6394106Z * [new branch] gh/ankitageorge/20/head -> origin/gh/ankitageorge/20/head 2025-08-14T21:24:12.6395089Z * [new branch] gh/ankitageorge/20/orig -> origin/gh/ankitageorge/20/orig 2025-08-14T21:24:12.6395859Z * [new branch] gh/ankitageorge/21/base -> origin/gh/ankitageorge/21/base 2025-08-14T21:24:12.6396763Z * [new branch] gh/ankitageorge/21/head -> origin/gh/ankitageorge/21/head 2025-08-14T21:24:12.6397236Z * [new branch] gh/ankitageorge/21/orig -> origin/gh/ankitageorge/21/orig 2025-08-14T21:24:12.6398830Z * [new branch] gh/anshul-si/1/base -> origin/gh/anshul-si/1/base 2025-08-14T21:24:12.6399212Z * [new branch] gh/anshul-si/1/head -> origin/gh/anshul-si/1/head 2025-08-14T21:24:12.6401064Z * [new branch] gh/anshul-si/10/base -> origin/gh/anshul-si/10/base 2025-08-14T21:24:12.6401246Z * [new branch] gh/anshul-si/10/head -> origin/gh/anshul-si/10/head 2025-08-14T21:24:12.6402117Z * [new branch] gh/anshul-si/10/orig -> origin/gh/anshul-si/10/orig 2025-08-14T21:24:12.6403354Z * [new branch] gh/anshul-si/11/base -> origin/gh/anshul-si/11/base 2025-08-14T21:24:12.6403741Z * [new branch] gh/anshul-si/11/head -> origin/gh/anshul-si/11/head 2025-08-14T21:24:12.6404507Z * [new branch] gh/anshul-si/11/orig -> origin/gh/anshul-si/11/orig 2025-08-14T21:24:12.6405810Z * [new branch] gh/anshul-si/12/base -> origin/gh/anshul-si/12/base 2025-08-14T21:24:12.6406380Z * [new branch] gh/anshul-si/12/head -> origin/gh/anshul-si/12/head 2025-08-14T21:24:12.6411135Z * [new branch] gh/anshul-si/12/orig -> origin/gh/anshul-si/12/orig 2025-08-14T21:24:12.6411333Z * [new branch] gh/anshul-si/13/base -> origin/gh/anshul-si/13/base 2025-08-14T21:24:12.6411477Z * [new branch] gh/anshul-si/13/head -> origin/gh/anshul-si/13/head 2025-08-14T21:24:12.6411617Z * [new branch] gh/anshul-si/13/orig -> origin/gh/anshul-si/13/orig 2025-08-14T21:24:12.6411786Z * [new branch] gh/anshul-si/14/base -> origin/gh/anshul-si/14/base 2025-08-14T21:24:12.6411934Z * [new branch] gh/anshul-si/14/head -> origin/gh/anshul-si/14/head 2025-08-14T21:24:12.6412095Z * [new branch] gh/anshul-si/14/orig -> origin/gh/anshul-si/14/orig 2025-08-14T21:24:12.6412834Z * [new branch] gh/anshul-si/15/base -> origin/gh/anshul-si/15/base 2025-08-14T21:24:12.6413540Z * [new branch] gh/anshul-si/15/head -> origin/gh/anshul-si/15/head 2025-08-14T21:24:12.6414181Z * [new branch] gh/anshul-si/15/orig -> origin/gh/anshul-si/15/orig 2025-08-14T21:24:12.6418074Z * [new branch] gh/anshul-si/16/base -> origin/gh/anshul-si/16/base 2025-08-14T21:24:12.6418263Z * [new branch] gh/anshul-si/16/head -> origin/gh/anshul-si/16/head 2025-08-14T21:24:12.6418549Z * [new branch] gh/anshul-si/16/orig -> origin/gh/anshul-si/16/orig 2025-08-14T21:24:12.6418706Z * [new branch] gh/anshul-si/17/base -> origin/gh/anshul-si/17/base 2025-08-14T21:24:12.6423202Z * [new branch] gh/anshul-si/17/head -> origin/gh/anshul-si/17/head 2025-08-14T21:24:12.6423388Z * [new branch] gh/anshul-si/17/orig -> origin/gh/anshul-si/17/orig 2025-08-14T21:24:12.6423540Z * [new branch] gh/anshul-si/18/base -> origin/gh/anshul-si/18/base 2025-08-14T21:24:12.6423675Z * [new branch] gh/anshul-si/18/head -> origin/gh/anshul-si/18/head 2025-08-14T21:24:12.6423819Z * [new branch] gh/anshul-si/18/orig -> origin/gh/anshul-si/18/orig 2025-08-14T21:24:12.6423954Z * [new branch] gh/anshul-si/19/base -> origin/gh/anshul-si/19/base 2025-08-14T21:24:12.6424088Z * [new branch] gh/anshul-si/19/head -> origin/gh/anshul-si/19/head 2025-08-14T21:24:12.6424249Z * [new branch] gh/anshul-si/19/orig -> origin/gh/anshul-si/19/orig 2025-08-14T21:24:12.6424437Z * [new branch] gh/anshul-si/2/base -> origin/gh/anshul-si/2/base 2025-08-14T21:24:12.6425120Z * [new branch] gh/anshul-si/2/head -> origin/gh/anshul-si/2/head 2025-08-14T21:24:12.6426361Z * [new branch] gh/anshul-si/20/base -> origin/gh/anshul-si/20/base 2025-08-14T21:24:12.6429561Z * [new branch] gh/anshul-si/20/head -> origin/gh/anshul-si/20/head 2025-08-14T21:24:12.6429716Z * [new branch] gh/anshul-si/20/orig -> origin/gh/anshul-si/20/orig 2025-08-14T21:24:12.6429850Z * [new branch] gh/anshul-si/21/base -> origin/gh/anshul-si/21/base 2025-08-14T21:24:12.6429993Z * [new branch] gh/anshul-si/21/head -> origin/gh/anshul-si/21/head 2025-08-14T21:24:12.6430131Z * [new branch] gh/anshul-si/21/orig -> origin/gh/anshul-si/21/orig 2025-08-14T21:24:12.6431309Z * [new branch] gh/anshul-si/22/base -> origin/gh/anshul-si/22/base 2025-08-14T21:24:12.6431863Z * [new branch] gh/anshul-si/22/head -> origin/gh/anshul-si/22/head 2025-08-14T21:24:12.6432035Z * [new branch] gh/anshul-si/22/orig -> origin/gh/anshul-si/22/orig 2025-08-14T21:24:12.6435749Z * [new branch] gh/anshul-si/23/base -> origin/gh/anshul-si/23/base 2025-08-14T21:24:12.6436051Z * [new branch] gh/anshul-si/23/head -> origin/gh/anshul-si/23/head 2025-08-14T21:24:12.6436196Z * [new branch] gh/anshul-si/23/orig -> origin/gh/anshul-si/23/orig 2025-08-14T21:24:12.6436340Z * [new branch] gh/anshul-si/24/base -> origin/gh/anshul-si/24/base 2025-08-14T21:24:12.6436471Z * [new branch] gh/anshul-si/24/head -> origin/gh/anshul-si/24/head 2025-08-14T21:24:12.6436602Z * [new branch] gh/anshul-si/24/orig -> origin/gh/anshul-si/24/orig 2025-08-14T21:24:12.6441199Z * [new branch] gh/anshul-si/25/base -> origin/gh/anshul-si/25/base 2025-08-14T21:24:12.6441404Z * [new branch] gh/anshul-si/25/head -> origin/gh/anshul-si/25/head 2025-08-14T21:24:12.6441557Z * [new branch] gh/anshul-si/25/orig -> origin/gh/anshul-si/25/orig 2025-08-14T21:24:12.6441721Z * [new branch] gh/anshul-si/26/base -> origin/gh/anshul-si/26/base 2025-08-14T21:24:12.6441875Z * [new branch] gh/anshul-si/26/head -> origin/gh/anshul-si/26/head 2025-08-14T21:24:12.6442034Z * [new branch] gh/anshul-si/26/orig -> origin/gh/anshul-si/26/orig 2025-08-14T21:24:12.6442180Z * [new branch] gh/anshul-si/27/base -> origin/gh/anshul-si/27/base 2025-08-14T21:24:12.6442331Z * [new branch] gh/anshul-si/27/head -> origin/gh/anshul-si/27/head 2025-08-14T21:24:12.6443304Z * [new branch] gh/anshul-si/27/orig -> origin/gh/anshul-si/27/orig 2025-08-14T21:24:12.6444315Z * [new branch] gh/anshul-si/3/base -> origin/gh/anshul-si/3/base 2025-08-14T21:24:12.6446063Z * [new branch] gh/anshul-si/3/head -> origin/gh/anshul-si/3/head 2025-08-14T21:24:12.6446555Z * [new branch] gh/anshul-si/4/base -> origin/gh/anshul-si/4/base 2025-08-14T21:24:12.6446726Z * [new branch] gh/anshul-si/4/head -> origin/gh/anshul-si/4/head 2025-08-14T21:24:12.6454422Z * [new branch] gh/anshul-si/5/base -> origin/gh/anshul-si/5/base 2025-08-14T21:24:12.6454591Z * [new branch] gh/anshul-si/5/head -> origin/gh/anshul-si/5/head 2025-08-14T21:24:12.6454732Z * [new branch] gh/anshul-si/6/base -> origin/gh/anshul-si/6/base 2025-08-14T21:24:12.6454859Z * [new branch] gh/anshul-si/6/head -> origin/gh/anshul-si/6/head 2025-08-14T21:24:12.6455000Z * [new branch] gh/anshul-si/6/orig -> origin/gh/anshul-si/6/orig 2025-08-14T21:24:12.6455156Z * [new branch] gh/anshul-si/7/base -> origin/gh/anshul-si/7/base 2025-08-14T21:24:12.6455284Z * [new branch] gh/anshul-si/7/head -> origin/gh/anshul-si/7/head 2025-08-14T21:24:12.6455418Z * [new branch] gh/anshul-si/7/orig -> origin/gh/anshul-si/7/orig 2025-08-14T21:24:12.6455542Z * [new branch] gh/anshul-si/8/base -> origin/gh/anshul-si/8/base 2025-08-14T21:24:12.6455668Z * [new branch] gh/anshul-si/8/head -> origin/gh/anshul-si/8/head 2025-08-14T21:24:12.6455806Z * [new branch] gh/anshul-si/8/orig -> origin/gh/anshul-si/8/orig 2025-08-14T21:24:12.6456100Z * [new branch] gh/anshul-si/9/base -> origin/gh/anshul-si/9/base 2025-08-14T21:24:12.6461588Z * [new branch] gh/anshul-si/9/head -> origin/gh/anshul-si/9/head 2025-08-14T21:24:12.6461921Z * [new branch] gh/anshul-si/9/orig -> origin/gh/anshul-si/9/orig 2025-08-14T21:24:12.6462419Z * [new branch] gh/aorenste/132/base -> origin/gh/aorenste/132/base 2025-08-14T21:24:12.6462657Z * [new branch] gh/aorenste/132/head -> origin/gh/aorenste/132/head 2025-08-14T21:24:12.6462926Z * [new branch] gh/aorenste/235/base -> origin/gh/aorenste/235/base 2025-08-14T21:24:12.6463570Z * [new branch] gh/aorenste/235/head -> origin/gh/aorenste/235/head 2025-08-14T21:24:12.6463745Z * [new branch] gh/aorenste/235/orig -> origin/gh/aorenste/235/orig 2025-08-14T21:24:12.6463891Z * [new branch] gh/aorenste/236/base -> origin/gh/aorenste/236/base 2025-08-14T21:24:12.6464029Z * [new branch] gh/aorenste/236/head -> origin/gh/aorenste/236/head 2025-08-14T21:24:12.6464168Z * [new branch] gh/aorenste/236/orig -> origin/gh/aorenste/236/orig 2025-08-14T21:24:12.6464370Z * [new branch] gh/aorenste/237/base -> origin/gh/aorenste/237/base 2025-08-14T21:24:12.6465035Z * [new branch] gh/aorenste/237/head -> origin/gh/aorenste/237/head 2025-08-14T21:24:12.6466412Z * [new branch] gh/aorenste/237/orig -> origin/gh/aorenste/237/orig 2025-08-14T21:24:12.6466725Z * [new branch] gh/aorenste/238/base -> origin/gh/aorenste/238/base 2025-08-14T21:24:12.6467493Z * [new branch] gh/aorenste/238/head -> origin/gh/aorenste/238/head 2025-08-14T21:24:12.6467999Z * [new branch] gh/aorenste/238/orig -> origin/gh/aorenste/238/orig 2025-08-14T21:24:12.6471634Z * [new branch] gh/bdhirsh/650/base -> origin/gh/bdhirsh/650/base 2025-08-14T21:24:12.6472152Z * [new branch] gh/bdhirsh/650/head -> origin/gh/bdhirsh/650/head 2025-08-14T21:24:12.6472327Z * [new branch] gh/bdhirsh/650/orig -> origin/gh/bdhirsh/650/orig 2025-08-14T21:24:12.6472599Z * [new branch] gh/bdhirsh/656/base -> origin/gh/bdhirsh/656/base 2025-08-14T21:24:12.6472763Z * [new branch] gh/bdhirsh/656/head -> origin/gh/bdhirsh/656/head 2025-08-14T21:24:12.6473389Z * [new branch] gh/bdhirsh/657/base -> origin/gh/bdhirsh/657/base 2025-08-14T21:24:12.6474306Z * [new branch] gh/bdhirsh/657/head -> origin/gh/bdhirsh/657/head 2025-08-14T21:24:12.6474756Z * [new branch] gh/bdhirsh/659/base -> origin/gh/bdhirsh/659/base 2025-08-14T21:24:12.6478699Z * [new branch] gh/bdhirsh/659/head -> origin/gh/bdhirsh/659/head 2025-08-14T21:24:12.6478883Z * [new branch] gh/bdhirsh/659/orig -> origin/gh/bdhirsh/659/orig 2025-08-14T21:24:12.6479018Z * [new branch] gh/bdhirsh/663/base -> origin/gh/bdhirsh/663/base 2025-08-14T21:24:12.6479155Z * [new branch] gh/bdhirsh/663/head -> origin/gh/bdhirsh/663/head 2025-08-14T21:24:12.6479311Z * [new branch] gh/bdhirsh/663/orig -> origin/gh/bdhirsh/663/orig 2025-08-14T21:24:12.6479468Z * [new branch] gh/bdhirsh/665/base -> origin/gh/bdhirsh/665/base 2025-08-14T21:24:12.6480118Z * [new branch] gh/bdhirsh/665/head -> origin/gh/bdhirsh/665/head 2025-08-14T21:24:12.6480684Z * [new branch] gh/bdhirsh/665/orig -> origin/gh/bdhirsh/665/orig 2025-08-14T21:24:12.6481936Z * [new branch] gh/bdhirsh/666/base -> origin/gh/bdhirsh/666/base 2025-08-14T21:24:12.6482391Z * [new branch] gh/bdhirsh/666/head -> origin/gh/bdhirsh/666/head 2025-08-14T21:24:12.6482953Z * [new branch] gh/bdhirsh/666/orig -> origin/gh/bdhirsh/666/orig 2025-08-14T21:24:12.6484447Z * [new branch] gh/benjaminglass1/79/base -> origin/gh/benjaminglass1/79/base 2025-08-14T21:24:12.6484785Z * [new branch] gh/benjaminglass1/79/head -> origin/gh/benjaminglass1/79/head 2025-08-14T21:24:12.6485651Z * [new branch] gh/benjaminglass1/79/orig -> origin/gh/benjaminglass1/79/orig 2025-08-14T21:24:12.6486811Z * [new branch] gh/benjaminglass1/86/base -> origin/gh/benjaminglass1/86/base 2025-08-14T21:24:12.6487177Z * [new branch] gh/benjaminglass1/86/head -> origin/gh/benjaminglass1/86/head 2025-08-14T21:24:12.6488189Z * [new branch] gh/benjaminglass1/86/orig -> origin/gh/benjaminglass1/86/orig 2025-08-14T21:24:12.6488951Z * [new branch] gh/benjaminglass1/89/base -> origin/gh/benjaminglass1/89/base 2025-08-14T21:24:12.6489475Z * [new branch] gh/benjaminglass1/89/head -> origin/gh/benjaminglass1/89/head 2025-08-14T21:24:12.6490443Z * [new branch] gh/benjaminglass1/89/orig -> origin/gh/benjaminglass1/89/orig 2025-08-14T21:24:12.6491233Z * [new branch] gh/benjaminglass1/91/base -> origin/gh/benjaminglass1/91/base 2025-08-14T21:24:12.6491770Z * [new branch] gh/benjaminglass1/91/head -> origin/gh/benjaminglass1/91/head 2025-08-14T21:24:12.6492721Z * [new branch] gh/benjaminglass1/91/orig -> origin/gh/benjaminglass1/91/orig 2025-08-14T21:24:12.6493757Z * [new branch] gh/benjaminglass1/93/base -> origin/gh/benjaminglass1/93/base 2025-08-14T21:24:12.6494031Z * [new branch] gh/benjaminglass1/93/head -> origin/gh/benjaminglass1/93/head 2025-08-14T21:24:12.6495166Z * [new branch] gh/benjaminglass1/93/orig -> origin/gh/benjaminglass1/93/orig 2025-08-14T21:24:12.6496082Z * [new branch] gh/benjaminglass1/94/base -> origin/gh/benjaminglass1/94/base 2025-08-14T21:24:12.6496355Z * [new branch] gh/benjaminglass1/94/head -> origin/gh/benjaminglass1/94/head 2025-08-14T21:24:12.6497479Z * [new branch] gh/benjaminglass1/94/orig -> origin/gh/benjaminglass1/94/orig 2025-08-14T21:24:12.6498316Z * [new branch] gh/benjaminglass1/95/base -> origin/gh/benjaminglass1/95/base 2025-08-14T21:24:12.6498712Z * [new branch] gh/benjaminglass1/95/head -> origin/gh/benjaminglass1/95/head 2025-08-14T21:24:12.6501673Z * [new branch] gh/benjaminglass1/95/orig -> origin/gh/benjaminglass1/95/orig 2025-08-14T21:24:12.6505872Z * [new branch] gh/benjaminglass1/96/base -> origin/gh/benjaminglass1/96/base 2025-08-14T21:24:12.6506601Z * [new branch] gh/benjaminglass1/96/head -> origin/gh/benjaminglass1/96/head 2025-08-14T21:24:12.6506816Z * [new branch] gh/benjaminglass1/96/orig -> origin/gh/benjaminglass1/96/orig 2025-08-14T21:24:12.6506990Z * [new branch] gh/benjaminglass1/97/base -> origin/gh/benjaminglass1/97/base 2025-08-14T21:24:12.6507160Z * [new branch] gh/benjaminglass1/97/head -> origin/gh/benjaminglass1/97/head 2025-08-14T21:24:12.6507358Z * [new branch] gh/benjaminglass1/97/orig -> origin/gh/benjaminglass1/97/orig 2025-08-14T21:24:12.6507543Z * [new branch] gh/benjaminglass1/98/base -> origin/gh/benjaminglass1/98/base 2025-08-14T21:24:12.6507713Z * [new branch] gh/benjaminglass1/98/head -> origin/gh/benjaminglass1/98/head 2025-08-14T21:24:12.6507880Z * [new branch] gh/benjaminglass1/98/orig -> origin/gh/benjaminglass1/98/orig 2025-08-14T21:24:12.6508050Z * [new branch] gh/bobrenjc93/478/base -> origin/gh/bobrenjc93/478/base 2025-08-14T21:24:12.6508209Z * [new branch] gh/bobrenjc93/478/head -> origin/gh/bobrenjc93/478/head 2025-08-14T21:24:12.6510596Z * [new branch] gh/bobrenjc93/478/orig -> origin/gh/bobrenjc93/478/orig 2025-08-14T21:24:12.6510768Z * [new branch] gh/bobrenjc93/514/base -> origin/gh/bobrenjc93/514/base 2025-08-14T21:24:12.6510933Z * [new branch] gh/bobrenjc93/514/head -> origin/gh/bobrenjc93/514/head 2025-08-14T21:24:12.6511097Z * [new branch] gh/bobrenjc93/514/orig -> origin/gh/bobrenjc93/514/orig 2025-08-14T21:24:12.6512366Z * [new branch] gh/bobrenjc93/521/base -> origin/gh/bobrenjc93/521/base 2025-08-14T21:24:12.6512547Z * [new branch] gh/bobrenjc93/521/head -> origin/gh/bobrenjc93/521/head 2025-08-14T21:24:12.6514391Z * [new branch] gh/bobrenjc93/521/orig -> origin/gh/bobrenjc93/521/orig 2025-08-14T21:24:12.6514738Z * [new branch] gh/bobrenjc93/522/base -> origin/gh/bobrenjc93/522/base 2025-08-14T21:24:12.6515003Z * [new branch] gh/bobrenjc93/522/head -> origin/gh/bobrenjc93/522/head 2025-08-14T21:24:12.6517076Z * [new branch] gh/bobrenjc93/522/orig -> origin/gh/bobrenjc93/522/orig 2025-08-14T21:24:12.6517436Z * [new branch] gh/bobrenjc93/525/base -> origin/gh/bobrenjc93/525/base 2025-08-14T21:24:12.6517688Z * [new branch] gh/bobrenjc93/525/head -> origin/gh/bobrenjc93/525/head 2025-08-14T21:24:12.6518092Z * [new branch] gh/bobrenjc93/525/orig -> origin/gh/bobrenjc93/525/orig 2025-08-14T21:24:12.6519329Z * [new branch] gh/bobrenjc93/526/base -> origin/gh/bobrenjc93/526/base 2025-08-14T21:24:12.6520056Z * [new branch] gh/bobrenjc93/526/head -> origin/gh/bobrenjc93/526/head 2025-08-14T21:24:12.6520499Z * [new branch] gh/bobrenjc93/526/orig -> origin/gh/bobrenjc93/526/orig 2025-08-14T21:24:12.6521707Z * [new branch] gh/bobrenjc93/527/base -> origin/gh/bobrenjc93/527/base 2025-08-14T21:24:12.6521988Z * [new branch] gh/bobrenjc93/527/head -> origin/gh/bobrenjc93/527/head 2025-08-14T21:24:12.6523119Z * [new branch] gh/bobrenjc93/527/orig -> origin/gh/bobrenjc93/527/orig 2025-08-14T21:24:12.6523707Z * [new branch] gh/bobrenjc93/528/base -> origin/gh/bobrenjc93/528/base 2025-08-14T21:24:12.6524661Z * [new branch] gh/bobrenjc93/528/head -> origin/gh/bobrenjc93/528/head 2025-08-14T21:24:12.6525241Z * [new branch] gh/bobrenjc93/528/orig -> origin/gh/bobrenjc93/528/orig 2025-08-14T21:24:12.6526657Z * [new branch] gh/bobrenjc93/529/base -> origin/gh/bobrenjc93/529/base 2025-08-14T21:24:12.6527058Z * [new branch] gh/bobrenjc93/529/head -> origin/gh/bobrenjc93/529/head 2025-08-14T21:24:12.6528158Z * [new branch] gh/bobrenjc93/529/orig -> origin/gh/bobrenjc93/529/orig 2025-08-14T21:24:12.6528663Z * [new branch] gh/bobrenjc93/534/base -> origin/gh/bobrenjc93/534/base 2025-08-14T21:24:12.6529447Z * [new branch] gh/bobrenjc93/534/head -> origin/gh/bobrenjc93/534/head 2025-08-14T21:24:12.6529981Z * [new branch] gh/bobrenjc93/534/orig -> origin/gh/bobrenjc93/534/orig 2025-08-14T21:24:12.6531245Z * [new branch] gh/bobrenjc93/535/base -> origin/gh/bobrenjc93/535/base 2025-08-14T21:24:12.6531539Z * [new branch] gh/bobrenjc93/535/head -> origin/gh/bobrenjc93/535/head 2025-08-14T21:24:12.6532568Z * [new branch] gh/bobrenjc93/535/orig -> origin/gh/bobrenjc93/535/orig 2025-08-14T21:24:12.6533350Z * [new branch] gh/bobrenjc93/536/base -> origin/gh/bobrenjc93/536/base 2025-08-14T21:24:12.6533963Z * [new branch] gh/bobrenjc93/536/head -> origin/gh/bobrenjc93/536/head 2025-08-14T21:24:12.6534916Z * [new branch] gh/bobrenjc93/536/orig -> origin/gh/bobrenjc93/536/orig 2025-08-14T21:24:12.6535908Z * [new branch] gh/bobrenjc93/537/base -> origin/gh/bobrenjc93/537/base 2025-08-14T21:24:12.6536288Z * [new branch] gh/bobrenjc93/537/head -> origin/gh/bobrenjc93/537/head 2025-08-14T21:24:12.6537314Z * [new branch] gh/bobrenjc93/537/orig -> origin/gh/bobrenjc93/537/orig 2025-08-14T21:24:12.6537906Z * [new branch] gh/bobrenjc93/538/base -> origin/gh/bobrenjc93/538/base 2025-08-14T21:24:12.6538786Z * [new branch] gh/bobrenjc93/538/head -> origin/gh/bobrenjc93/538/head 2025-08-14T21:24:12.6539365Z * [new branch] gh/bobrenjc93/538/orig -> origin/gh/bobrenjc93/538/orig 2025-08-14T21:24:12.6540876Z * [new branch] gh/bobrenjc93/539/base -> origin/gh/bobrenjc93/539/base 2025-08-14T21:24:12.6541183Z * [new branch] gh/bobrenjc93/539/head -> origin/gh/bobrenjc93/539/head 2025-08-14T21:24:12.6542280Z * [new branch] gh/bobrenjc93/539/orig -> origin/gh/bobrenjc93/539/orig 2025-08-14T21:24:12.6543251Z * [new branch] gh/bobrenjc93/540/base -> origin/gh/bobrenjc93/540/base 2025-08-14T21:24:12.6543615Z * [new branch] gh/bobrenjc93/540/head -> origin/gh/bobrenjc93/540/head 2025-08-14T21:24:12.6544654Z * [new branch] gh/bobrenjc93/540/orig -> origin/gh/bobrenjc93/540/orig 2025-08-14T21:24:12.6545963Z * [new branch] gh/bobrenjc93/541/base -> origin/gh/bobrenjc93/541/base 2025-08-14T21:24:12.6546352Z * [new branch] gh/bobrenjc93/541/head -> origin/gh/bobrenjc93/541/head 2025-08-14T21:24:12.6547300Z * [new branch] gh/bobrenjc93/541/orig -> origin/gh/bobrenjc93/541/orig 2025-08-14T21:24:12.6547968Z * [new branch] gh/bobrenjc93/542/base -> origin/gh/bobrenjc93/542/base 2025-08-14T21:24:12.6548564Z * [new branch] gh/bobrenjc93/542/head -> origin/gh/bobrenjc93/542/head 2025-08-14T21:24:12.6549372Z * [new branch] gh/bobrenjc93/542/orig -> origin/gh/bobrenjc93/542/orig 2025-08-14T21:24:12.6550511Z * [new branch] gh/bobrenjc93/543/base -> origin/gh/bobrenjc93/543/base 2025-08-14T21:24:12.6550765Z * [new branch] gh/bobrenjc93/543/head -> origin/gh/bobrenjc93/543/head 2025-08-14T21:24:12.6551872Z * [new branch] gh/bobrenjc93/543/orig -> origin/gh/bobrenjc93/543/orig 2025-08-14T21:24:12.6552495Z * [new branch] gh/bobrenjc93/544/base -> origin/gh/bobrenjc93/544/base 2025-08-14T21:24:12.6553041Z * [new branch] gh/bobrenjc93/544/head -> origin/gh/bobrenjc93/544/head 2025-08-14T21:24:12.6553939Z * [new branch] gh/bobrenjc93/544/orig -> origin/gh/bobrenjc93/544/orig 2025-08-14T21:24:12.6554548Z * [new branch] gh/bobrenjc93/545/base -> origin/gh/bobrenjc93/545/base 2025-08-14T21:24:12.6555525Z * [new branch] gh/bobrenjc93/545/head -> origin/gh/bobrenjc93/545/head 2025-08-14T21:24:12.6555992Z * [new branch] gh/bobrenjc93/545/orig -> origin/gh/bobrenjc93/545/orig 2025-08-14T21:24:12.6557209Z * [new branch] gh/bobrenjc93/546/base -> origin/gh/bobrenjc93/546/base 2025-08-14T21:24:12.6557781Z * [new branch] gh/bobrenjc93/546/head -> origin/gh/bobrenjc93/546/head 2025-08-14T21:24:12.6559372Z * [new branch] gh/bobrenjc93/546/orig -> origin/gh/bobrenjc93/546/orig 2025-08-14T21:24:12.6559763Z * [new branch] gh/bobrenjc93/547/base -> origin/gh/bobrenjc93/547/base 2025-08-14T21:24:12.6561337Z * [new branch] gh/bobrenjc93/547/head -> origin/gh/bobrenjc93/547/head 2025-08-14T21:24:12.6561559Z * [new branch] gh/bobrenjc93/547/orig -> origin/gh/bobrenjc93/547/orig 2025-08-14T21:24:12.6561711Z * [new branch] gh/bobrenjc93/548/base -> origin/gh/bobrenjc93/548/base 2025-08-14T21:24:12.6562778Z * [new branch] gh/bobrenjc93/548/head -> origin/gh/bobrenjc93/548/head 2025-08-14T21:24:12.6563054Z * [new branch] gh/bobrenjc93/548/orig -> origin/gh/bobrenjc93/548/orig 2025-08-14T21:24:12.6564249Z * [new branch] gh/bobrenjc93/549/base -> origin/gh/bobrenjc93/549/base 2025-08-14T21:24:12.6564659Z * [new branch] gh/bobrenjc93/549/head -> origin/gh/bobrenjc93/549/head 2025-08-14T21:24:12.6565856Z * [new branch] gh/bobrenjc93/549/orig -> origin/gh/bobrenjc93/549/orig 2025-08-14T21:24:12.6569634Z * [new branch] gh/briancoutinho/2/base -> origin/gh/briancoutinho/2/base 2025-08-14T21:24:12.6569797Z * [new branch] gh/briancoutinho/2/head -> origin/gh/briancoutinho/2/head 2025-08-14T21:24:12.6570045Z * [new branch] gh/c00w/23/base -> origin/gh/c00w/23/base 2025-08-14T21:24:12.6572822Z * [new branch] gh/c00w/23/head -> origin/gh/c00w/23/head 2025-08-14T21:24:12.6572988Z * [new branch] gh/c00w/38/base -> origin/gh/c00w/38/base 2025-08-14T21:24:12.6573127Z * [new branch] gh/c00w/38/head -> origin/gh/c00w/38/head 2025-08-14T21:24:12.6573252Z * [new branch] gh/c00w/38/orig -> origin/gh/c00w/38/orig 2025-08-14T21:24:12.6573377Z * [new branch] gh/c00w/48/base -> origin/gh/c00w/48/base 2025-08-14T21:24:12.6573840Z * [new branch] gh/c00w/48/head -> origin/gh/c00w/48/head 2025-08-14T21:24:12.6574000Z * [new branch] gh/c00w/48/orig -> origin/gh/c00w/48/orig 2025-08-14T21:24:12.6578718Z * [new branch] gh/c00w/50/base -> origin/gh/c00w/50/base 2025-08-14T21:24:12.6578893Z * [new branch] gh/c00w/50/head -> origin/gh/c00w/50/head 2025-08-14T21:24:12.6579019Z * [new branch] gh/c00w/50/orig -> origin/gh/c00w/50/orig 2025-08-14T21:24:12.6579152Z * [new branch] gh/c00w/51/base -> origin/gh/c00w/51/base 2025-08-14T21:24:12.6584127Z * [new branch] gh/c00w/51/head -> origin/gh/c00w/51/head 2025-08-14T21:24:12.6584297Z * [new branch] gh/c00w/51/orig -> origin/gh/c00w/51/orig 2025-08-14T21:24:12.6584414Z * [new branch] gh/c00w/52/base -> origin/gh/c00w/52/base 2025-08-14T21:24:12.6584690Z * [new branch] gh/c00w/52/head -> origin/gh/c00w/52/head 2025-08-14T21:24:12.6584827Z * [new branch] gh/c00w/52/orig -> origin/gh/c00w/52/orig 2025-08-14T21:24:12.6584939Z * [new branch] gh/c00w/53/base -> origin/gh/c00w/53/base 2025-08-14T21:24:12.6585063Z * [new branch] gh/c00w/53/head -> origin/gh/c00w/53/head 2025-08-14T21:24:12.6589266Z * [new branch] gh/c00w/53/orig -> origin/gh/c00w/53/orig 2025-08-14T21:24:12.6589446Z * [new branch] gh/c00w/54/base -> origin/gh/c00w/54/base 2025-08-14T21:24:12.6589583Z * [new branch] gh/c00w/54/head -> origin/gh/c00w/54/head 2025-08-14T21:24:12.6589715Z * [new branch] gh/c00w/54/orig -> origin/gh/c00w/54/orig 2025-08-14T21:24:12.6589879Z * [new branch] gh/chenmillie/1/base -> origin/gh/chenmillie/1/base 2025-08-14T21:24:12.6590036Z * [new branch] gh/chenmillie/1/head -> origin/gh/chenmillie/1/head 2025-08-14T21:24:12.6590181Z * [new branch] gh/chenmillie/1/orig -> origin/gh/chenmillie/1/orig 2025-08-14T21:24:12.6596026Z * [new branch] gh/clee2000/1/base -> origin/gh/clee2000/1/base 2025-08-14T21:24:12.6596199Z * [new branch] gh/clee2000/1/head -> origin/gh/clee2000/1/head 2025-08-14T21:24:12.6596336Z * [new branch] gh/clee2000/1/orig -> origin/gh/clee2000/1/orig 2025-08-14T21:24:12.6596497Z * [new branch] gh/coconutruben/1/base -> origin/gh/coconutruben/1/base 2025-08-14T21:24:12.6596646Z * [new branch] gh/coconutruben/1/head -> origin/gh/coconutruben/1/head 2025-08-14T21:24:12.6596808Z * [new branch] gh/coconutruben/11/base -> origin/gh/coconutruben/11/base 2025-08-14T21:24:12.6597397Z * [new branch] gh/coconutruben/11/head -> origin/gh/coconutruben/11/head 2025-08-14T21:24:12.6597585Z * [new branch] gh/coconutruben/11/orig -> origin/gh/coconutruben/11/orig 2025-08-14T21:24:12.6597890Z * [new branch] gh/coconutruben/12/base -> origin/gh/coconutruben/12/base 2025-08-14T21:24:12.6598037Z * [new branch] gh/coconutruben/12/head -> origin/gh/coconutruben/12/head 2025-08-14T21:24:12.6598191Z * [new branch] gh/coconutruben/12/orig -> origin/gh/coconutruben/12/orig 2025-08-14T21:24:12.6599872Z * [new branch] gh/coconutruben/13/base -> origin/gh/coconutruben/13/base 2025-08-14T21:24:12.6600073Z * [new branch] gh/coconutruben/13/head -> origin/gh/coconutruben/13/head 2025-08-14T21:24:12.6600545Z * [new branch] gh/coconutruben/13/orig -> origin/gh/coconutruben/13/orig 2025-08-14T21:24:12.6601958Z * [new branch] gh/coconutruben/14/base -> origin/gh/coconutruben/14/base 2025-08-14T21:24:12.6602288Z * [new branch] gh/coconutruben/14/head -> origin/gh/coconutruben/14/head 2025-08-14T21:24:12.6603322Z * [new branch] gh/coconutruben/14/orig -> origin/gh/coconutruben/14/orig 2025-08-14T21:24:12.6604564Z * [new branch] gh/coconutruben/15/base -> origin/gh/coconutruben/15/base 2025-08-14T21:24:12.6605281Z * [new branch] gh/coconutruben/15/head -> origin/gh/coconutruben/15/head 2025-08-14T21:24:12.6606188Z * [new branch] gh/coconutruben/15/orig -> origin/gh/coconutruben/15/orig 2025-08-14T21:24:12.6610345Z * [new branch] gh/coconutruben/16/base -> origin/gh/coconutruben/16/base 2025-08-14T21:24:12.6610708Z * [new branch] gh/coconutruben/16/head -> origin/gh/coconutruben/16/head 2025-08-14T21:24:12.6610964Z * [new branch] gh/coconutruben/16/orig -> origin/gh/coconutruben/16/orig 2025-08-14T21:24:12.6611144Z * [new branch] gh/coconutruben/17/base -> origin/gh/coconutruben/17/base 2025-08-14T21:24:12.6611573Z * [new branch] gh/coconutruben/17/head -> origin/gh/coconutruben/17/head 2025-08-14T21:24:12.6612185Z * [new branch] gh/coconutruben/17/orig -> origin/gh/coconutruben/17/orig 2025-08-14T21:24:12.6612390Z * [new branch] gh/coconutruben/18/base -> origin/gh/coconutruben/18/base 2025-08-14T21:24:12.6613003Z * [new branch] gh/coconutruben/18/head -> origin/gh/coconutruben/18/head 2025-08-14T21:24:12.6613668Z * [new branch] gh/coconutruben/18/orig -> origin/gh/coconutruben/18/orig 2025-08-14T21:24:12.6618450Z * [new branch] gh/coconutruben/19/base -> origin/gh/coconutruben/19/base 2025-08-14T21:24:12.6618642Z * [new branch] gh/coconutruben/19/head -> origin/gh/coconutruben/19/head 2025-08-14T21:24:12.6618813Z * [new branch] gh/coconutruben/19/orig -> origin/gh/coconutruben/19/orig 2025-08-14T21:24:12.6619000Z * [new branch] gh/coconutruben/20/base -> origin/gh/coconutruben/20/base 2025-08-14T21:24:12.6619160Z * [new branch] gh/coconutruben/20/head -> origin/gh/coconutruben/20/head 2025-08-14T21:24:12.6619326Z * [new branch] gh/coconutruben/20/orig -> origin/gh/coconutruben/20/orig 2025-08-14T21:24:12.6619906Z * [new branch] gh/coconutruben/21/base -> origin/gh/coconutruben/21/base 2025-08-14T21:24:12.6620437Z * [new branch] gh/coconutruben/21/head -> origin/gh/coconutruben/21/head 2025-08-14T21:24:12.6621347Z * [new branch] gh/coconutruben/21/orig -> origin/gh/coconutruben/21/orig 2025-08-14T21:24:12.6622775Z * [new branch] gh/coconutruben/22/base -> origin/gh/coconutruben/22/base 2025-08-14T21:24:12.6622958Z * [new branch] gh/coconutruben/22/head -> origin/gh/coconutruben/22/head 2025-08-14T21:24:12.6623882Z * [new branch] gh/coconutruben/22/orig -> origin/gh/coconutruben/22/orig 2025-08-14T21:24:12.6627961Z * [new branch] gh/coconutruben/23/base -> origin/gh/coconutruben/23/base 2025-08-14T21:24:12.6628327Z * [new branch] gh/coconutruben/23/head -> origin/gh/coconutruben/23/head 2025-08-14T21:24:12.6628476Z * [new branch] gh/coconutruben/23/orig -> origin/gh/coconutruben/23/orig 2025-08-14T21:24:12.6628626Z * [new branch] gh/coconutruben/24/base -> origin/gh/coconutruben/24/base 2025-08-14T21:24:12.6628767Z * [new branch] gh/coconutruben/24/head -> origin/gh/coconutruben/24/head 2025-08-14T21:24:12.6628921Z * [new branch] gh/coconutruben/24/orig -> origin/gh/coconutruben/24/orig 2025-08-14T21:24:12.6630427Z * [new branch] gh/coconutruben/25/base -> origin/gh/coconutruben/25/base 2025-08-14T21:24:12.6631228Z * [new branch] gh/coconutruben/25/head -> origin/gh/coconutruben/25/head 2025-08-14T21:24:12.6634384Z * [new branch] gh/coconutruben/25/orig -> origin/gh/coconutruben/25/orig 2025-08-14T21:24:12.6634584Z * [new branch] gh/coconutruben/26/base -> origin/gh/coconutruben/26/base 2025-08-14T21:24:12.6634745Z * [new branch] gh/coconutruben/26/head -> origin/gh/coconutruben/26/head 2025-08-14T21:24:12.6634906Z * [new branch] gh/coconutruben/26/orig -> origin/gh/coconutruben/26/orig 2025-08-14T21:24:12.6635543Z * [new branch] gh/coconutruben/27/base -> origin/gh/coconutruben/27/base 2025-08-14T21:24:12.6636595Z * [new branch] gh/coconutruben/27/head -> origin/gh/coconutruben/27/head 2025-08-14T21:24:12.6637096Z * [new branch] gh/coconutruben/27/orig -> origin/gh/coconutruben/27/orig 2025-08-14T21:24:12.6644683Z * [new branch] gh/codingwithsurya/10/base -> origin/gh/codingwithsurya/10/base 2025-08-14T21:24:12.6644944Z * [new branch] gh/codingwithsurya/10/head -> origin/gh/codingwithsurya/10/head 2025-08-14T21:24:12.6645365Z * [new branch] gh/codingwithsurya/10/orig -> origin/gh/codingwithsurya/10/orig 2025-08-14T21:24:12.6645777Z * [new branch] gh/codingwithsurya/11/base -> origin/gh/codingwithsurya/11/base 2025-08-14T21:24:12.6646212Z * [new branch] gh/codingwithsurya/11/head -> origin/gh/codingwithsurya/11/head 2025-08-14T21:24:12.6646766Z * [new branch] gh/codingwithsurya/11/orig -> origin/gh/codingwithsurya/11/orig 2025-08-14T21:24:12.6654037Z * [new branch] gh/codingwithsurya/12/base -> origin/gh/codingwithsurya/12/base 2025-08-14T21:24:12.6656131Z * [new branch] gh/codingwithsurya/12/head -> origin/gh/codingwithsurya/12/head 2025-08-14T21:24:12.6656437Z * [new branch] gh/codingwithsurya/12/orig -> origin/gh/codingwithsurya/12/orig 2025-08-14T21:24:12.6659994Z * [new branch] gh/codingwithsurya/13/base -> origin/gh/codingwithsurya/13/base 2025-08-14T21:24:12.6665262Z * [new branch] gh/codingwithsurya/13/head -> origin/gh/codingwithsurya/13/head 2025-08-14T21:24:12.6667243Z * [new branch] gh/codingwithsurya/13/orig -> origin/gh/codingwithsurya/13/orig 2025-08-14T21:24:12.6667550Z * [new branch] gh/codingwithsurya/14/base -> origin/gh/codingwithsurya/14/base 2025-08-14T21:24:12.6673386Z * [new branch] gh/codingwithsurya/14/head -> origin/gh/codingwithsurya/14/head 2025-08-14T21:24:12.6678459Z * [new branch] gh/codingwithsurya/14/orig -> origin/gh/codingwithsurya/14/orig 2025-08-14T21:24:12.6678690Z * [new branch] gh/codingwithsurya/15/base -> origin/gh/codingwithsurya/15/base 2025-08-14T21:24:12.6678865Z * [new branch] gh/codingwithsurya/15/head -> origin/gh/codingwithsurya/15/head 2025-08-14T21:24:12.6679030Z * [new branch] gh/codingwithsurya/15/orig -> origin/gh/codingwithsurya/15/orig 2025-08-14T21:24:12.6679208Z * [new branch] gh/codingwithsurya/16/base -> origin/gh/codingwithsurya/16/base 2025-08-14T21:24:12.6679595Z * [new branch] gh/codingwithsurya/16/head -> origin/gh/codingwithsurya/16/head 2025-08-14T21:24:12.6679761Z * [new branch] gh/codingwithsurya/16/orig -> origin/gh/codingwithsurya/16/orig 2025-08-14T21:24:12.6679920Z * [new branch] gh/codingwithsurya/17/base -> origin/gh/codingwithsurya/17/base 2025-08-14T21:24:12.6680091Z * [new branch] gh/codingwithsurya/17/head -> origin/gh/codingwithsurya/17/head 2025-08-14T21:24:12.6680257Z * [new branch] gh/codingwithsurya/17/orig -> origin/gh/codingwithsurya/17/orig 2025-08-14T21:24:12.6680423Z * [new branch] gh/codingwithsurya/18/base -> origin/gh/codingwithsurya/18/base 2025-08-14T21:24:12.6680583Z * [new branch] gh/codingwithsurya/18/head -> origin/gh/codingwithsurya/18/head 2025-08-14T21:24:12.6680741Z * [new branch] gh/codingwithsurya/18/orig -> origin/gh/codingwithsurya/18/orig 2025-08-14T21:24:12.6680912Z * [new branch] gh/codingwithsurya/19/base -> origin/gh/codingwithsurya/19/base 2025-08-14T21:24:12.6681078Z * [new branch] gh/codingwithsurya/19/head -> origin/gh/codingwithsurya/19/head 2025-08-14T21:24:12.6681241Z * [new branch] gh/codingwithsurya/19/orig -> origin/gh/codingwithsurya/19/orig 2025-08-14T21:24:12.6681397Z * [new branch] gh/codingwithsurya/20/base -> origin/gh/codingwithsurya/20/base 2025-08-14T21:24:12.6681555Z * [new branch] gh/codingwithsurya/20/head -> origin/gh/codingwithsurya/20/head 2025-08-14T21:24:12.6681718Z * [new branch] gh/codingwithsurya/20/orig -> origin/gh/codingwithsurya/20/orig 2025-08-14T21:24:12.6681876Z * [new branch] gh/codingwithsurya/21/base -> origin/gh/codingwithsurya/21/base 2025-08-14T21:24:12.6682041Z * [new branch] gh/codingwithsurya/21/head -> origin/gh/codingwithsurya/21/head 2025-08-14T21:24:12.6682255Z * [new branch] gh/codingwithsurya/21/orig -> origin/gh/codingwithsurya/21/orig 2025-08-14T21:24:12.6682442Z * [new branch] gh/codingwithsurya/8/base -> origin/gh/codingwithsurya/8/base 2025-08-14T21:24:12.6682616Z * [new branch] gh/codingwithsurya/8/head -> origin/gh/codingwithsurya/8/head 2025-08-14T21:24:12.6682777Z * [new branch] gh/codingwithsurya/8/orig -> origin/gh/codingwithsurya/8/orig 2025-08-14T21:24:12.6682945Z * [new branch] gh/codingwithsurya/9/base -> origin/gh/codingwithsurya/9/base 2025-08-14T21:24:12.6683106Z * [new branch] gh/codingwithsurya/9/head -> origin/gh/codingwithsurya/9/head 2025-08-14T21:24:12.6683264Z * [new branch] gh/codingwithsurya/9/orig -> origin/gh/codingwithsurya/9/orig 2025-08-14T21:24:12.6683437Z * [new branch] gh/colinchan15/1/base -> origin/gh/colinchan15/1/base 2025-08-14T21:24:12.6683587Z * [new branch] gh/colinchan15/1/head -> origin/gh/colinchan15/1/head 2025-08-14T21:24:12.6683745Z * [new branch] gh/colinchan15/2/base -> origin/gh/colinchan15/2/base 2025-08-14T21:24:12.6683893Z * [new branch] gh/colinchan15/2/head -> origin/gh/colinchan15/2/head 2025-08-14T21:24:12.6684038Z * [new branch] gh/colinchan15/3/base -> origin/gh/colinchan15/3/base 2025-08-14T21:24:12.6684191Z * [new branch] gh/colinchan15/3/head -> origin/gh/colinchan15/3/head 2025-08-14T21:24:12.6684335Z * [new branch] gh/colinchan15/4/base -> origin/gh/colinchan15/4/base 2025-08-14T21:24:12.6684492Z * [new branch] gh/colinchan15/4/head -> origin/gh/colinchan15/4/head 2025-08-14T21:24:12.6684970Z * [new branch] gh/colinchan15/5/base -> origin/gh/colinchan15/5/base 2025-08-14T21:24:12.6685803Z * [new branch] gh/colinchan15/5/head -> origin/gh/colinchan15/5/head 2025-08-14T21:24:12.6686724Z * [new branch] gh/colinchan15/6/base -> origin/gh/colinchan15/6/base 2025-08-14T21:24:12.6687133Z * [new branch] gh/colinchan15/6/head -> origin/gh/colinchan15/6/head 2025-08-14T21:24:12.6691336Z * [new branch] gh/davidberard98/351/base -> origin/gh/davidberard98/351/base 2025-08-14T21:24:12.6691534Z * [new branch] gh/davidberard98/351/head -> origin/gh/davidberard98/351/head 2025-08-14T21:24:12.6691689Z * [new branch] gh/davidberard98/351/orig -> origin/gh/davidberard98/351/orig 2025-08-14T21:24:12.6691881Z * [new branch] gh/davidberard98/353/base -> origin/gh/davidberard98/353/base 2025-08-14T21:24:12.6692034Z * [new branch] gh/davidberard98/353/head -> origin/gh/davidberard98/353/head 2025-08-14T21:24:12.6692199Z * [new branch] gh/davidberard98/353/orig -> origin/gh/davidberard98/353/orig 2025-08-14T21:24:12.6692710Z * [new branch] gh/davidberard98/356/base -> origin/gh/davidberard98/356/base 2025-08-14T21:24:12.6694582Z * [new branch] gh/davidberard98/356/head -> origin/gh/davidberard98/356/head 2025-08-14T21:24:12.6694928Z * [new branch] gh/davidberard98/356/orig -> origin/gh/davidberard98/356/orig 2025-08-14T21:24:12.6695114Z * [new branch] gh/davidberard98/382/base -> origin/gh/davidberard98/382/base 2025-08-14T21:24:12.6698121Z * [new branch] gh/davidberard98/382/head -> origin/gh/davidberard98/382/head 2025-08-14T21:24:12.6698463Z * [new branch] gh/davidberard98/382/orig -> origin/gh/davidberard98/382/orig 2025-08-14T21:24:12.6698704Z * [new branch] gh/davidberard98/386/base -> origin/gh/davidberard98/386/base 2025-08-14T21:24:12.6698899Z * [new branch] gh/davidberard98/386/head -> origin/gh/davidberard98/386/head 2025-08-14T21:24:12.6699134Z * [new branch] gh/davidberard98/386/orig -> origin/gh/davidberard98/386/orig 2025-08-14T21:24:12.6700711Z * [new branch] gh/davidberard98/389/base -> origin/gh/davidberard98/389/base 2025-08-14T21:24:12.6701023Z * [new branch] gh/davidberard98/389/head -> origin/gh/davidberard98/389/head 2025-08-14T21:24:12.6701192Z * [new branch] gh/davidberard98/389/orig -> origin/gh/davidberard98/389/orig 2025-08-14T21:24:12.6703379Z * [new branch] gh/davidberard98/390/base -> origin/gh/davidberard98/390/base 2025-08-14T21:24:12.6703740Z * [new branch] gh/davidberard98/390/head -> origin/gh/davidberard98/390/head 2025-08-14T21:24:12.6704005Z * [new branch] gh/davidberard98/390/orig -> origin/gh/davidberard98/390/orig 2025-08-14T21:24:12.6706018Z * [new branch] gh/davidberard98/391/base -> origin/gh/davidberard98/391/base 2025-08-14T21:24:12.6706367Z * [new branch] gh/davidberard98/391/head -> origin/gh/davidberard98/391/head 2025-08-14T21:24:12.6706621Z * [new branch] gh/davidberard98/391/orig -> origin/gh/davidberard98/391/orig 2025-08-14T21:24:12.6707010Z * [new branch] gh/davidberard98/392/base -> origin/gh/davidberard98/392/base 2025-08-14T21:24:12.6708499Z * [new branch] gh/davidberard98/392/head -> origin/gh/davidberard98/392/head 2025-08-14T21:24:12.6708859Z * [new branch] gh/davidberard98/392/orig -> origin/gh/davidberard98/392/orig 2025-08-14T21:24:12.6709281Z * [new branch] gh/davidberard98/393/base -> origin/gh/davidberard98/393/base 2025-08-14T21:24:12.6710629Z * [new branch] gh/davidberard98/393/head -> origin/gh/davidberard98/393/head 2025-08-14T21:24:12.6710939Z * [new branch] gh/davidberard98/393/orig -> origin/gh/davidberard98/393/orig 2025-08-14T21:24:12.6713514Z * [new branch] gh/davidberard98/394/base -> origin/gh/davidberard98/394/base 2025-08-14T21:24:12.6713881Z * [new branch] gh/davidberard98/394/head -> origin/gh/davidberard98/394/head 2025-08-14T21:24:12.6714156Z * [new branch] gh/davidberard98/394/orig -> origin/gh/davidberard98/394/orig 2025-08-14T21:24:12.6714594Z * [new branch] gh/davidberard98/395/base -> origin/gh/davidberard98/395/base 2025-08-14T21:24:12.6716678Z * [new branch] gh/davidberard98/395/head -> origin/gh/davidberard98/395/head 2025-08-14T21:24:12.6717035Z * [new branch] gh/davidberard98/395/orig -> origin/gh/davidberard98/395/orig 2025-08-14T21:24:12.6717270Z * [new branch] gh/davidberard98/396/base -> origin/gh/davidberard98/396/base 2025-08-14T21:24:12.6717739Z * [new branch] gh/davidberard98/396/head -> origin/gh/davidberard98/396/head 2025-08-14T21:24:12.6718492Z * [new branch] gh/davidberard98/396/orig -> origin/gh/davidberard98/396/orig 2025-08-14T21:24:12.6720134Z * [new branch] gh/davidberard98/397/base -> origin/gh/davidberard98/397/base 2025-08-14T21:24:12.6720366Z * [new branch] gh/davidberard98/397/head -> origin/gh/davidberard98/397/head 2025-08-14T21:24:12.6721514Z * [new branch] gh/davidberard98/397/orig -> origin/gh/davidberard98/397/orig 2025-08-14T21:24:12.6722446Z * [new branch] gh/davidberard98/398/base -> origin/gh/davidberard98/398/base 2025-08-14T21:24:12.6722731Z * [new branch] gh/davidberard98/398/head -> origin/gh/davidberard98/398/head 2025-08-14T21:24:12.6723816Z * [new branch] gh/davidberard98/398/orig -> origin/gh/davidberard98/398/orig 2025-08-14T21:24:12.6725123Z * [new branch] gh/desertfire/570/base -> origin/gh/desertfire/570/base 2025-08-14T21:24:12.6725453Z * [new branch] gh/desertfire/570/head -> origin/gh/desertfire/570/head 2025-08-14T21:24:12.6729783Z * [new branch] gh/desertfire/570/orig -> origin/gh/desertfire/570/orig 2025-08-14T21:24:12.6730131Z * [new branch] gh/desertfire/572/base -> origin/gh/desertfire/572/base 2025-08-14T21:24:12.6730513Z * [new branch] gh/desertfire/572/head -> origin/gh/desertfire/572/head 2025-08-14T21:24:12.6730809Z * [new branch] gh/desertfire/572/orig -> origin/gh/desertfire/572/orig 2025-08-14T21:24:12.6731009Z * [new branch] gh/desertfire/589/base -> origin/gh/desertfire/589/base 2025-08-14T21:24:12.6731175Z * [new branch] gh/desertfire/589/head -> origin/gh/desertfire/589/head 2025-08-14T21:24:12.6731346Z * [new branch] gh/desertfire/589/orig -> origin/gh/desertfire/589/orig 2025-08-14T21:24:12.6732883Z * [new branch] gh/desertfire/590/base -> origin/gh/desertfire/590/base 2025-08-14T21:24:12.6733057Z * [new branch] gh/desertfire/590/head -> origin/gh/desertfire/590/head 2025-08-14T21:24:12.6736029Z * [new branch] gh/desertfire/590/orig -> origin/gh/desertfire/590/orig 2025-08-14T21:24:12.6736250Z * [new branch] gh/desertfire/591/base -> origin/gh/desertfire/591/base 2025-08-14T21:24:12.6736417Z * [new branch] gh/desertfire/591/head -> origin/gh/desertfire/591/head 2025-08-14T21:24:12.6736568Z * [new branch] gh/desertfire/591/orig -> origin/gh/desertfire/591/orig 2025-08-14T21:24:12.6737867Z * [new branch] gh/desertfire/592/base -> origin/gh/desertfire/592/base 2025-08-14T21:24:12.6738155Z * [new branch] gh/desertfire/592/head -> origin/gh/desertfire/592/head 2025-08-14T21:24:12.6741473Z * [new branch] gh/desertfire/592/orig -> origin/gh/desertfire/592/orig 2025-08-14T21:24:12.6741668Z * [new branch] gh/desertfire/593/base -> origin/gh/desertfire/593/base 2025-08-14T21:24:12.6741828Z * [new branch] gh/desertfire/593/head -> origin/gh/desertfire/593/head 2025-08-14T21:24:12.6742018Z * [new branch] gh/desertfire/593/orig -> origin/gh/desertfire/593/orig 2025-08-14T21:24:12.6742205Z * [new branch] gh/desertfire/594/base -> origin/gh/desertfire/594/base 2025-08-14T21:24:12.6746251Z * [new branch] gh/desertfire/594/head -> origin/gh/desertfire/594/head 2025-08-14T21:24:12.6746448Z * [new branch] gh/desertfire/594/orig -> origin/gh/desertfire/594/orig 2025-08-14T21:24:12.6746598Z * [new branch] gh/desertfire/595/base -> origin/gh/desertfire/595/base 2025-08-14T21:24:12.6746756Z * [new branch] gh/desertfire/595/head -> origin/gh/desertfire/595/head 2025-08-14T21:24:12.6746908Z * [new branch] gh/desertfire/595/orig -> origin/gh/desertfire/595/orig 2025-08-14T21:24:12.6750479Z * [new branch] gh/desertfire/596/base -> origin/gh/desertfire/596/base 2025-08-14T21:24:12.6751005Z * [new branch] gh/desertfire/596/head -> origin/gh/desertfire/596/head 2025-08-14T21:24:12.6751207Z * [new branch] gh/desertfire/596/orig -> origin/gh/desertfire/596/orig 2025-08-14T21:24:12.6751385Z * [new branch] gh/desertfire/597/base -> origin/gh/desertfire/597/base 2025-08-14T21:24:12.6751569Z * [new branch] gh/desertfire/597/head -> origin/gh/desertfire/597/head 2025-08-14T21:24:12.6751713Z * [new branch] gh/desertfire/597/orig -> origin/gh/desertfire/597/orig 2025-08-14T21:24:12.6755041Z * [new branch] gh/dharakk/1/base -> origin/gh/dharakk/1/base 2025-08-14T21:24:12.6755193Z * [new branch] gh/dharakk/1/head -> origin/gh/dharakk/1/head 2025-08-14T21:24:12.6755331Z * [new branch] gh/dharakk/4/base -> origin/gh/dharakk/4/base 2025-08-14T21:24:12.6755473Z * [new branch] gh/dharakk/4/head -> origin/gh/dharakk/4/head 2025-08-14T21:24:12.6755606Z * [new branch] gh/dharakk/4/orig -> origin/gh/dharakk/4/orig 2025-08-14T21:24:12.6759196Z * [new branch] gh/drisspg/140/base -> origin/gh/drisspg/140/base 2025-08-14T21:24:12.6759341Z * [new branch] gh/drisspg/140/head -> origin/gh/drisspg/140/head 2025-08-14T21:24:12.6759471Z * [new branch] gh/drisspg/140/orig -> origin/gh/drisspg/140/orig 2025-08-14T21:24:12.6759606Z * [new branch] gh/drisspg/149/base -> origin/gh/drisspg/149/base 2025-08-14T21:24:12.6759738Z * [new branch] gh/drisspg/149/head -> origin/gh/drisspg/149/head 2025-08-14T21:24:12.6759875Z * [new branch] gh/drisspg/149/orig -> origin/gh/drisspg/149/orig 2025-08-14T21:24:12.6763220Z * [new branch] gh/drisspg/150/base -> origin/gh/drisspg/150/base 2025-08-14T21:24:12.6764032Z * [new branch] gh/drisspg/150/head -> origin/gh/drisspg/150/head 2025-08-14T21:24:12.6764325Z * [new branch] gh/drisspg/150/orig -> origin/gh/drisspg/150/orig 2025-08-14T21:24:12.6764498Z * [new branch] gh/drisspg/151/base -> origin/gh/drisspg/151/base 2025-08-14T21:24:12.6764662Z * [new branch] gh/drisspg/151/head -> origin/gh/drisspg/151/head 2025-08-14T21:24:12.6764804Z * [new branch] gh/drisspg/151/orig -> origin/gh/drisspg/151/orig 2025-08-14T21:24:12.6764939Z * [new branch] gh/drisspg/158/base -> origin/gh/drisspg/158/base 2025-08-14T21:24:12.6765324Z * [new branch] gh/drisspg/158/head -> origin/gh/drisspg/158/head 2025-08-14T21:24:12.6766051Z * [new branch] gh/drisspg/158/orig -> origin/gh/drisspg/158/orig 2025-08-14T21:24:12.6771942Z * [new branch] gh/drisspg/159/base -> origin/gh/drisspg/159/base 2025-08-14T21:24:12.6772109Z * [new branch] gh/drisspg/159/head -> origin/gh/drisspg/159/head 2025-08-14T21:24:12.6772255Z * [new branch] gh/drisspg/159/orig -> origin/gh/drisspg/159/orig 2025-08-14T21:24:12.6772503Z * [new branch] gh/drisspg/166/base -> origin/gh/drisspg/166/base 2025-08-14T21:24:12.6772852Z * [new branch] gh/drisspg/166/head -> origin/gh/drisspg/166/head 2025-08-14T21:24:12.6773086Z * [new branch] gh/drisspg/166/orig -> origin/gh/drisspg/166/orig 2025-08-14T21:24:12.6773241Z * [new branch] gh/drisspg/168/base -> origin/gh/drisspg/168/base 2025-08-14T21:24:12.6773387Z * [new branch] gh/drisspg/168/head -> origin/gh/drisspg/168/head 2025-08-14T21:24:12.6773595Z * [new branch] gh/drisspg/168/orig -> origin/gh/drisspg/168/orig 2025-08-14T21:24:12.6773743Z * [new branch] gh/drisspg/169/base -> origin/gh/drisspg/169/base 2025-08-14T21:24:12.6773947Z * [new branch] gh/drisspg/169/head -> origin/gh/drisspg/169/head 2025-08-14T21:24:12.6774085Z * [new branch] gh/drisspg/169/orig -> origin/gh/drisspg/169/orig 2025-08-14T21:24:12.6780641Z * [new branch] gh/drisspg/170/base -> origin/gh/drisspg/170/base 2025-08-14T21:24:12.6782837Z * [new branch] gh/drisspg/170/head -> origin/gh/drisspg/170/head 2025-08-14T21:24:12.6783015Z * [new branch] gh/drisspg/170/orig -> origin/gh/drisspg/170/orig 2025-08-14T21:24:12.6783161Z * [new branch] gh/drisspg/171/base -> origin/gh/drisspg/171/base 2025-08-14T21:24:12.6783298Z * [new branch] gh/drisspg/171/head -> origin/gh/drisspg/171/head 2025-08-14T21:24:12.6783455Z * [new branch] gh/drisspg/171/orig -> origin/gh/drisspg/171/orig 2025-08-14T21:24:12.6783599Z * [new branch] gh/drisspg/172/base -> origin/gh/drisspg/172/base 2025-08-14T21:24:12.6783745Z * [new branch] gh/drisspg/172/head -> origin/gh/drisspg/172/head 2025-08-14T21:24:12.6783887Z * [new branch] gh/drisspg/172/orig -> origin/gh/drisspg/172/orig 2025-08-14T21:24:12.6784203Z * [new branch] gh/drisspg/173/base -> origin/gh/drisspg/173/base 2025-08-14T21:24:12.6784365Z * [new branch] gh/drisspg/173/head -> origin/gh/drisspg/173/head 2025-08-14T21:24:12.6784506Z * [new branch] gh/drisspg/173/orig -> origin/gh/drisspg/173/orig 2025-08-14T21:24:12.6784649Z * [new branch] gh/drisspg/174/base -> origin/gh/drisspg/174/base 2025-08-14T21:24:12.6788177Z * [new branch] gh/drisspg/174/head -> origin/gh/drisspg/174/head 2025-08-14T21:24:12.6788345Z * [new branch] gh/drisspg/174/orig -> origin/gh/drisspg/174/orig 2025-08-14T21:24:12.6788499Z * [new branch] gh/drisspg/175/base -> origin/gh/drisspg/175/base 2025-08-14T21:24:12.6788705Z * [new branch] gh/drisspg/175/head -> origin/gh/drisspg/175/head 2025-08-14T21:24:12.6788860Z * [new branch] gh/drisspg/175/orig -> origin/gh/drisspg/175/orig 2025-08-14T21:24:12.6789080Z * [new branch] gh/drisspg/176/base -> origin/gh/drisspg/176/base 2025-08-14T21:24:12.6793146Z * [new branch] gh/drisspg/176/head -> origin/gh/drisspg/176/head 2025-08-14T21:24:12.6793336Z * [new branch] gh/drisspg/176/orig -> origin/gh/drisspg/176/orig 2025-08-14T21:24:12.6793496Z * [new branch] gh/drisspg/177/base -> origin/gh/drisspg/177/base 2025-08-14T21:24:12.6793738Z * [new branch] gh/drisspg/177/head -> origin/gh/drisspg/177/head 2025-08-14T21:24:12.6793885Z * [new branch] gh/drisspg/177/orig -> origin/gh/drisspg/177/orig 2025-08-14T21:24:12.6794036Z * [new branch] gh/drisspg/178/base -> origin/gh/drisspg/178/base 2025-08-14T21:24:12.6794165Z * [new branch] gh/drisspg/178/head -> origin/gh/drisspg/178/head 2025-08-14T21:24:12.6794377Z * [new branch] gh/drisspg/178/orig -> origin/gh/drisspg/178/orig 2025-08-14T21:24:12.6800328Z * [new branch] gh/drisspg/179/base -> origin/gh/drisspg/179/base 2025-08-14T21:24:12.6800518Z * [new branch] gh/drisspg/179/head -> origin/gh/drisspg/179/head 2025-08-14T21:24:12.6800671Z * [new branch] gh/drisspg/179/orig -> origin/gh/drisspg/179/orig 2025-08-14T21:24:12.6800816Z * [new branch] gh/drisspg/180/base -> origin/gh/drisspg/180/base 2025-08-14T21:24:12.6800966Z * [new branch] gh/drisspg/180/head -> origin/gh/drisspg/180/head 2025-08-14T21:24:12.6801117Z * [new branch] gh/drisspg/180/orig -> origin/gh/drisspg/180/orig 2025-08-14T21:24:12.6801266Z * [new branch] gh/drisspg/181/base -> origin/gh/drisspg/181/base 2025-08-14T21:24:12.6801421Z * [new branch] gh/drisspg/181/head -> origin/gh/drisspg/181/head 2025-08-14T21:24:12.6801602Z * [new branch] gh/drisspg/181/orig -> origin/gh/drisspg/181/orig 2025-08-14T21:24:12.6801758Z * [new branch] gh/drisspg/182/base -> origin/gh/drisspg/182/base 2025-08-14T21:24:12.6801898Z * [new branch] gh/drisspg/182/head -> origin/gh/drisspg/182/head 2025-08-14T21:24:12.6802044Z * [new branch] gh/drisspg/183/base -> origin/gh/drisspg/183/base 2025-08-14T21:24:12.6802370Z * [new branch] gh/drisspg/183/head -> origin/gh/drisspg/183/head 2025-08-14T21:24:12.6802526Z * [new branch] gh/drisspg/184/base -> origin/gh/drisspg/184/base 2025-08-14T21:24:12.6803819Z * [new branch] gh/drisspg/184/head -> origin/gh/drisspg/184/head 2025-08-14T21:24:12.6804487Z * [new branch] gh/drisspg/185/base -> origin/gh/drisspg/185/base 2025-08-14T21:24:12.6804673Z * [new branch] gh/drisspg/185/head -> origin/gh/drisspg/185/head 2025-08-14T21:24:12.6806253Z * [new branch] gh/dsjohns2/1/base -> origin/gh/dsjohns2/1/base 2025-08-14T21:24:12.6806441Z * [new branch] gh/dsjohns2/1/head -> origin/gh/dsjohns2/1/head 2025-08-14T21:24:12.6814044Z * [new branch] gh/eellison/784/base -> origin/gh/eellison/784/base 2025-08-14T21:24:12.6814240Z * [new branch] gh/eellison/784/head -> origin/gh/eellison/784/head 2025-08-14T21:24:12.6814389Z * [new branch] gh/eellison/784/orig -> origin/gh/eellison/784/orig 2025-08-14T21:24:12.6814538Z * [new branch] gh/eellison/785/base -> origin/gh/eellison/785/base 2025-08-14T21:24:12.6814685Z * [new branch] gh/eellison/785/head -> origin/gh/eellison/785/head 2025-08-14T21:24:12.6814830Z * [new branch] gh/eellison/785/orig -> origin/gh/eellison/785/orig 2025-08-14T21:24:12.6814988Z * [new branch] gh/eellison/789/base -> origin/gh/eellison/789/base 2025-08-14T21:24:12.6815153Z * [new branch] gh/eellison/789/head -> origin/gh/eellison/789/head 2025-08-14T21:24:12.6820282Z * [new branch] gh/eellison/789/orig -> origin/gh/eellison/789/orig 2025-08-14T21:24:12.6820462Z * [new branch] gh/eellison/800/base -> origin/gh/eellison/800/base 2025-08-14T21:24:12.6820625Z * [new branch] gh/eellison/800/head -> origin/gh/eellison/800/head 2025-08-14T21:24:12.6820784Z * [new branch] gh/eellison/800/orig -> origin/gh/eellison/800/orig 2025-08-14T21:24:12.6820931Z * [new branch] gh/eellison/801/base -> origin/gh/eellison/801/base 2025-08-14T21:24:12.6821083Z * [new branch] gh/eellison/801/head -> origin/gh/eellison/801/head 2025-08-14T21:24:12.6821228Z * [new branch] gh/eellison/801/orig -> origin/gh/eellison/801/orig 2025-08-14T21:24:12.6821374Z * [new branch] gh/eellison/802/base -> origin/gh/eellison/802/base 2025-08-14T21:24:12.6821709Z * [new branch] gh/eellison/802/head -> origin/gh/eellison/802/head 2025-08-14T21:24:12.6821900Z * [new branch] gh/eellison/802/orig -> origin/gh/eellison/802/orig 2025-08-14T21:24:12.6822043Z * [new branch] gh/eellison/805/base -> origin/gh/eellison/805/base 2025-08-14T21:24:12.6822191Z * [new branch] gh/eellison/805/head -> origin/gh/eellison/805/head 2025-08-14T21:24:12.6822466Z * [new branch] gh/eellison/805/orig -> origin/gh/eellison/805/orig 2025-08-14T21:24:12.6822618Z * [new branch] gh/eellison/808/base -> origin/gh/eellison/808/base 2025-08-14T21:24:12.6823592Z * [new branch] gh/eellison/808/head -> origin/gh/eellison/808/head 2025-08-14T21:24:12.6823915Z * [new branch] gh/eellison/808/orig -> origin/gh/eellison/808/orig 2025-08-14T21:24:12.6825169Z * [new branch] gh/eellison/809/base -> origin/gh/eellison/809/base 2025-08-14T21:24:12.6825522Z * [new branch] gh/eellison/809/head -> origin/gh/eellison/809/head 2025-08-14T21:24:12.6826540Z * [new branch] gh/eellison/809/orig -> origin/gh/eellison/809/orig 2025-08-14T21:24:12.6827476Z * [new branch] gh/eellison/810/base -> origin/gh/eellison/810/base 2025-08-14T21:24:12.6827997Z * [new branch] gh/eellison/810/head -> origin/gh/eellison/810/head 2025-08-14T21:24:12.6828613Z * [new branch] gh/eellison/810/orig -> origin/gh/eellison/810/orig 2025-08-14T21:24:12.6829827Z * [new branch] gh/eellison/811/base -> origin/gh/eellison/811/base 2025-08-14T21:24:12.6830256Z * [new branch] gh/eellison/811/head -> origin/gh/eellison/811/head 2025-08-14T21:24:12.6831226Z * [new branch] gh/eellison/811/orig -> origin/gh/eellison/811/orig 2025-08-14T21:24:12.6832170Z * [new branch] gh/eellison/812/base -> origin/gh/eellison/812/base 2025-08-14T21:24:12.6832480Z * [new branch] gh/eellison/812/head -> origin/gh/eellison/812/head 2025-08-14T21:24:12.6833572Z * [new branch] gh/eellison/812/orig -> origin/gh/eellison/812/orig 2025-08-14T21:24:12.6834631Z * [new branch] gh/eellison/813/base -> origin/gh/eellison/813/base 2025-08-14T21:24:12.6834850Z * [new branch] gh/eellison/813/head -> origin/gh/eellison/813/head 2025-08-14T21:24:12.6835934Z * [new branch] gh/eellison/813/orig -> origin/gh/eellison/813/orig 2025-08-14T21:24:12.6837026Z * [new branch] gh/etaf/132/base -> origin/gh/etaf/132/base 2025-08-14T21:24:12.6843163Z * [new branch] gh/etaf/132/head -> origin/gh/etaf/132/head 2025-08-14T21:24:12.6843360Z * [new branch] gh/etaf/132/orig -> origin/gh/etaf/132/orig 2025-08-14T21:24:12.6843531Z * [new branch] gh/etaf/138/base -> origin/gh/etaf/138/base 2025-08-14T21:24:12.6843686Z * [new branch] gh/etaf/138/head -> origin/gh/etaf/138/head 2025-08-14T21:24:12.6843828Z * [new branch] gh/etaf/138/orig -> origin/gh/etaf/138/orig 2025-08-14T21:24:12.6843965Z * [new branch] gh/etaf/140/base -> origin/gh/etaf/140/base 2025-08-14T21:24:12.6844099Z * [new branch] gh/etaf/140/head -> origin/gh/etaf/140/head 2025-08-14T21:24:12.6844241Z * [new branch] gh/etaf/140/orig -> origin/gh/etaf/140/orig 2025-08-14T21:24:12.6844378Z * [new branch] gh/etaf/143/base -> origin/gh/etaf/143/base 2025-08-14T21:24:12.6844552Z * [new branch] gh/etaf/143/head -> origin/gh/etaf/143/head 2025-08-14T21:24:12.6845352Z * [new branch] gh/etaf/143/orig -> origin/gh/etaf/143/orig 2025-08-14T21:24:12.6851618Z * [new branch] gh/etaf/147/base -> origin/gh/etaf/147/base 2025-08-14T21:24:12.6856006Z * [new branch] gh/etaf/147/head -> origin/gh/etaf/147/head 2025-08-14T21:24:12.6860193Z * [new branch] gh/etaf/148/base -> origin/gh/etaf/148/base 2025-08-14T21:24:12.6865327Z * [new branch] gh/etaf/148/head -> origin/gh/etaf/148/head 2025-08-14T21:24:12.6869434Z * [new branch] gh/etaf/148/orig -> origin/gh/etaf/148/orig 2025-08-14T21:24:12.6875356Z * [new branch] gh/etaf/149/base -> origin/gh/etaf/149/base 2025-08-14T21:24:12.6880930Z * [new branch] gh/etaf/149/head -> origin/gh/etaf/149/head 2025-08-14T21:24:12.6881110Z * [new branch] gh/etaf/149/orig -> origin/gh/etaf/149/orig 2025-08-14T21:24:12.6881251Z * [new branch] gh/etaf/150/base -> origin/gh/etaf/150/base 2025-08-14T21:24:12.6881399Z * [new branch] gh/etaf/150/head -> origin/gh/etaf/150/head 2025-08-14T21:24:12.6881547Z * [new branch] gh/etaf/150/orig -> origin/gh/etaf/150/orig 2025-08-14T21:24:12.6881678Z * [new branch] gh/etaf/151/base -> origin/gh/etaf/151/base 2025-08-14T21:24:12.6881818Z * [new branch] gh/etaf/151/head -> origin/gh/etaf/151/head 2025-08-14T21:24:12.6881959Z * [new branch] gh/etaf/151/orig -> origin/gh/etaf/151/orig 2025-08-14T21:24:12.6882089Z * [new branch] gh/etaf/152/base -> origin/gh/etaf/152/base 2025-08-14T21:24:12.6882224Z * [new branch] gh/etaf/152/head -> origin/gh/etaf/152/head 2025-08-14T21:24:12.6882348Z * [new branch] gh/etaf/152/orig -> origin/gh/etaf/152/orig 2025-08-14T21:24:12.6882474Z * [new branch] gh/etaf/153/base -> origin/gh/etaf/153/base 2025-08-14T21:24:12.6882804Z * [new branch] gh/etaf/153/head -> origin/gh/etaf/153/head 2025-08-14T21:24:12.6882949Z * [new branch] gh/etaf/153/orig -> origin/gh/etaf/153/orig 2025-08-14T21:24:12.6883085Z * [new branch] gh/etaf/154/base -> origin/gh/etaf/154/base 2025-08-14T21:24:12.6883212Z * [new branch] gh/etaf/154/head -> origin/gh/etaf/154/head 2025-08-14T21:24:12.6883341Z * [new branch] gh/etaf/154/orig -> origin/gh/etaf/154/orig 2025-08-14T21:24:12.6883479Z * [new branch] gh/etaf/155/base -> origin/gh/etaf/155/base 2025-08-14T21:24:12.6883607Z * [new branch] gh/etaf/155/head -> origin/gh/etaf/155/head 2025-08-14T21:24:12.6883741Z * [new branch] gh/etaf/155/orig -> origin/gh/etaf/155/orig 2025-08-14T21:24:12.6883891Z * [new branch] gh/ezyang/2374/base -> origin/gh/ezyang/2374/base 2025-08-14T21:24:12.6884040Z * [new branch] gh/ezyang/2374/head -> origin/gh/ezyang/2374/head 2025-08-14T21:24:12.6884190Z * [new branch] gh/ezyang/2374/orig -> origin/gh/ezyang/2374/orig 2025-08-14T21:24:12.6884326Z * [new branch] gh/ezyang/2973/base -> origin/gh/ezyang/2973/base 2025-08-14T21:24:12.6884464Z * [new branch] gh/ezyang/2973/head -> origin/gh/ezyang/2973/head 2025-08-14T21:24:12.6884610Z * [new branch] gh/ezyang/2973/orig -> origin/gh/ezyang/2973/orig 2025-08-14T21:24:12.6887817Z * [new branch] gh/ezyang/2974/base -> origin/gh/ezyang/2974/base 2025-08-14T21:24:12.6887988Z * [new branch] gh/ezyang/2974/head -> origin/gh/ezyang/2974/head 2025-08-14T21:24:12.6888158Z * [new branch] gh/ezyang/2974/orig -> origin/gh/ezyang/2974/orig 2025-08-14T21:24:12.6888326Z * [new branch] gh/ezyang/3068/base -> origin/gh/ezyang/3068/base 2025-08-14T21:24:12.6888499Z * [new branch] gh/ezyang/3068/head -> origin/gh/ezyang/3068/head 2025-08-14T21:24:12.6888757Z * [new branch] gh/ezyang/3068/orig -> origin/gh/ezyang/3068/orig 2025-08-14T21:24:12.6888924Z * [new branch] gh/ezyang/3071/base -> origin/gh/ezyang/3071/base 2025-08-14T21:24:12.6889082Z * [new branch] gh/ezyang/3071/head -> origin/gh/ezyang/3071/head 2025-08-14T21:24:12.6889228Z * [new branch] gh/ezyang/3071/orig -> origin/gh/ezyang/3071/orig 2025-08-14T21:24:12.6889377Z * [new branch] gh/ezyang/3074/base -> origin/gh/ezyang/3074/base 2025-08-14T21:24:12.6889525Z * [new branch] gh/ezyang/3074/head -> origin/gh/ezyang/3074/head 2025-08-14T21:24:12.6889673Z * [new branch] gh/ezyang/3074/orig -> origin/gh/ezyang/3074/orig 2025-08-14T21:24:12.6889820Z * [new branch] gh/ezyang/3088/base -> origin/gh/ezyang/3088/base 2025-08-14T21:24:12.6889978Z * [new branch] gh/ezyang/3088/head -> origin/gh/ezyang/3088/head 2025-08-14T21:24:12.6890126Z * [new branch] gh/ezyang/3088/orig -> origin/gh/ezyang/3088/orig 2025-08-14T21:24:12.6890273Z * [new branch] gh/ezyang/3092/base -> origin/gh/ezyang/3092/base 2025-08-14T21:24:12.6890418Z * [new branch] gh/ezyang/3092/head -> origin/gh/ezyang/3092/head 2025-08-14T21:24:12.6890563Z * [new branch] gh/ezyang/3092/orig -> origin/gh/ezyang/3092/orig 2025-08-14T21:24:12.6890708Z * [new branch] gh/ezyang/3097/base -> origin/gh/ezyang/3097/base 2025-08-14T21:24:12.6890861Z * [new branch] gh/ezyang/3097/head -> origin/gh/ezyang/3097/head 2025-08-14T21:24:12.6891948Z * [new branch] gh/ezyang/3097/orig -> origin/gh/ezyang/3097/orig 2025-08-14T21:24:12.6892096Z * [new branch] gh/ezyang/3098/base -> origin/gh/ezyang/3098/base 2025-08-14T21:24:12.6892430Z * [new branch] gh/ezyang/3098/head -> origin/gh/ezyang/3098/head 2025-08-14T21:24:12.6892953Z * [new branch] gh/ezyang/3098/orig -> origin/gh/ezyang/3098/orig 2025-08-14T21:24:12.6899340Z * [new branch] gh/ezyang/3099/base -> origin/gh/ezyang/3099/base 2025-08-14T21:24:12.6903492Z * [new branch] gh/ezyang/3099/head -> origin/gh/ezyang/3099/head 2025-08-14T21:24:12.6907795Z * [new branch] gh/ezyang/3099/orig -> origin/gh/ezyang/3099/orig 2025-08-14T21:24:12.6912149Z * [new branch] gh/ezyang/3100/base -> origin/gh/ezyang/3100/base 2025-08-14T21:24:12.6912349Z * [new branch] gh/ezyang/3100/head -> origin/gh/ezyang/3100/head 2025-08-14T21:24:12.6912576Z * [new branch] gh/ezyang/3100/orig -> origin/gh/ezyang/3100/orig 2025-08-14T21:24:12.6912737Z * [new branch] gh/ezyang/3101/base -> origin/gh/ezyang/3101/base 2025-08-14T21:24:12.6912923Z * [new branch] gh/ezyang/3101/head -> origin/gh/ezyang/3101/head 2025-08-14T21:24:12.6913068Z * [new branch] gh/ezyang/3101/orig -> origin/gh/ezyang/3101/orig 2025-08-14T21:24:12.6913212Z * [new branch] gh/ezyang/3102/base -> origin/gh/ezyang/3102/base 2025-08-14T21:24:12.6913355Z * [new branch] gh/ezyang/3102/head -> origin/gh/ezyang/3102/head 2025-08-14T21:24:12.6913498Z * [new branch] gh/ezyang/3102/orig -> origin/gh/ezyang/3102/orig 2025-08-14T21:24:12.6913640Z * [new branch] gh/ezyang/3103/base -> origin/gh/ezyang/3103/base 2025-08-14T21:24:12.6913784Z * [new branch] gh/ezyang/3103/head -> origin/gh/ezyang/3103/head 2025-08-14T21:24:12.6913935Z * [new branch] gh/ezyang/3103/orig -> origin/gh/ezyang/3103/orig 2025-08-14T21:24:12.6914085Z * [new branch] gh/ezyang/3104/base -> origin/gh/ezyang/3104/base 2025-08-14T21:24:12.6914401Z * [new branch] gh/ezyang/3104/head -> origin/gh/ezyang/3104/head 2025-08-14T21:24:12.6914551Z * [new branch] gh/ezyang/3104/orig -> origin/gh/ezyang/3104/orig 2025-08-14T21:24:12.6914697Z * [new branch] gh/ezyang/3105/base -> origin/gh/ezyang/3105/base 2025-08-14T21:24:12.6914840Z * [new branch] gh/ezyang/3105/head -> origin/gh/ezyang/3105/head 2025-08-14T21:24:12.6915000Z * [new branch] gh/ezyang/3105/orig -> origin/gh/ezyang/3105/orig 2025-08-14T21:24:12.6915150Z * [new branch] gh/ezyang/3106/base -> origin/gh/ezyang/3106/base 2025-08-14T21:24:12.6915300Z * [new branch] gh/ezyang/3106/head -> origin/gh/ezyang/3106/head 2025-08-14T21:24:12.6915447Z * [new branch] gh/ezyang/3106/orig -> origin/gh/ezyang/3106/orig 2025-08-14T21:24:12.6915598Z * [new branch] gh/ezyang/3107/base -> origin/gh/ezyang/3107/base 2025-08-14T21:24:12.6915748Z * [new branch] gh/ezyang/3107/head -> origin/gh/ezyang/3107/head 2025-08-14T21:24:12.6915895Z * [new branch] gh/ezyang/3107/orig -> origin/gh/ezyang/3107/orig 2025-08-14T21:24:12.6916202Z * [new branch] gh/ezyang/3108/base -> origin/gh/ezyang/3108/base 2025-08-14T21:24:12.6916371Z * [new branch] gh/ezyang/3108/head -> origin/gh/ezyang/3108/head 2025-08-14T21:24:12.6916508Z * [new branch] gh/ezyang/3108/orig -> origin/gh/ezyang/3108/orig 2025-08-14T21:24:12.6916784Z * [new branch] gh/ezyang/3109/base -> origin/gh/ezyang/3109/base 2025-08-14T21:24:12.6916948Z * [new branch] gh/ezyang/3109/head -> origin/gh/ezyang/3109/head 2025-08-14T21:24:12.6922121Z * [new branch] gh/ezyang/3109/orig -> origin/gh/ezyang/3109/orig 2025-08-14T21:24:12.6922477Z * [new branch] gh/ezyang/3110/base -> origin/gh/ezyang/3110/base 2025-08-14T21:24:12.6922641Z * [new branch] gh/ezyang/3110/head -> origin/gh/ezyang/3110/head 2025-08-14T21:24:12.6922785Z * [new branch] gh/ezyang/3110/orig -> origin/gh/ezyang/3110/orig 2025-08-14T21:24:12.6922935Z * [new branch] gh/ezyang/3111/base -> origin/gh/ezyang/3111/base 2025-08-14T21:24:12.6923140Z * [new branch] gh/ezyang/3111/head -> origin/gh/ezyang/3111/head 2025-08-14T21:24:12.6923293Z * [new branch] gh/ezyang/3111/orig -> origin/gh/ezyang/3111/orig 2025-08-14T21:24:12.6923432Z * [new branch] gh/ezyang/3112/base -> origin/gh/ezyang/3112/base 2025-08-14T21:24:12.6923572Z * [new branch] gh/ezyang/3112/head -> origin/gh/ezyang/3112/head 2025-08-14T21:24:12.6923723Z * [new branch] gh/ezyang/3112/orig -> origin/gh/ezyang/3112/orig 2025-08-14T21:24:12.6923869Z * [new branch] gh/ezyang/3113/base -> origin/gh/ezyang/3113/base 2025-08-14T21:24:12.6924473Z * [new branch] gh/ezyang/3113/head -> origin/gh/ezyang/3113/head 2025-08-14T21:24:12.6925235Z * [new branch] gh/ezyang/3113/orig -> origin/gh/ezyang/3113/orig 2025-08-14T21:24:12.6926469Z * [new branch] gh/ezyang/3114/base -> origin/gh/ezyang/3114/base 2025-08-14T21:24:12.6926882Z * [new branch] gh/ezyang/3114/head -> origin/gh/ezyang/3114/head 2025-08-14T21:24:12.6927859Z * [new branch] gh/ezyang/3114/orig -> origin/gh/ezyang/3114/orig 2025-08-14T21:24:12.6929288Z * [new branch] gh/ezyang/3115/base -> origin/gh/ezyang/3115/base 2025-08-14T21:24:12.6929443Z * [new branch] gh/ezyang/3115/head -> origin/gh/ezyang/3115/head 2025-08-14T21:24:12.6930467Z * [new branch] gh/ezyang/3115/orig -> origin/gh/ezyang/3115/orig 2025-08-14T21:24:12.6931252Z * [new branch] gh/ezyang/3116/base -> origin/gh/ezyang/3116/base 2025-08-14T21:24:12.6931693Z * [new branch] gh/ezyang/3116/head -> origin/gh/ezyang/3116/head 2025-08-14T21:24:12.6933013Z * [new branch] gh/ezyang/3116/orig -> origin/gh/ezyang/3116/orig 2025-08-14T21:24:12.6933485Z * [new branch] gh/ezyang/3117/base -> origin/gh/ezyang/3117/base 2025-08-14T21:24:12.6934118Z * [new branch] gh/ezyang/3117/head -> origin/gh/ezyang/3117/head 2025-08-14T21:24:12.6934984Z * [new branch] gh/ezyang/3117/orig -> origin/gh/ezyang/3117/orig 2025-08-14T21:24:12.6936260Z * [new branch] gh/ezyang/3118/base -> origin/gh/ezyang/3118/base 2025-08-14T21:24:12.6936408Z * [new branch] gh/ezyang/3118/head -> origin/gh/ezyang/3118/head 2025-08-14T21:24:12.6937414Z * [new branch] gh/ezyang/3118/orig -> origin/gh/ezyang/3118/orig 2025-08-14T21:24:12.6938079Z * [new branch] gh/ezyang/3119/base -> origin/gh/ezyang/3119/base 2025-08-14T21:24:12.6938945Z * [new branch] gh/ezyang/3119/head -> origin/gh/ezyang/3119/head 2025-08-14T21:24:12.6939378Z * [new branch] gh/ezyang/3119/orig -> origin/gh/ezyang/3119/orig 2025-08-14T21:24:12.6940559Z * [new branch] gh/ezyang/3120/base -> origin/gh/ezyang/3120/base 2025-08-14T21:24:12.6940913Z * [new branch] gh/ezyang/3120/head -> origin/gh/ezyang/3120/head 2025-08-14T21:24:12.6941913Z * [new branch] gh/ezyang/3120/orig -> origin/gh/ezyang/3120/orig 2025-08-14T21:24:12.6942899Z * [new branch] gh/ezyang/3121/base -> origin/gh/ezyang/3121/base 2025-08-14T21:24:12.6943142Z * [new branch] gh/ezyang/3121/head -> origin/gh/ezyang/3121/head 2025-08-14T21:24:12.6944329Z * [new branch] gh/ezyang/3121/orig -> origin/gh/ezyang/3121/orig 2025-08-14T21:24:12.6944886Z * [new branch] gh/ezyang/3122/base -> origin/gh/ezyang/3122/base 2025-08-14T21:24:12.6945636Z * [new branch] gh/ezyang/3122/head -> origin/gh/ezyang/3122/head 2025-08-14T21:24:12.6946249Z * [new branch] gh/ezyang/3122/orig -> origin/gh/ezyang/3122/orig 2025-08-14T21:24:12.6947409Z * [new branch] gh/ezyang/3123/base -> origin/gh/ezyang/3123/base 2025-08-14T21:24:12.6947824Z * [new branch] gh/ezyang/3123/head -> origin/gh/ezyang/3123/head 2025-08-14T21:24:12.6948717Z * [new branch] gh/ezyang/3123/orig -> origin/gh/ezyang/3123/orig 2025-08-14T21:24:12.6949708Z * [new branch] gh/ezyang/3124/base -> origin/gh/ezyang/3124/base 2025-08-14T21:24:12.6949977Z * [new branch] gh/ezyang/3124/head -> origin/gh/ezyang/3124/head 2025-08-14T21:24:12.6951057Z * [new branch] gh/ezyang/3124/orig -> origin/gh/ezyang/3124/orig 2025-08-14T21:24:12.6951958Z * [new branch] gh/ezyang/3125/base -> origin/gh/ezyang/3125/base 2025-08-14T21:24:12.6952265Z * [new branch] gh/ezyang/3125/head -> origin/gh/ezyang/3125/head 2025-08-14T21:24:12.6953286Z * [new branch] gh/ezyang/3125/orig -> origin/gh/ezyang/3125/orig 2025-08-14T21:24:12.6954218Z * [new branch] gh/ezyang/3126/base -> origin/gh/ezyang/3126/base 2025-08-14T21:24:12.6954816Z * [new branch] gh/ezyang/3126/head -> origin/gh/ezyang/3126/head 2025-08-14T21:24:12.6955717Z * [new branch] gh/ezyang/3126/orig -> origin/gh/ezyang/3126/orig 2025-08-14T21:24:12.6956712Z * [new branch] gh/ezyang/3127/base -> origin/gh/ezyang/3127/base 2025-08-14T21:24:12.6957039Z * [new branch] gh/ezyang/3127/head -> origin/gh/ezyang/3127/head 2025-08-14T21:24:12.6958074Z * [new branch] gh/ezyang/3127/orig -> origin/gh/ezyang/3127/orig 2025-08-14T21:24:12.6958758Z * [new branch] gh/ezyang/3128/base -> origin/gh/ezyang/3128/base 2025-08-14T21:24:12.6959445Z * [new branch] gh/ezyang/3128/head -> origin/gh/ezyang/3128/head 2025-08-14T21:24:12.6960396Z * [new branch] gh/ezyang/3128/orig -> origin/gh/ezyang/3128/orig 2025-08-14T21:24:12.6961175Z * [new branch] gh/ezyang/3129/base -> origin/gh/ezyang/3129/base 2025-08-14T21:24:12.6961642Z * [new branch] gh/ezyang/3129/head -> origin/gh/ezyang/3129/head 2025-08-14T21:24:12.6962975Z * [new branch] gh/ezyang/3129/orig -> origin/gh/ezyang/3129/orig 2025-08-14T21:24:12.6963892Z * [new branch] gh/ezyang/3130/base -> origin/gh/ezyang/3130/base 2025-08-14T21:24:12.6964302Z * [new branch] gh/ezyang/3130/head -> origin/gh/ezyang/3130/head 2025-08-14T21:24:12.6965386Z * [new branch] gh/ezyang/3130/orig -> origin/gh/ezyang/3130/orig 2025-08-14T21:24:12.6966095Z * [new branch] gh/ezyang/3131/base -> origin/gh/ezyang/3131/base 2025-08-14T21:24:12.6967831Z * [new branch] gh/ezyang/3131/head -> origin/gh/ezyang/3131/head 2025-08-14T21:24:12.6968016Z * [new branch] gh/ezyang/3131/orig -> origin/gh/ezyang/3131/orig 2025-08-14T21:24:12.6968593Z * [new branch] gh/ezyang/3132/base -> origin/gh/ezyang/3132/base 2025-08-14T21:24:12.6970687Z * [new branch] gh/ezyang/3132/head -> origin/gh/ezyang/3132/head 2025-08-14T21:24:12.6971070Z * [new branch] gh/ezyang/3132/orig -> origin/gh/ezyang/3132/orig 2025-08-14T21:24:12.6971405Z * [new branch] gh/ezyang/3133/base -> origin/gh/ezyang/3133/base 2025-08-14T21:24:12.6971566Z * [new branch] gh/ezyang/3133/head -> origin/gh/ezyang/3133/head 2025-08-14T21:24:12.6974961Z * [new branch] gh/ezyang/3133/orig -> origin/gh/ezyang/3133/orig 2025-08-14T21:24:12.6975517Z * [new branch] gh/ezyang/3134/base -> origin/gh/ezyang/3134/base 2025-08-14T21:24:12.6978885Z * [new branch] gh/ezyang/3134/head -> origin/gh/ezyang/3134/head 2025-08-14T21:24:12.6979037Z * [new branch] gh/ezyang/3134/orig -> origin/gh/ezyang/3134/orig 2025-08-14T21:24:12.6979196Z * [new branch] gh/ezyang/3135/base -> origin/gh/ezyang/3135/base 2025-08-14T21:24:12.6979338Z * [new branch] gh/ezyang/3135/head -> origin/gh/ezyang/3135/head 2025-08-14T21:24:12.6979477Z * [new branch] gh/ezyang/3135/orig -> origin/gh/ezyang/3135/orig 2025-08-14T21:24:12.6979624Z * [new branch] gh/ezyang/3136/base -> origin/gh/ezyang/3136/base 2025-08-14T21:24:12.6979797Z * [new branch] gh/ezyang/3136/head -> origin/gh/ezyang/3136/head 2025-08-14T21:24:12.6979964Z * [new branch] gh/ezyang/3136/orig -> origin/gh/ezyang/3136/orig 2025-08-14T21:24:12.6982893Z * [new branch] gh/fadara01/1/base -> origin/gh/fadara01/1/base 2025-08-14T21:24:12.6983124Z * [new branch] gh/fadara01/1/head -> origin/gh/fadara01/1/head 2025-08-14T21:24:12.6983404Z * [new branch] gh/fadara01/1/orig -> origin/gh/fadara01/1/orig 2025-08-14T21:24:12.6983564Z * [new branch] gh/fduwjj/168/base -> origin/gh/fduwjj/168/base 2025-08-14T21:24:12.6983809Z * [new branch] gh/fduwjj/168/head -> origin/gh/fduwjj/168/head 2025-08-14T21:24:12.6984016Z * [new branch] gh/fduwjj/168/orig -> origin/gh/fduwjj/168/orig 2025-08-14T21:24:12.6987247Z * [new branch] gh/fduwjj/169/base -> origin/gh/fduwjj/169/base 2025-08-14T21:24:12.6987404Z * [new branch] gh/fduwjj/169/head -> origin/gh/fduwjj/169/head 2025-08-14T21:24:12.6987734Z * [new branch] gh/fduwjj/169/orig -> origin/gh/fduwjj/169/orig 2025-08-14T21:24:12.6988083Z * [new branch] gh/fduwjj/170/base -> origin/gh/fduwjj/170/base 2025-08-14T21:24:12.6993226Z * [new branch] gh/fduwjj/170/head -> origin/gh/fduwjj/170/head 2025-08-14T21:24:12.6997616Z * [new branch] gh/fduwjj/170/orig -> origin/gh/fduwjj/170/orig 2025-08-14T21:24:12.6997795Z * [new branch] gh/fduwjj/171/base -> origin/gh/fduwjj/171/base 2025-08-14T21:24:12.6997941Z * [new branch] gh/fduwjj/171/head -> origin/gh/fduwjj/171/head 2025-08-14T21:24:12.6998104Z * [new branch] gh/fduwjj/171/orig -> origin/gh/fduwjj/171/orig 2025-08-14T21:24:12.6998256Z * [new branch] gh/fduwjj/172/base -> origin/gh/fduwjj/172/base 2025-08-14T21:24:12.6998431Z * [new branch] gh/fduwjj/172/head -> origin/gh/fduwjj/172/head 2025-08-14T21:24:12.6998595Z * [new branch] gh/fduwjj/172/orig -> origin/gh/fduwjj/172/orig 2025-08-14T21:24:12.6998744Z * [new branch] gh/fduwjj/173/base -> origin/gh/fduwjj/173/base 2025-08-14T21:24:12.6998883Z * [new branch] gh/fduwjj/173/head -> origin/gh/fduwjj/173/head 2025-08-14T21:24:12.6999017Z * [new branch] gh/fduwjj/173/orig -> origin/gh/fduwjj/173/orig 2025-08-14T21:24:12.6999162Z * [new branch] gh/fduwjj/174/base -> origin/gh/fduwjj/174/base 2025-08-14T21:24:12.6999297Z * [new branch] gh/fduwjj/174/head -> origin/gh/fduwjj/174/head 2025-08-14T21:24:12.7000016Z * [new branch] gh/fduwjj/174/orig -> origin/gh/fduwjj/174/orig 2025-08-14T21:24:12.7000219Z * [new branch] gh/fduwjj/175/base -> origin/gh/fduwjj/175/base 2025-08-14T21:24:12.7001496Z * [new branch] gh/fduwjj/175/head -> origin/gh/fduwjj/175/head 2025-08-14T21:24:12.7001676Z * [new branch] gh/fduwjj/175/orig -> origin/gh/fduwjj/175/orig 2025-08-14T21:24:12.7003411Z * [new branch] gh/fduwjj/176/base -> origin/gh/fduwjj/176/base 2025-08-14T21:24:12.7003586Z * [new branch] gh/fduwjj/176/head -> origin/gh/fduwjj/176/head 2025-08-14T21:24:12.7004096Z * [new branch] gh/fduwjj/176/orig -> origin/gh/fduwjj/176/orig 2025-08-14T21:24:12.7005738Z * [new branch] gh/fduwjj/177/base -> origin/gh/fduwjj/177/base 2025-08-14T21:24:12.7005911Z * [new branch] gh/fduwjj/177/head -> origin/gh/fduwjj/177/head 2025-08-14T21:24:12.7006433Z * [new branch] gh/fduwjj/177/orig -> origin/gh/fduwjj/177/orig 2025-08-14T21:24:12.7011058Z * [new branch] gh/fduwjj/178/base -> origin/gh/fduwjj/178/base 2025-08-14T21:24:12.7011264Z * [new branch] gh/fduwjj/178/head -> origin/gh/fduwjj/178/head 2025-08-14T21:24:12.7011414Z * [new branch] gh/fduwjj/178/orig -> origin/gh/fduwjj/178/orig 2025-08-14T21:24:12.7011550Z * [new branch] gh/fduwjj/179/base -> origin/gh/fduwjj/179/base 2025-08-14T21:24:12.7011698Z * [new branch] gh/fduwjj/179/head -> origin/gh/fduwjj/179/head 2025-08-14T21:24:12.7011824Z * [new branch] gh/fduwjj/179/orig -> origin/gh/fduwjj/179/orig 2025-08-14T21:24:12.7012085Z * [new branch] gh/fduwjj/180/base -> origin/gh/fduwjj/180/base 2025-08-14T21:24:12.7016397Z * [new branch] gh/fduwjj/180/head -> origin/gh/fduwjj/180/head 2025-08-14T21:24:12.7016727Z * [new branch] gh/fduwjj/180/orig -> origin/gh/fduwjj/180/orig 2025-08-14T21:24:12.7016877Z * [new branch] gh/fduwjj/181/base -> origin/gh/fduwjj/181/base 2025-08-14T21:24:12.7018399Z * [new branch] gh/fduwjj/181/head -> origin/gh/fduwjj/181/head 2025-08-14T21:24:12.7018866Z * [new branch] gh/fduwjj/181/orig -> origin/gh/fduwjj/181/orig 2025-08-14T21:24:12.7019096Z * [new branch] gh/fegin/306/base -> origin/gh/fegin/306/base 2025-08-14T21:24:12.7019708Z * [new branch] gh/fegin/306/head -> origin/gh/fegin/306/head 2025-08-14T21:24:12.7019849Z * [new branch] gh/fegin/306/orig -> origin/gh/fegin/306/orig 2025-08-14T21:24:12.7019975Z * [new branch] gh/fegin/307/base -> origin/gh/fegin/307/base 2025-08-14T21:24:12.7020100Z * [new branch] gh/fegin/307/head -> origin/gh/fegin/307/head 2025-08-14T21:24:12.7025619Z * [new branch] gh/fegin/307/orig -> origin/gh/fegin/307/orig 2025-08-14T21:24:12.7026008Z * [new branch] gh/fffrog/114/base -> origin/gh/fffrog/114/base 2025-08-14T21:24:12.7026162Z * [new branch] gh/fffrog/114/head -> origin/gh/fffrog/114/head 2025-08-14T21:24:12.7026303Z * [new branch] gh/fffrog/114/orig -> origin/gh/fffrog/114/orig 2025-08-14T21:24:12.7026433Z * [new branch] gh/fffrog/117/base -> origin/gh/fffrog/117/base 2025-08-14T21:24:12.7026711Z * [new branch] gh/fffrog/117/head -> origin/gh/fffrog/117/head 2025-08-14T21:24:12.7026860Z * [new branch] gh/fffrog/117/orig -> origin/gh/fffrog/117/orig 2025-08-14T21:24:12.7031149Z * [new branch] gh/fffrog/119/base -> origin/gh/fffrog/119/base 2025-08-14T21:24:12.7031336Z * [new branch] gh/fffrog/119/head -> origin/gh/fffrog/119/head 2025-08-14T21:24:12.7031484Z * [new branch] gh/fffrog/119/orig -> origin/gh/fffrog/119/orig 2025-08-14T21:24:12.7031624Z * [new branch] gh/fffrog/120/base -> origin/gh/fffrog/120/base 2025-08-14T21:24:12.7031927Z * [new branch] gh/fffrog/120/head -> origin/gh/fffrog/120/head 2025-08-14T21:24:12.7032076Z * [new branch] gh/fffrog/120/orig -> origin/gh/fffrog/120/orig 2025-08-14T21:24:12.7036212Z * [new branch] gh/fffrog/121/base -> origin/gh/fffrog/121/base 2025-08-14T21:24:12.7041246Z * [new branch] gh/fffrog/121/head -> origin/gh/fffrog/121/head 2025-08-14T21:24:12.7041445Z * [new branch] gh/fffrog/121/orig -> origin/gh/fffrog/121/orig 2025-08-14T21:24:12.7041600Z * [new branch] gh/fffrog/122/base -> origin/gh/fffrog/122/base 2025-08-14T21:24:12.7041742Z * [new branch] gh/fffrog/122/head -> origin/gh/fffrog/122/head 2025-08-14T21:24:12.7041878Z * [new branch] gh/fffrog/122/orig -> origin/gh/fffrog/122/orig 2025-08-14T21:24:12.7042037Z * [new branch] gh/fffrog/123/base -> origin/gh/fffrog/123/base 2025-08-14T21:24:12.7042185Z * [new branch] gh/fffrog/123/head -> origin/gh/fffrog/123/head 2025-08-14T21:24:12.7042329Z * [new branch] gh/fffrog/123/orig -> origin/gh/fffrog/123/orig 2025-08-14T21:24:12.7042456Z * [new branch] gh/fffrog/124/base -> origin/gh/fffrog/124/base 2025-08-14T21:24:12.7042586Z * [new branch] gh/fffrog/124/head -> origin/gh/fffrog/124/head 2025-08-14T21:24:12.7042725Z * [new branch] gh/fffrog/124/orig -> origin/gh/fffrog/124/orig 2025-08-14T21:24:12.7042856Z * [new branch] gh/fffrog/125/base -> origin/gh/fffrog/125/base 2025-08-14T21:24:12.7042991Z * [new branch] gh/fffrog/125/head -> origin/gh/fffrog/125/head 2025-08-14T21:24:12.7043121Z * [new branch] gh/fffrog/125/orig -> origin/gh/fffrog/125/orig 2025-08-14T21:24:12.7043250Z * [new branch] gh/fffrog/126/base -> origin/gh/fffrog/126/base 2025-08-14T21:24:12.7043631Z * [new branch] gh/fffrog/126/head -> origin/gh/fffrog/126/head 2025-08-14T21:24:12.7043764Z * [new branch] gh/fffrog/126/orig -> origin/gh/fffrog/126/orig 2025-08-14T21:24:12.7043910Z * [new branch] gh/fffrog/127/base -> origin/gh/fffrog/127/base 2025-08-14T21:24:12.7044043Z * [new branch] gh/fffrog/127/head -> origin/gh/fffrog/127/head 2025-08-14T21:24:12.7044175Z * [new branch] gh/fffrog/127/orig -> origin/gh/fffrog/127/orig 2025-08-14T21:24:12.7044658Z * [new branch] gh/fffrog/128/base -> origin/gh/fffrog/128/base 2025-08-14T21:24:12.7045887Z * [new branch] gh/fffrog/128/head -> origin/gh/fffrog/128/head 2025-08-14T21:24:12.7046159Z * [new branch] gh/fffrog/128/orig -> origin/gh/fffrog/128/orig 2025-08-14T21:24:12.7052197Z * [new branch] gh/fffrog/129/base -> origin/gh/fffrog/129/base 2025-08-14T21:24:12.7054608Z * [new branch] gh/fffrog/129/head -> origin/gh/fffrog/129/head 2025-08-14T21:24:12.7061134Z * [new branch] gh/fffrog/129/orig -> origin/gh/fffrog/129/orig 2025-08-14T21:24:12.7063398Z * [new branch] gh/fffrog/130/base -> origin/gh/fffrog/130/base 2025-08-14T21:24:12.7069462Z * [new branch] gh/fffrog/130/head -> origin/gh/fffrog/130/head 2025-08-14T21:24:12.7074411Z * [new branch] gh/fffrog/130/orig -> origin/gh/fffrog/130/orig 2025-08-14T21:24:12.7079873Z * [new branch] gh/fffrog/131/base -> origin/gh/fffrog/131/base 2025-08-14T21:24:12.7081823Z * [new branch] gh/fffrog/131/head -> origin/gh/fffrog/131/head 2025-08-14T21:24:12.7082003Z * [new branch] gh/fffrog/131/orig -> origin/gh/fffrog/131/orig 2025-08-14T21:24:12.7082339Z * [new branch] gh/fffrog/132/base -> origin/gh/fffrog/132/base 2025-08-14T21:24:12.7082489Z * [new branch] gh/fffrog/132/head -> origin/gh/fffrog/132/head 2025-08-14T21:24:12.7082627Z * [new branch] gh/fffrog/132/orig -> origin/gh/fffrog/132/orig 2025-08-14T21:24:12.7082755Z * [new branch] gh/fffrog/133/base -> origin/gh/fffrog/133/base 2025-08-14T21:24:12.7082890Z * [new branch] gh/fffrog/133/head -> origin/gh/fffrog/133/head 2025-08-14T21:24:12.7083032Z * [new branch] gh/fffrog/133/orig -> origin/gh/fffrog/133/orig 2025-08-14T21:24:12.7083162Z * [new branch] gh/fffrog/134/base -> origin/gh/fffrog/134/base 2025-08-14T21:24:12.7083300Z * [new branch] gh/fffrog/134/head -> origin/gh/fffrog/134/head 2025-08-14T21:24:12.7083429Z * [new branch] gh/fffrog/134/orig -> origin/gh/fffrog/134/orig 2025-08-14T21:24:12.7083560Z * [new branch] gh/fffrog/135/base -> origin/gh/fffrog/135/base 2025-08-14T21:24:12.7083700Z * [new branch] gh/fffrog/135/head -> origin/gh/fffrog/135/head 2025-08-14T21:24:12.7083829Z * [new branch] gh/fffrog/135/orig -> origin/gh/fffrog/135/orig 2025-08-14T21:24:12.7083963Z * [new branch] gh/fffrog/136/base -> origin/gh/fffrog/136/base 2025-08-14T21:24:12.7084098Z * [new branch] gh/fffrog/136/head -> origin/gh/fffrog/136/head 2025-08-14T21:24:12.7084220Z * [new branch] gh/fffrog/136/orig -> origin/gh/fffrog/136/orig 2025-08-14T21:24:12.7084350Z * [new branch] gh/fffrog/137/base -> origin/gh/fffrog/137/base 2025-08-14T21:24:12.7084471Z * [new branch] gh/fffrog/137/head -> origin/gh/fffrog/137/head 2025-08-14T21:24:12.7084595Z * [new branch] gh/fffrog/137/orig -> origin/gh/fffrog/137/orig 2025-08-14T21:24:12.7084723Z * [new branch] gh/fffrog/138/base -> origin/gh/fffrog/138/base 2025-08-14T21:24:12.7084900Z * [new branch] gh/fffrog/138/head -> origin/gh/fffrog/138/head 2025-08-14T21:24:12.7085031Z * [new branch] gh/fffrog/138/orig -> origin/gh/fffrog/138/orig 2025-08-14T21:24:12.7085174Z * [new branch] gh/gmagogsfm/1/base -> origin/gh/gmagogsfm/1/base 2025-08-14T21:24:12.7085314Z * [new branch] gh/gmagogsfm/1/head -> origin/gh/gmagogsfm/1/head 2025-08-14T21:24:12.7085457Z * [new branch] gh/gmagogsfm/1/orig -> origin/gh/gmagogsfm/1/orig 2025-08-14T21:24:12.7085774Z * [new branch] gh/gmagogsfm/2/base -> origin/gh/gmagogsfm/2/base 2025-08-14T21:24:12.7085928Z * [new branch] gh/gmagogsfm/2/head -> origin/gh/gmagogsfm/2/head 2025-08-14T21:24:12.7086063Z * [new branch] gh/gmagogsfm/2/orig -> origin/gh/gmagogsfm/2/orig 2025-08-14T21:24:12.7086202Z * [new branch] gh/gmagogsfm/3/base -> origin/gh/gmagogsfm/3/base 2025-08-14T21:24:12.7086348Z * [new branch] gh/gmagogsfm/3/head -> origin/gh/gmagogsfm/3/head 2025-08-14T21:24:12.7086486Z * [new branch] gh/gmagogsfm/3/orig -> origin/gh/gmagogsfm/3/orig 2025-08-14T21:24:12.7086620Z * [new branch] gh/gmagogsfm/4/base -> origin/gh/gmagogsfm/4/base 2025-08-14T21:24:12.7086765Z * [new branch] gh/gmagogsfm/4/head -> origin/gh/gmagogsfm/4/head 2025-08-14T21:24:12.7086901Z * [new branch] gh/gmagogsfm/4/orig -> origin/gh/gmagogsfm/4/orig 2025-08-14T21:24:12.7087053Z * [new branch] gh/guangyey/130/base -> origin/gh/guangyey/130/base 2025-08-14T21:24:12.7087190Z * [new branch] gh/guangyey/130/head -> origin/gh/guangyey/130/head 2025-08-14T21:24:12.7087332Z * [new branch] gh/guangyey/130/orig -> origin/gh/guangyey/130/orig 2025-08-14T21:24:12.7087513Z * [new branch] gh/guangyey/133/base -> origin/gh/guangyey/133/base 2025-08-14T21:24:12.7087647Z * [new branch] gh/guangyey/133/head -> origin/gh/guangyey/133/head 2025-08-14T21:24:12.7087787Z * [new branch] gh/guangyey/133/orig -> origin/gh/guangyey/133/orig 2025-08-14T21:24:12.7087914Z * [new branch] gh/guangyey/134/base -> origin/gh/guangyey/134/base 2025-08-14T21:24:12.7088052Z * [new branch] gh/guangyey/134/head -> origin/gh/guangyey/134/head 2025-08-14T21:24:12.7088190Z * [new branch] gh/guangyey/134/orig -> origin/gh/guangyey/134/orig 2025-08-14T21:24:12.7088317Z * [new branch] gh/guangyey/135/base -> origin/gh/guangyey/135/base 2025-08-14T21:24:12.7088453Z * [new branch] gh/guangyey/135/head -> origin/gh/guangyey/135/head 2025-08-14T21:24:12.7088590Z * [new branch] gh/guangyey/135/orig -> origin/gh/guangyey/135/orig 2025-08-14T21:24:12.7088717Z * [new branch] gh/guangyey/139/base -> origin/gh/guangyey/139/base 2025-08-14T21:24:12.7088853Z * [new branch] gh/guangyey/139/head -> origin/gh/guangyey/139/head 2025-08-14T21:24:12.7088977Z * [new branch] gh/guangyey/139/orig -> origin/gh/guangyey/139/orig 2025-08-14T21:24:12.7089289Z * [new branch] gh/guangyey/140/base -> origin/gh/guangyey/140/base 2025-08-14T21:24:12.7089923Z * [new branch] gh/guangyey/140/head -> origin/gh/guangyey/140/head 2025-08-14T21:24:12.7090265Z * [new branch] gh/guangyey/140/orig -> origin/gh/guangyey/140/orig 2025-08-14T21:24:12.7091701Z * [new branch] gh/guangyey/142/base -> origin/gh/guangyey/142/base 2025-08-14T21:24:12.7096485Z * [new branch] gh/guangyey/142/head -> origin/gh/guangyey/142/head 2025-08-14T21:24:12.7096682Z * [new branch] gh/guangyey/142/orig -> origin/gh/guangyey/142/orig 2025-08-14T21:24:12.7096993Z * [new branch] gh/guangyey/145/base -> origin/gh/guangyey/145/base 2025-08-14T21:24:12.7097142Z * [new branch] gh/guangyey/145/head -> origin/gh/guangyey/145/head 2025-08-14T21:24:12.7097278Z * [new branch] gh/guangyey/145/orig -> origin/gh/guangyey/145/orig 2025-08-14T21:24:12.7097425Z * [new branch] gh/guangyey/153/base -> origin/gh/guangyey/153/base 2025-08-14T21:24:12.7097558Z * [new branch] gh/guangyey/153/head -> origin/gh/guangyey/153/head 2025-08-14T21:24:12.7097698Z * [new branch] gh/guangyey/153/orig -> origin/gh/guangyey/153/orig 2025-08-14T21:24:12.7097832Z * [new branch] gh/guangyey/158/base -> origin/gh/guangyey/158/base 2025-08-14T21:24:12.7098004Z * [new branch] gh/guangyey/158/head -> origin/gh/guangyey/158/head 2025-08-14T21:24:12.7098814Z * [new branch] gh/guangyey/158/orig -> origin/gh/guangyey/158/orig 2025-08-14T21:24:12.7099653Z * [new branch] gh/guangyey/159/base -> origin/gh/guangyey/159/base 2025-08-14T21:24:12.7100087Z * [new branch] gh/guangyey/159/head -> origin/gh/guangyey/159/head 2025-08-14T21:24:12.7101009Z * [new branch] gh/guangyey/159/orig -> origin/gh/guangyey/159/orig 2025-08-14T21:24:12.7101945Z * [new branch] gh/guangyey/163/base -> origin/gh/guangyey/163/base 2025-08-14T21:24:12.7102344Z * [new branch] gh/guangyey/163/head -> origin/gh/guangyey/163/head 2025-08-14T21:24:12.7103253Z * [new branch] gh/guangyey/163/orig -> origin/gh/guangyey/163/orig 2025-08-14T21:24:12.7104157Z * [new branch] gh/guangyey/165/base -> origin/gh/guangyey/165/base 2025-08-14T21:24:12.7104548Z * [new branch] gh/guangyey/165/head -> origin/gh/guangyey/165/head 2025-08-14T21:24:12.7105964Z * [new branch] gh/guangyey/165/orig -> origin/gh/guangyey/165/orig 2025-08-14T21:24:12.7106469Z * [new branch] gh/guangyey/168/base -> origin/gh/guangyey/168/base 2025-08-14T21:24:12.7107193Z * [new branch] gh/guangyey/168/head -> origin/gh/guangyey/168/head 2025-08-14T21:24:12.7107741Z * [new branch] gh/guangyey/168/orig -> origin/gh/guangyey/168/orig 2025-08-14T21:24:12.7108865Z * [new branch] gh/guangyey/169/base -> origin/gh/guangyey/169/base 2025-08-14T21:24:12.7109139Z * [new branch] gh/guangyey/169/head -> origin/gh/guangyey/169/head 2025-08-14T21:24:12.7111286Z * [new branch] gh/guangyey/169/orig -> origin/gh/guangyey/169/orig 2025-08-14T21:24:12.7111468Z * [new branch] gh/guangyey/170/base -> origin/gh/guangyey/170/base 2025-08-14T21:24:12.7111627Z * [new branch] gh/guangyey/170/head -> origin/gh/guangyey/170/head 2025-08-14T21:24:12.7113867Z * [new branch] gh/guangyey/170/orig -> origin/gh/guangyey/170/orig 2025-08-14T21:24:12.7114212Z * [new branch] gh/guangyey/171/base -> origin/gh/guangyey/171/base 2025-08-14T21:24:12.7114401Z * [new branch] gh/guangyey/171/head -> origin/gh/guangyey/171/head 2025-08-14T21:24:12.7114746Z * [new branch] gh/guangyey/171/orig -> origin/gh/guangyey/171/orig 2025-08-14T21:24:12.7117102Z * [new branch] gh/guangyey/172/base -> origin/gh/guangyey/172/base 2025-08-14T21:24:12.7117292Z * [new branch] gh/guangyey/172/head -> origin/gh/guangyey/172/head 2025-08-14T21:24:12.7117438Z * [new branch] gh/guangyey/172/orig -> origin/gh/guangyey/172/orig 2025-08-14T21:24:12.7118150Z * [new branch] gh/guangyey/173/base -> origin/gh/guangyey/173/base 2025-08-14T21:24:12.7118633Z * [new branch] gh/guangyey/173/head -> origin/gh/guangyey/173/head 2025-08-14T21:24:12.7119435Z * [new branch] gh/guangyey/173/orig -> origin/gh/guangyey/173/orig 2025-08-14T21:24:12.7120672Z * [new branch] gh/guangyey/174/base -> origin/gh/guangyey/174/base 2025-08-14T21:24:12.7120993Z * [new branch] gh/guangyey/174/head -> origin/gh/guangyey/174/head 2025-08-14T21:24:12.7121969Z * [new branch] gh/guangyey/174/orig -> origin/gh/guangyey/174/orig 2025-08-14T21:24:12.7123013Z * [new branch] gh/guangyey/175/base -> origin/gh/guangyey/175/base 2025-08-14T21:24:12.7123231Z * [new branch] gh/guangyey/175/head -> origin/gh/guangyey/175/head 2025-08-14T21:24:12.7124663Z * [new branch] gh/guangyey/175/orig -> origin/gh/guangyey/175/orig 2025-08-14T21:24:12.7124895Z * [new branch] gh/guangyey/176/base -> origin/gh/guangyey/176/base 2025-08-14T21:24:12.7126024Z * [new branch] gh/guangyey/176/head -> origin/gh/guangyey/176/head 2025-08-14T21:24:12.7126200Z * [new branch] gh/guangyey/176/orig -> origin/gh/guangyey/176/orig 2025-08-14T21:24:12.7133352Z * [new branch] gh/guangyey/177/base -> origin/gh/guangyey/177/base 2025-08-14T21:24:12.7138482Z * [new branch] gh/guangyey/177/head -> origin/gh/guangyey/177/head 2025-08-14T21:24:12.7143722Z * [new branch] gh/guangyey/177/orig -> origin/gh/guangyey/177/orig 2025-08-14T21:24:12.7149249Z * [new branch] gh/guangyey/178/base -> origin/gh/guangyey/178/base 2025-08-14T21:24:12.7154079Z * [new branch] gh/guangyey/178/head -> origin/gh/guangyey/178/head 2025-08-14T21:24:12.7158483Z * [new branch] gh/guangyey/178/orig -> origin/gh/guangyey/178/orig 2025-08-14T21:24:12.7158809Z * [new branch] gh/guangyey/179/base -> origin/gh/guangyey/179/base 2025-08-14T21:24:12.7159202Z * [new branch] gh/guangyey/179/head -> origin/gh/guangyey/179/head 2025-08-14T21:24:12.7159488Z * [new branch] gh/guangyey/179/orig -> origin/gh/guangyey/179/orig 2025-08-14T21:24:12.7159646Z * [new branch] gh/guangyey/180/base -> origin/gh/guangyey/180/base 2025-08-14T21:24:12.7160344Z * [new branch] gh/guangyey/180/head -> origin/gh/guangyey/180/head 2025-08-14T21:24:12.7160691Z * [new branch] gh/guangyey/180/orig -> origin/gh/guangyey/180/orig 2025-08-14T21:24:12.7160936Z * [new branch] gh/guangyey/181/base -> origin/gh/guangyey/181/base 2025-08-14T21:24:12.7161113Z * [new branch] gh/guangyey/181/head -> origin/gh/guangyey/181/head 2025-08-14T21:24:12.7161273Z * [new branch] gh/guangyey/181/orig -> origin/gh/guangyey/181/orig 2025-08-14T21:24:12.7161548Z * [new branch] gh/guangyey/182/base -> origin/gh/guangyey/182/base 2025-08-14T21:24:12.7161897Z * [new branch] gh/guangyey/182/head -> origin/gh/guangyey/182/head 2025-08-14T21:24:12.7162061Z * [new branch] gh/guangyey/182/orig -> origin/gh/guangyey/182/orig 2025-08-14T21:24:12.7162212Z * [new branch] gh/guangyey/183/base -> origin/gh/guangyey/183/base 2025-08-14T21:24:12.7162365Z * [new branch] gh/guangyey/183/head -> origin/gh/guangyey/183/head 2025-08-14T21:24:12.7162507Z * [new branch] gh/guangyey/183/orig -> origin/gh/guangyey/183/orig 2025-08-14T21:24:12.7162683Z * [new branch] gh/guangyey/184/base -> origin/gh/guangyey/184/base 2025-08-14T21:24:12.7162827Z * [new branch] gh/guangyey/184/head -> origin/gh/guangyey/184/head 2025-08-14T21:24:12.7162968Z * [new branch] gh/guangyey/184/orig -> origin/gh/guangyey/184/orig 2025-08-14T21:24:12.7163121Z * [new branch] gh/guangyey/185/base -> origin/gh/guangyey/185/base 2025-08-14T21:24:12.7163492Z * [new branch] gh/guangyey/185/head -> origin/gh/guangyey/185/head 2025-08-14T21:24:12.7163644Z * [new branch] gh/guangyey/185/orig -> origin/gh/guangyey/185/orig 2025-08-14T21:24:12.7163791Z * [new branch] gh/guangyey/79/base -> origin/gh/guangyey/79/base 2025-08-14T21:24:12.7163937Z * [new branch] gh/guangyey/79/head -> origin/gh/guangyey/79/head 2025-08-14T21:24:12.7164088Z * [new branch] gh/guangyey/79/orig -> origin/gh/guangyey/79/orig 2025-08-14T21:24:12.7164233Z * [new branch] gh/guangyey/89/base -> origin/gh/guangyey/89/base 2025-08-14T21:24:12.7164380Z * [new branch] gh/guangyey/89/head -> origin/gh/guangyey/89/head 2025-08-14T21:24:12.7164524Z * [new branch] gh/guangyey/89/orig -> origin/gh/guangyey/89/orig 2025-08-14T21:24:12.7164721Z * [new branch] gh/guilhermeleobas/107/base -> origin/gh/guilhermeleobas/107/base 2025-08-14T21:24:12.7164912Z * [new branch] gh/guilhermeleobas/107/head -> origin/gh/guilhermeleobas/107/head 2025-08-14T21:24:12.7165086Z * [new branch] gh/guilhermeleobas/107/orig -> origin/gh/guilhermeleobas/107/orig 2025-08-14T21:24:12.7165266Z * [new branch] gh/guilhermeleobas/108/base -> origin/gh/guilhermeleobas/108/base 2025-08-14T21:24:12.7165437Z * [new branch] gh/guilhermeleobas/108/head -> origin/gh/guilhermeleobas/108/head 2025-08-14T21:24:12.7165709Z * [new branch] gh/guilhermeleobas/108/orig -> origin/gh/guilhermeleobas/108/orig 2025-08-14T21:24:12.7165894Z * [new branch] gh/guilhermeleobas/124/base -> origin/gh/guilhermeleobas/124/base 2025-08-14T21:24:12.7166065Z * [new branch] gh/guilhermeleobas/124/head -> origin/gh/guilhermeleobas/124/head 2025-08-14T21:24:12.7166338Z * [new branch] gh/guilhermeleobas/124/orig -> origin/gh/guilhermeleobas/124/orig 2025-08-14T21:24:12.7166525Z * [new branch] gh/guilhermeleobas/147/base -> origin/gh/guilhermeleobas/147/base 2025-08-14T21:24:12.7166702Z * [new branch] gh/guilhermeleobas/147/head -> origin/gh/guilhermeleobas/147/head 2025-08-14T21:24:12.7166879Z * [new branch] gh/guilhermeleobas/147/orig -> origin/gh/guilhermeleobas/147/orig 2025-08-14T21:24:12.7167104Z * [new branch] gh/guilhermeleobas/150/base -> origin/gh/guilhermeleobas/150/base 2025-08-14T21:24:12.7167409Z * [new branch] gh/guilhermeleobas/150/head -> origin/gh/guilhermeleobas/150/head 2025-08-14T21:24:12.7167574Z * [new branch] gh/guilhermeleobas/150/orig -> origin/gh/guilhermeleobas/150/orig 2025-08-14T21:24:12.7169618Z * [new branch] gh/guilhermeleobas/163/base -> origin/gh/guilhermeleobas/163/base 2025-08-14T21:24:12.7169982Z * [new branch] gh/guilhermeleobas/163/head -> origin/gh/guilhermeleobas/163/head 2025-08-14T21:24:12.7170250Z * [new branch] gh/guilhermeleobas/163/orig -> origin/gh/guilhermeleobas/163/orig 2025-08-14T21:24:12.7170445Z * [new branch] gh/guilhermeleobas/164/base -> origin/gh/guilhermeleobas/164/base 2025-08-14T21:24:12.7172072Z * [new branch] gh/guilhermeleobas/164/head -> origin/gh/guilhermeleobas/164/head 2025-08-14T21:24:12.7172420Z * [new branch] gh/guilhermeleobas/164/orig -> origin/gh/guilhermeleobas/164/orig 2025-08-14T21:24:12.7172678Z * [new branch] gh/guilhermeleobas/165/base -> origin/gh/guilhermeleobas/165/base 2025-08-14T21:24:12.7174418Z * [new branch] gh/guilhermeleobas/165/head -> origin/gh/guilhermeleobas/165/head 2025-08-14T21:24:12.7174787Z * [new branch] gh/guilhermeleobas/165/orig -> origin/gh/guilhermeleobas/165/orig 2025-08-14T21:24:12.7175041Z * [new branch] gh/guilhermeleobas/166/base -> origin/gh/guilhermeleobas/166/base 2025-08-14T21:24:12.7175399Z * [new branch] gh/guilhermeleobas/166/head -> origin/gh/guilhermeleobas/166/head 2025-08-14T21:24:12.7176808Z * [new branch] gh/guilhermeleobas/166/orig -> origin/gh/guilhermeleobas/166/orig 2025-08-14T21:24:12.7177140Z * [new branch] gh/guilhermeleobas/167/base -> origin/gh/guilhermeleobas/167/base 2025-08-14T21:24:12.7177827Z * [new branch] gh/guilhermeleobas/167/head -> origin/gh/guilhermeleobas/167/head 2025-08-14T21:24:12.7178317Z * [new branch] gh/guilhermeleobas/167/orig -> origin/gh/guilhermeleobas/167/orig 2025-08-14T21:24:12.7180322Z * [new branch] gh/guilhermeleobas/168/base -> origin/gh/guilhermeleobas/168/base 2025-08-14T21:24:12.7180643Z * [new branch] gh/guilhermeleobas/168/head -> origin/gh/guilhermeleobas/168/head 2025-08-14T21:24:12.7180848Z * [new branch] gh/guilhermeleobas/168/orig -> origin/gh/guilhermeleobas/168/orig 2025-08-14T21:24:12.7182238Z * [new branch] gh/guilhermeleobas/169/base -> origin/gh/guilhermeleobas/169/base 2025-08-14T21:24:12.7182620Z * [new branch] gh/guilhermeleobas/169/head -> origin/gh/guilhermeleobas/169/head 2025-08-14T21:24:12.7182873Z * [new branch] gh/guilhermeleobas/169/orig -> origin/gh/guilhermeleobas/169/orig 2025-08-14T21:24:12.7186260Z * [new branch] gh/guilhermeleobas/170/base -> origin/gh/guilhermeleobas/170/base 2025-08-14T21:24:12.7186604Z * [new branch] gh/guilhermeleobas/170/head -> origin/gh/guilhermeleobas/170/head 2025-08-14T21:24:12.7186800Z * [new branch] gh/guilhermeleobas/170/orig -> origin/gh/guilhermeleobas/170/orig 2025-08-14T21:24:12.7187036Z * [new branch] gh/guilhermeleobas/171/base -> origin/gh/guilhermeleobas/171/base 2025-08-14T21:24:12.7187210Z * [new branch] gh/guilhermeleobas/171/head -> origin/gh/guilhermeleobas/171/head 2025-08-14T21:24:12.7187752Z * [new branch] gh/guilhermeleobas/171/orig -> origin/gh/guilhermeleobas/171/orig 2025-08-14T21:24:12.7189105Z * [new branch] gh/guilhermeleobas/173/base -> origin/gh/guilhermeleobas/173/base 2025-08-14T21:24:12.7189384Z * [new branch] gh/guilhermeleobas/173/head -> origin/gh/guilhermeleobas/173/head 2025-08-14T21:24:12.7189809Z * [new branch] gh/guilhermeleobas/173/orig -> origin/gh/guilhermeleobas/173/orig 2025-08-14T21:24:12.7191515Z * [new branch] gh/guilhermeleobas/181/base -> origin/gh/guilhermeleobas/181/base 2025-08-14T21:24:12.7191868Z * [new branch] gh/guilhermeleobas/181/head -> origin/gh/guilhermeleobas/181/head 2025-08-14T21:24:12.7192113Z * [new branch] gh/guilhermeleobas/181/orig -> origin/gh/guilhermeleobas/181/orig 2025-08-14T21:24:12.7196070Z * [new branch] gh/guilhermeleobas/182/base -> origin/gh/guilhermeleobas/182/base 2025-08-14T21:24:12.7196268Z * [new branch] gh/guilhermeleobas/182/head -> origin/gh/guilhermeleobas/182/head 2025-08-14T21:24:12.7196441Z * [new branch] gh/guilhermeleobas/182/orig -> origin/gh/guilhermeleobas/182/orig 2025-08-14T21:24:12.7196603Z * [new branch] gh/guilhermeleobas/183/base -> origin/gh/guilhermeleobas/183/base 2025-08-14T21:24:12.7196766Z * [new branch] gh/guilhermeleobas/183/head -> origin/gh/guilhermeleobas/183/head 2025-08-14T21:24:12.7197379Z * [new branch] gh/guilhermeleobas/183/orig -> origin/gh/guilhermeleobas/183/orig 2025-08-14T21:24:12.7198520Z * [new branch] gh/guilhermeleobas/184/base -> origin/gh/guilhermeleobas/184/base 2025-08-14T21:24:12.7198724Z * [new branch] gh/guilhermeleobas/184/head -> origin/gh/guilhermeleobas/184/head 2025-08-14T21:24:12.7199939Z * [new branch] gh/guilhermeleobas/184/orig -> origin/gh/guilhermeleobas/184/orig 2025-08-14T21:24:12.7200661Z * [new branch] gh/guilhermeleobas/185/base -> origin/gh/guilhermeleobas/185/base 2025-08-14T21:24:12.7201380Z * [new branch] gh/guilhermeleobas/185/head -> origin/gh/guilhermeleobas/185/head 2025-08-14T21:24:12.7202066Z * [new branch] gh/guilhermeleobas/185/orig -> origin/gh/guilhermeleobas/185/orig 2025-08-14T21:24:12.7203266Z * [new branch] gh/guilhermeleobas/188/base -> origin/gh/guilhermeleobas/188/base 2025-08-14T21:24:12.7203671Z * [new branch] gh/guilhermeleobas/188/head -> origin/gh/guilhermeleobas/188/head 2025-08-14T21:24:12.7204587Z * [new branch] gh/guilhermeleobas/188/orig -> origin/gh/guilhermeleobas/188/orig 2025-08-14T21:24:12.7207096Z * [new branch] gh/guilhermeleobas/189/base -> origin/gh/guilhermeleobas/189/base 2025-08-14T21:24:12.7207430Z * [new branch] gh/guilhermeleobas/189/head -> origin/gh/guilhermeleobas/189/head 2025-08-14T21:24:12.7207608Z * [new branch] gh/guilhermeleobas/189/orig -> origin/gh/guilhermeleobas/189/orig 2025-08-14T21:24:12.7212471Z * [new branch] gh/guilhermeleobas/190/base -> origin/gh/guilhermeleobas/190/base 2025-08-14T21:24:12.7212678Z * [new branch] gh/guilhermeleobas/190/head -> origin/gh/guilhermeleobas/190/head 2025-08-14T21:24:12.7212838Z * [new branch] gh/guilhermeleobas/190/orig -> origin/gh/guilhermeleobas/190/orig 2025-08-14T21:24:12.7212998Z * [new branch] gh/guilhermeleobas/192/base -> origin/gh/guilhermeleobas/192/base 2025-08-14T21:24:12.7213149Z * [new branch] gh/guilhermeleobas/192/head -> origin/gh/guilhermeleobas/192/head 2025-08-14T21:24:12.7213984Z * [new branch] gh/guilhermeleobas/192/orig -> origin/gh/guilhermeleobas/192/orig 2025-08-14T21:24:12.7214147Z * [new branch] gh/guilhermeleobas/193/base -> origin/gh/guilhermeleobas/193/base 2025-08-14T21:24:12.7214488Z * [new branch] gh/guilhermeleobas/193/head -> origin/gh/guilhermeleobas/193/head 2025-08-14T21:24:12.7214832Z * [new branch] gh/guilhermeleobas/193/orig -> origin/gh/guilhermeleobas/193/orig 2025-08-14T21:24:12.7215006Z * [new branch] gh/guilhermeleobas/194/base -> origin/gh/guilhermeleobas/194/base 2025-08-14T21:24:12.7215166Z * [new branch] gh/guilhermeleobas/194/head -> origin/gh/guilhermeleobas/194/head 2025-08-14T21:24:12.7223790Z * [new branch] gh/guilhermeleobas/194/orig -> origin/gh/guilhermeleobas/194/orig 2025-08-14T21:24:12.7223982Z * [new branch] gh/guilhermeleobas/203/base -> origin/gh/guilhermeleobas/203/base 2025-08-14T21:24:12.7224151Z * [new branch] gh/guilhermeleobas/203/head -> origin/gh/guilhermeleobas/203/head 2025-08-14T21:24:12.7224304Z * [new branch] gh/guilhermeleobas/203/orig -> origin/gh/guilhermeleobas/203/orig 2025-08-14T21:24:12.7224466Z * [new branch] gh/guilhermeleobas/204/base -> origin/gh/guilhermeleobas/204/base 2025-08-14T21:24:12.7224632Z * [new branch] gh/guilhermeleobas/204/head -> origin/gh/guilhermeleobas/204/head 2025-08-14T21:24:12.7224796Z * [new branch] gh/guilhermeleobas/204/orig -> origin/gh/guilhermeleobas/204/orig 2025-08-14T21:24:12.7224971Z * [new branch] gh/guilhermeleobas/205/base -> origin/gh/guilhermeleobas/205/base 2025-08-14T21:24:12.7225125Z * [new branch] gh/guilhermeleobas/205/head -> origin/gh/guilhermeleobas/205/head 2025-08-14T21:24:12.7225283Z * [new branch] gh/guilhermeleobas/205/orig -> origin/gh/guilhermeleobas/205/orig 2025-08-14T21:24:12.7225437Z * [new branch] gh/guilhermeleobas/206/base -> origin/gh/guilhermeleobas/206/base 2025-08-14T21:24:12.7225587Z * [new branch] gh/guilhermeleobas/206/head -> origin/gh/guilhermeleobas/206/head 2025-08-14T21:24:12.7225745Z * [new branch] gh/guilhermeleobas/206/orig -> origin/gh/guilhermeleobas/206/orig 2025-08-14T21:24:12.7230469Z * [new branch] gh/guilhermeleobas/207/base -> origin/gh/guilhermeleobas/207/base 2025-08-14T21:24:12.7230779Z * [new branch] gh/guilhermeleobas/207/head -> origin/gh/guilhermeleobas/207/head 2025-08-14T21:24:12.7231211Z * [new branch] gh/guilhermeleobas/207/orig -> origin/gh/guilhermeleobas/207/orig 2025-08-14T21:24:12.7231397Z * [new branch] gh/guilhermeleobas/208/base -> origin/gh/guilhermeleobas/208/base 2025-08-14T21:24:12.7231583Z * [new branch] gh/guilhermeleobas/208/head -> origin/gh/guilhermeleobas/208/head 2025-08-14T21:24:12.7232159Z * [new branch] gh/guilhermeleobas/208/orig -> origin/gh/guilhermeleobas/208/orig 2025-08-14T21:24:12.7232401Z * [new branch] gh/guilhermeleobas/209/base -> origin/gh/guilhermeleobas/209/base 2025-08-14T21:24:12.7232736Z * [new branch] gh/guilhermeleobas/209/head -> origin/gh/guilhermeleobas/209/head 2025-08-14T21:24:12.7232889Z * [new branch] gh/guilhermeleobas/209/orig -> origin/gh/guilhermeleobas/209/orig 2025-08-14T21:24:12.7233068Z * [new branch] gh/guilhermeleobas/210/base -> origin/gh/guilhermeleobas/210/base 2025-08-14T21:24:12.7233419Z * [new branch] gh/guilhermeleobas/210/head -> origin/gh/guilhermeleobas/210/head 2025-08-14T21:24:12.7233616Z * [new branch] gh/guilhermeleobas/210/orig -> origin/gh/guilhermeleobas/210/orig 2025-08-14T21:24:12.7233824Z * [new branch] gh/guilhermeleobas/211/base -> origin/gh/guilhermeleobas/211/base 2025-08-14T21:24:12.7234109Z * [new branch] gh/guilhermeleobas/211/head -> origin/gh/guilhermeleobas/211/head 2025-08-14T21:24:12.7235028Z * [new branch] gh/guilhermeleobas/211/orig -> origin/gh/guilhermeleobas/211/orig 2025-08-14T21:24:12.7237324Z * [new branch] gh/guilhermeleobas/212/base -> origin/gh/guilhermeleobas/212/base 2025-08-14T21:24:12.7237829Z * [new branch] gh/guilhermeleobas/212/head -> origin/gh/guilhermeleobas/212/head 2025-08-14T21:24:12.7238032Z * [new branch] gh/guilhermeleobas/212/orig -> origin/gh/guilhermeleobas/212/orig 2025-08-14T21:24:12.7238443Z * [new branch] gh/guilhermeleobas/213/base -> origin/gh/guilhermeleobas/213/base 2025-08-14T21:24:12.7239172Z * [new branch] gh/guilhermeleobas/213/head -> origin/gh/guilhermeleobas/213/head 2025-08-14T21:24:12.7239680Z * [new branch] gh/guilhermeleobas/213/orig -> origin/gh/guilhermeleobas/213/orig 2025-08-14T21:24:12.7241873Z * [new branch] gh/guilhermeleobas/214/base -> origin/gh/guilhermeleobas/214/base 2025-08-14T21:24:12.7242106Z * [new branch] gh/guilhermeleobas/214/head -> origin/gh/guilhermeleobas/214/head 2025-08-14T21:24:12.7242288Z * [new branch] gh/guilhermeleobas/214/orig -> origin/gh/guilhermeleobas/214/orig 2025-08-14T21:24:12.7243545Z * [new branch] gh/guilhermeleobas/215/base -> origin/gh/guilhermeleobas/215/base 2025-08-14T21:24:12.7244203Z * [new branch] gh/guilhermeleobas/215/head -> origin/gh/guilhermeleobas/215/head 2025-08-14T21:24:12.7244815Z * [new branch] gh/guilhermeleobas/215/orig -> origin/gh/guilhermeleobas/215/orig 2025-08-14T21:24:12.7246027Z * [new branch] gh/guilhermeleobas/216/base -> origin/gh/guilhermeleobas/216/base 2025-08-14T21:24:12.7246354Z * [new branch] gh/guilhermeleobas/216/head -> origin/gh/guilhermeleobas/216/head 2025-08-14T21:24:12.7247434Z * [new branch] gh/guilhermeleobas/216/orig -> origin/gh/guilhermeleobas/216/orig 2025-08-14T21:24:12.7248058Z * [new branch] gh/guilhermeleobas/217/base -> origin/gh/guilhermeleobas/217/base 2025-08-14T21:24:12.7249053Z * [new branch] gh/guilhermeleobas/217/head -> origin/gh/guilhermeleobas/217/head 2025-08-14T21:24:12.7249361Z * [new branch] gh/guilhermeleobas/217/orig -> origin/gh/guilhermeleobas/217/orig 2025-08-14T21:24:12.7251287Z * [new branch] gh/guilhermeleobas/218/base -> origin/gh/guilhermeleobas/218/base 2025-08-14T21:24:12.7251486Z * [new branch] gh/guilhermeleobas/218/head -> origin/gh/guilhermeleobas/218/head 2025-08-14T21:24:12.7251853Z * [new branch] gh/guilhermeleobas/218/orig -> origin/gh/guilhermeleobas/218/orig 2025-08-14T21:24:12.7253676Z * [new branch] gh/guilhermeleobas/219/base -> origin/gh/guilhermeleobas/219/base 2025-08-14T21:24:12.7253884Z * [new branch] gh/guilhermeleobas/219/head -> origin/gh/guilhermeleobas/219/head 2025-08-14T21:24:12.7254226Z * [new branch] gh/guilhermeleobas/219/orig -> origin/gh/guilhermeleobas/219/orig 2025-08-14T21:24:12.7258659Z * [new branch] gh/guilhermeleobas/220/base -> origin/gh/guilhermeleobas/220/base 2025-08-14T21:24:12.7258874Z * [new branch] gh/guilhermeleobas/220/head -> origin/gh/guilhermeleobas/220/head 2025-08-14T21:24:12.7259152Z * [new branch] gh/guilhermeleobas/220/orig -> origin/gh/guilhermeleobas/220/orig 2025-08-14T21:24:12.7259315Z * [new branch] gh/guilhermeleobas/221/base -> origin/gh/guilhermeleobas/221/base 2025-08-14T21:24:12.7259479Z * [new branch] gh/guilhermeleobas/221/head -> origin/gh/guilhermeleobas/221/head 2025-08-14T21:24:12.7259633Z * [new branch] gh/guilhermeleobas/221/orig -> origin/gh/guilhermeleobas/221/orig 2025-08-14T21:24:12.7267956Z * [new branch] gh/guilhermeleobas/222/base -> origin/gh/guilhermeleobas/222/base 2025-08-14T21:24:12.7272152Z * [new branch] gh/guilhermeleobas/222/head -> origin/gh/guilhermeleobas/222/head 2025-08-14T21:24:12.7272375Z * [new branch] gh/guilhermeleobas/222/orig -> origin/gh/guilhermeleobas/222/orig 2025-08-14T21:24:12.7272549Z * [new branch] gh/guilhermeleobas/223/base -> origin/gh/guilhermeleobas/223/base 2025-08-14T21:24:12.7272732Z * [new branch] gh/guilhermeleobas/223/head -> origin/gh/guilhermeleobas/223/head 2025-08-14T21:24:12.7272909Z * [new branch] gh/guilhermeleobas/223/orig -> origin/gh/guilhermeleobas/223/orig 2025-08-14T21:24:12.7273280Z * [new branch] gh/guilhermeleobas/224/base -> origin/gh/guilhermeleobas/224/base 2025-08-14T21:24:12.7273456Z * [new branch] gh/guilhermeleobas/224/head -> origin/gh/guilhermeleobas/224/head 2025-08-14T21:24:12.7273637Z * [new branch] gh/guilhermeleobas/224/orig -> origin/gh/guilhermeleobas/224/orig 2025-08-14T21:24:12.7273814Z * [new branch] gh/guilhermeleobas/225/base -> origin/gh/guilhermeleobas/225/base 2025-08-14T21:24:12.7273992Z * [new branch] gh/guilhermeleobas/225/head -> origin/gh/guilhermeleobas/225/head 2025-08-14T21:24:12.7274161Z * [new branch] gh/guilhermeleobas/225/orig -> origin/gh/guilhermeleobas/225/orig 2025-08-14T21:24:12.7274341Z * [new branch] gh/guilhermeleobas/226/base -> origin/gh/guilhermeleobas/226/base 2025-08-14T21:24:12.7274547Z * [new branch] gh/guilhermeleobas/226/head -> origin/gh/guilhermeleobas/226/head 2025-08-14T21:24:12.7274728Z * [new branch] gh/guilhermeleobas/226/orig -> origin/gh/guilhermeleobas/226/orig 2025-08-14T21:24:12.7274912Z * [new branch] gh/guilhermeleobas/227/base -> origin/gh/guilhermeleobas/227/base 2025-08-14T21:24:12.7275069Z * [new branch] gh/guilhermeleobas/227/head -> origin/gh/guilhermeleobas/227/head 2025-08-14T21:24:12.7280834Z * [new branch] gh/guilhermeleobas/227/orig -> origin/gh/guilhermeleobas/227/orig 2025-08-14T21:24:12.7281190Z * [new branch] gh/guilhermeleobas/228/base -> origin/gh/guilhermeleobas/228/base 2025-08-14T21:24:12.7281461Z * [new branch] gh/guilhermeleobas/228/head -> origin/gh/guilhermeleobas/228/head 2025-08-14T21:24:12.7281803Z * [new branch] gh/guilhermeleobas/228/orig -> origin/gh/guilhermeleobas/228/orig 2025-08-14T21:24:12.7281990Z * [new branch] gh/guilhermeleobas/229/base -> origin/gh/guilhermeleobas/229/base 2025-08-14T21:24:12.7282189Z * [new branch] gh/guilhermeleobas/229/head -> origin/gh/guilhermeleobas/229/head 2025-08-14T21:24:12.7282501Z * [new branch] gh/guilhermeleobas/229/orig -> origin/gh/guilhermeleobas/229/orig 2025-08-14T21:24:12.7282690Z * [new branch] gh/guilhermeleobas/230/base -> origin/gh/guilhermeleobas/230/base 2025-08-14T21:24:12.7282858Z * [new branch] gh/guilhermeleobas/230/head -> origin/gh/guilhermeleobas/230/head 2025-08-14T21:24:12.7283029Z * [new branch] gh/guilhermeleobas/230/orig -> origin/gh/guilhermeleobas/230/orig 2025-08-14T21:24:12.7283192Z * [new branch] gh/guilhermeleobas/231/base -> origin/gh/guilhermeleobas/231/base 2025-08-14T21:24:12.7283354Z * [new branch] gh/guilhermeleobas/231/head -> origin/gh/guilhermeleobas/231/head 2025-08-14T21:24:12.7283525Z * [new branch] gh/guilhermeleobas/231/orig -> origin/gh/guilhermeleobas/231/orig 2025-08-14T21:24:12.7283689Z * [new branch] gh/guilhermeleobas/232/base -> origin/gh/guilhermeleobas/232/base 2025-08-14T21:24:12.7283864Z * [new branch] gh/guilhermeleobas/232/head -> origin/gh/guilhermeleobas/232/head 2025-08-14T21:24:12.7284158Z * [new branch] gh/guilhermeleobas/232/orig -> origin/gh/guilhermeleobas/232/orig 2025-08-14T21:24:12.7284375Z * [new branch] gh/guilhermeleobas/233/base -> origin/gh/guilhermeleobas/233/base 2025-08-14T21:24:12.7284724Z * [new branch] gh/guilhermeleobas/233/head -> origin/gh/guilhermeleobas/233/head 2025-08-14T21:24:12.7286756Z * [new branch] gh/guilhermeleobas/233/orig -> origin/gh/guilhermeleobas/233/orig 2025-08-14T21:24:12.7286970Z * [new branch] gh/guilhermeleobas/73/base -> origin/gh/guilhermeleobas/73/base 2025-08-14T21:24:12.7287503Z * [new branch] gh/guilhermeleobas/73/head -> origin/gh/guilhermeleobas/73/head 2025-08-14T21:24:12.7288085Z * [new branch] gh/guilhermeleobas/73/orig -> origin/gh/guilhermeleobas/73/orig 2025-08-14T21:24:12.7291956Z * [new branch] gh/henrylhtsang/103/base -> origin/gh/henrylhtsang/103/base 2025-08-14T21:24:12.7292163Z * [new branch] gh/henrylhtsang/103/head -> origin/gh/henrylhtsang/103/head 2025-08-14T21:24:12.7292331Z * [new branch] gh/henrylhtsang/103/orig -> origin/gh/henrylhtsang/103/orig 2025-08-14T21:24:12.7292481Z * [new branch] gh/henrylhtsang/108/base -> origin/gh/henrylhtsang/108/base 2025-08-14T21:24:12.7292668Z * [new branch] gh/henrylhtsang/108/head -> origin/gh/henrylhtsang/108/head 2025-08-14T21:24:12.7294254Z * [new branch] gh/henrylhtsang/108/orig -> origin/gh/henrylhtsang/108/orig 2025-08-14T21:24:12.7294615Z * [new branch] gh/henrylhtsang/118/base -> origin/gh/henrylhtsang/118/base 2025-08-14T21:24:12.7295049Z * [new branch] gh/henrylhtsang/118/head -> origin/gh/henrylhtsang/118/head 2025-08-14T21:24:12.7297921Z * [new branch] gh/henrylhtsang/118/orig -> origin/gh/henrylhtsang/118/orig 2025-08-14T21:24:12.7298269Z * [new branch] gh/henrylhtsang/123/base -> origin/gh/henrylhtsang/123/base 2025-08-14T21:24:12.7301155Z * [new branch] gh/henrylhtsang/123/head -> origin/gh/henrylhtsang/123/head 2025-08-14T21:24:12.7301508Z * [new branch] gh/henrylhtsang/123/orig -> origin/gh/henrylhtsang/123/orig 2025-08-14T21:24:12.7301757Z * [new branch] gh/henrylhtsang/124/base -> origin/gh/henrylhtsang/124/base 2025-08-14T21:24:12.7301999Z * [new branch] gh/henrylhtsang/124/head -> origin/gh/henrylhtsang/124/head 2025-08-14T21:24:12.7302221Z * [new branch] gh/henrylhtsang/124/orig -> origin/gh/henrylhtsang/124/orig 2025-08-14T21:24:12.7302387Z * [new branch] gh/henrylhtsang/125/base -> origin/gh/henrylhtsang/125/base 2025-08-14T21:24:12.7302625Z * [new branch] gh/henrylhtsang/125/head -> origin/gh/henrylhtsang/125/head 2025-08-14T21:24:12.7303311Z * [new branch] gh/henrylhtsang/125/orig -> origin/gh/henrylhtsang/125/orig 2025-08-14T21:24:12.7303502Z * [new branch] gh/henrylhtsang/126/base -> origin/gh/henrylhtsang/126/base 2025-08-14T21:24:12.7306678Z * [new branch] gh/henrylhtsang/126/head -> origin/gh/henrylhtsang/126/head 2025-08-14T21:24:12.7306868Z * [new branch] gh/henrylhtsang/126/orig -> origin/gh/henrylhtsang/126/orig 2025-08-14T21:24:12.7307029Z * [new branch] gh/henrylhtsang/127/base -> origin/gh/henrylhtsang/127/base 2025-08-14T21:24:12.7307180Z * [new branch] gh/henrylhtsang/127/head -> origin/gh/henrylhtsang/127/head 2025-08-14T21:24:12.7307399Z * [new branch] gh/henrylhtsang/127/orig -> origin/gh/henrylhtsang/127/orig 2025-08-14T21:24:12.7309131Z * [new branch] gh/henrylhtsang/128/base -> origin/gh/henrylhtsang/128/base 2025-08-14T21:24:12.7309505Z * [new branch] gh/henrylhtsang/128/head -> origin/gh/henrylhtsang/128/head 2025-08-14T21:24:12.7309705Z * [new branch] gh/henrylhtsang/128/orig -> origin/gh/henrylhtsang/128/orig 2025-08-14T21:24:12.7312403Z * [new branch] gh/henrylhtsang/129/base -> origin/gh/henrylhtsang/129/base 2025-08-14T21:24:12.7312601Z * [new branch] gh/henrylhtsang/129/head -> origin/gh/henrylhtsang/129/head 2025-08-14T21:24:12.7312762Z * [new branch] gh/henrylhtsang/129/orig -> origin/gh/henrylhtsang/129/orig 2025-08-14T21:24:12.7313127Z * [new branch] gh/henrylhtsang/130/base -> origin/gh/henrylhtsang/130/base 2025-08-14T21:24:12.7317205Z * [new branch] gh/henrylhtsang/130/head -> origin/gh/henrylhtsang/130/head 2025-08-14T21:24:12.7317396Z * [new branch] gh/henrylhtsang/131/base -> origin/gh/henrylhtsang/131/base 2025-08-14T21:24:12.7317713Z * [new branch] gh/henrylhtsang/131/head -> origin/gh/henrylhtsang/131/head 2025-08-14T21:24:12.7317906Z * [new branch] gh/henrylhtsang/131/orig -> origin/gh/henrylhtsang/131/orig 2025-08-14T21:24:12.7318058Z * [new branch] gh/henrylhtsang/132/base -> origin/gh/henrylhtsang/132/base 2025-08-14T21:24:12.7318222Z * [new branch] gh/henrylhtsang/132/head -> origin/gh/henrylhtsang/132/head 2025-08-14T21:24:12.7318909Z * [new branch] gh/henrylhtsang/132/orig -> origin/gh/henrylhtsang/132/orig 2025-08-14T21:24:12.7321277Z * [new branch] gh/henrylhtsang/133/base -> origin/gh/henrylhtsang/133/base 2025-08-14T21:24:12.7321489Z * [new branch] gh/henrylhtsang/133/head -> origin/gh/henrylhtsang/133/head 2025-08-14T21:24:12.7321655Z * [new branch] gh/henrylhtsang/133/orig -> origin/gh/henrylhtsang/133/orig 2025-08-14T21:24:12.7323846Z * [new branch] gh/henrylhtsang/134/base -> origin/gh/henrylhtsang/134/base 2025-08-14T21:24:12.7324159Z * [new branch] gh/henrylhtsang/134/head -> origin/gh/henrylhtsang/134/head 2025-08-14T21:24:12.7324328Z * [new branch] gh/henrylhtsang/134/orig -> origin/gh/henrylhtsang/134/orig 2025-08-14T21:24:12.7324577Z * [new branch] gh/henrylhtsang/135/base -> origin/gh/henrylhtsang/135/base 2025-08-14T21:24:12.7325696Z * [new branch] gh/henrylhtsang/135/head -> origin/gh/henrylhtsang/135/head 2025-08-14T21:24:12.7325957Z * [new branch] gh/henrylhtsang/135/orig -> origin/gh/henrylhtsang/135/orig 2025-08-14T21:24:12.7327606Z * [new branch] gh/henrylhtsang/136/base -> origin/gh/henrylhtsang/136/base 2025-08-14T21:24:12.7328110Z * [new branch] gh/henrylhtsang/136/head -> origin/gh/henrylhtsang/136/head 2025-08-14T21:24:12.7329092Z * [new branch] gh/henrylhtsang/136/orig -> origin/gh/henrylhtsang/136/orig 2025-08-14T21:24:12.7330688Z * [new branch] gh/henrylhtsang/137/base -> origin/gh/henrylhtsang/137/base 2025-08-14T21:24:12.7331011Z * [new branch] gh/henrylhtsang/137/head -> origin/gh/henrylhtsang/137/head 2025-08-14T21:24:12.7331480Z * [new branch] gh/henrylhtsang/137/orig -> origin/gh/henrylhtsang/137/orig 2025-08-14T21:24:12.7335029Z * [new branch] gh/henrylhtsang/138/base -> origin/gh/henrylhtsang/138/base 2025-08-14T21:24:12.7335223Z * [new branch] gh/henrylhtsang/138/head -> origin/gh/henrylhtsang/138/head 2025-08-14T21:24:12.7335400Z * [new branch] gh/henrylhtsang/138/orig -> origin/gh/henrylhtsang/138/orig 2025-08-14T21:24:12.7335558Z * [new branch] gh/henrylhtsang/139/base -> origin/gh/henrylhtsang/139/base 2025-08-14T21:24:12.7335728Z * [new branch] gh/henrylhtsang/139/head -> origin/gh/henrylhtsang/139/head 2025-08-14T21:24:12.7336343Z * [new branch] gh/henrylhtsang/139/orig -> origin/gh/henrylhtsang/139/orig 2025-08-14T21:24:12.7337563Z * [new branch] gh/henrylhtsang/140/base -> origin/gh/henrylhtsang/140/base 2025-08-14T21:24:12.7337943Z * [new branch] gh/henrylhtsang/140/head -> origin/gh/henrylhtsang/140/head 2025-08-14T21:24:12.7342218Z * [new branch] gh/henrylhtsang/140/orig -> origin/gh/henrylhtsang/140/orig 2025-08-14T21:24:12.7342453Z * [new branch] gh/henrylhtsang/141/base -> origin/gh/henrylhtsang/141/base 2025-08-14T21:24:12.7343166Z * [new branch] gh/henrylhtsang/141/head -> origin/gh/henrylhtsang/141/head 2025-08-14T21:24:12.7343749Z * [new branch] gh/henrylhtsang/141/orig -> origin/gh/henrylhtsang/141/orig 2025-08-14T21:24:12.7343932Z * [new branch] gh/henrylhtsang/142/base -> origin/gh/henrylhtsang/142/base 2025-08-14T21:24:12.7344091Z * [new branch] gh/henrylhtsang/142/head -> origin/gh/henrylhtsang/142/head 2025-08-14T21:24:12.7344452Z * [new branch] gh/henrylhtsang/142/orig -> origin/gh/henrylhtsang/142/orig 2025-08-14T21:24:12.7344813Z * [new branch] gh/henrylhtsang/143/base -> origin/gh/henrylhtsang/143/base 2025-08-14T21:24:12.7345253Z * [new branch] gh/henrylhtsang/143/head -> origin/gh/henrylhtsang/143/head 2025-08-14T21:24:12.7346390Z * [new branch] gh/henrylhtsang/143/orig -> origin/gh/henrylhtsang/143/orig 2025-08-14T21:24:12.7347297Z * [new branch] gh/henrylhtsang/144/base -> origin/gh/henrylhtsang/144/base 2025-08-14T21:24:12.7347479Z * [new branch] gh/henrylhtsang/144/head -> origin/gh/henrylhtsang/144/head 2025-08-14T21:24:12.7348034Z * [new branch] gh/henrylhtsang/144/orig -> origin/gh/henrylhtsang/144/orig 2025-08-14T21:24:12.7349358Z * [new branch] gh/henrylhtsang/145/base -> origin/gh/henrylhtsang/145/base 2025-08-14T21:24:12.7349669Z * [new branch] gh/henrylhtsang/145/head -> origin/gh/henrylhtsang/145/head 2025-08-14T21:24:12.7350296Z * [new branch] gh/henrylhtsang/145/orig -> origin/gh/henrylhtsang/145/orig 2025-08-14T21:24:12.7351562Z * [new branch] gh/henrylhtsang/146/base -> origin/gh/henrylhtsang/146/base 2025-08-14T21:24:12.7351847Z * [new branch] gh/henrylhtsang/146/head -> origin/gh/henrylhtsang/146/head 2025-08-14T21:24:12.7354458Z * [new branch] gh/henrylhtsang/146/orig -> origin/gh/henrylhtsang/146/orig 2025-08-14T21:24:12.7354618Z * [new branch] gh/huydhn/1/head -> origin/gh/huydhn/1/head 2025-08-14T21:24:12.7354892Z * [new branch] gh/huydhn/1/next -> origin/gh/huydhn/1/next 2025-08-14T21:24:12.7355035Z * [new branch] gh/huydhn/2/head -> origin/gh/huydhn/2/head 2025-08-14T21:24:12.7359677Z * [new branch] gh/huydhn/2/next -> origin/gh/huydhn/2/next 2025-08-14T21:24:12.7359971Z * [new branch] gh/huydhn/2/orig -> origin/gh/huydhn/2/orig 2025-08-14T21:24:12.7360205Z * [new branch] gh/huydhn/3/head -> origin/gh/huydhn/3/head 2025-08-14T21:24:12.7360357Z * [new branch] gh/huydhn/3/next -> origin/gh/huydhn/3/next 2025-08-14T21:24:12.7360496Z * [new branch] gh/huydhn/3/orig -> origin/gh/huydhn/3/orig 2025-08-14T21:24:12.7360635Z * [new branch] gh/huydhn/4/head -> origin/gh/huydhn/4/head 2025-08-14T21:24:12.7360771Z * [new branch] gh/huydhn/4/next -> origin/gh/huydhn/4/next 2025-08-14T21:24:12.7362870Z * [new branch] gh/huydhn/4/orig -> origin/gh/huydhn/4/orig 2025-08-14T21:24:12.7363111Z * [new branch] gh/huydhn/5/head -> origin/gh/huydhn/5/head 2025-08-14T21:24:12.7363254Z * [new branch] gh/huydhn/5/next -> origin/gh/huydhn/5/next 2025-08-14T21:24:12.7363381Z * [new branch] gh/huydhn/5/orig -> origin/gh/huydhn/5/orig 2025-08-14T21:24:12.7363546Z * [new branch] gh/huydhn/6/head -> origin/gh/huydhn/6/head 2025-08-14T21:24:12.7363714Z * [new branch] gh/huydhn/6/next -> origin/gh/huydhn/6/next 2025-08-14T21:24:12.7363863Z * [new branch] gh/huydhn/6/orig -> origin/gh/huydhn/6/orig 2025-08-14T21:24:12.7370339Z * [new branch] gh/int3/97/base -> origin/gh/int3/97/base 2025-08-14T21:24:12.7370517Z * [new branch] gh/int3/97/head -> origin/gh/int3/97/head 2025-08-14T21:24:12.7370680Z * [new branch] gh/isuruf/101/base -> origin/gh/isuruf/101/base 2025-08-14T21:24:12.7370829Z * [new branch] gh/isuruf/101/head -> origin/gh/isuruf/101/head 2025-08-14T21:24:12.7370970Z * [new branch] gh/isuruf/116/base -> origin/gh/isuruf/116/base 2025-08-14T21:24:12.7372626Z * [new branch] gh/isuruf/116/head -> origin/gh/isuruf/116/head 2025-08-14T21:24:12.7373184Z * [new branch] gh/isuruf/116/orig -> origin/gh/isuruf/116/orig 2025-08-14T21:24:12.7373497Z * [new branch] gh/isuruf/141/base -> origin/gh/isuruf/141/base 2025-08-14T21:24:12.7373661Z * [new branch] gh/isuruf/141/head -> origin/gh/isuruf/141/head 2025-08-14T21:24:12.7373904Z * [new branch] gh/isuruf/141/orig -> origin/gh/isuruf/141/orig 2025-08-14T21:24:12.7374067Z * [new branch] gh/isuruf/142/base -> origin/gh/isuruf/142/base 2025-08-14T21:24:12.7378988Z * [new branch] gh/isuruf/142/head -> origin/gh/isuruf/142/head 2025-08-14T21:24:12.7379177Z * [new branch] gh/isuruf/142/orig -> origin/gh/isuruf/142/orig 2025-08-14T21:24:12.7379346Z * [new branch] gh/isuruf/81/base -> origin/gh/isuruf/81/base 2025-08-14T21:24:12.7379492Z * [new branch] gh/isuruf/81/head -> origin/gh/isuruf/81/head 2025-08-14T21:24:12.7379678Z * [new branch] gh/isuruf/81/orig -> origin/gh/isuruf/81/orig 2025-08-14T21:24:12.7380362Z * [new branch] gh/jamesjwu/140/base -> origin/gh/jamesjwu/140/base 2025-08-14T21:24:12.7380547Z * [new branch] gh/jamesjwu/140/head -> origin/gh/jamesjwu/140/head 2025-08-14T21:24:12.7380709Z * [new branch] gh/jamesjwu/140/orig -> origin/gh/jamesjwu/140/orig 2025-08-14T21:24:12.7380859Z * [new branch] gh/jamesjwu/150/base -> origin/gh/jamesjwu/150/base 2025-08-14T21:24:12.7381003Z * [new branch] gh/jamesjwu/150/head -> origin/gh/jamesjwu/150/head 2025-08-14T21:24:12.7381160Z * [new branch] gh/jamesjwu/150/orig -> origin/gh/jamesjwu/150/orig 2025-08-14T21:24:12.7385992Z * [new branch] gh/jamesjwu/154/base -> origin/gh/jamesjwu/154/base 2025-08-14T21:24:12.7386308Z * [new branch] gh/jamesjwu/154/head -> origin/gh/jamesjwu/154/head 2025-08-14T21:24:12.7386555Z * [new branch] gh/jamesjwu/154/orig -> origin/gh/jamesjwu/154/orig 2025-08-14T21:24:12.7386687Z * [new branch] gh/jamesjwu/155/base -> origin/gh/jamesjwu/155/base 2025-08-14T21:24:12.7386822Z * [new branch] gh/jamesjwu/155/head -> origin/gh/jamesjwu/155/head 2025-08-14T21:24:12.7386950Z * [new branch] gh/jamesjwu/155/orig -> origin/gh/jamesjwu/155/orig 2025-08-14T21:24:12.7387077Z * [new branch] gh/jamesjwu/159/base -> origin/gh/jamesjwu/159/base 2025-08-14T21:24:12.7387210Z * [new branch] gh/jamesjwu/159/head -> origin/gh/jamesjwu/159/head 2025-08-14T21:24:12.7390515Z * [new branch] gh/jamesjwu/159/orig -> origin/gh/jamesjwu/159/orig 2025-08-14T21:24:12.7390836Z * [new branch] gh/jamesjwu/163/base -> origin/gh/jamesjwu/163/base 2025-08-14T21:24:12.7391002Z * [new branch] gh/jamesjwu/163/head -> origin/gh/jamesjwu/163/head 2025-08-14T21:24:12.7391153Z * [new branch] gh/jamesjwu/163/orig -> origin/gh/jamesjwu/163/orig 2025-08-14T21:24:12.7391306Z * [new branch] gh/jamesjwu/171/base -> origin/gh/jamesjwu/171/base 2025-08-14T21:24:12.7391454Z * [new branch] gh/jamesjwu/171/head -> origin/gh/jamesjwu/171/head 2025-08-14T21:24:12.7391602Z * [new branch] gh/jamesjwu/171/orig -> origin/gh/jamesjwu/171/orig 2025-08-14T21:24:12.7396255Z * [new branch] gh/jamesjwu/174/base -> origin/gh/jamesjwu/174/base 2025-08-14T21:24:12.7396655Z * [new branch] gh/jamesjwu/174/head -> origin/gh/jamesjwu/174/head 2025-08-14T21:24:12.7396820Z * [new branch] gh/jamesjwu/174/orig -> origin/gh/jamesjwu/174/orig 2025-08-14T21:24:12.7396975Z * [new branch] gh/jamesjwu/175/base -> origin/gh/jamesjwu/175/base 2025-08-14T21:24:12.7397217Z * [new branch] gh/jamesjwu/175/head -> origin/gh/jamesjwu/175/head 2025-08-14T21:24:12.7397391Z * [new branch] gh/jamesjwu/175/orig -> origin/gh/jamesjwu/175/orig 2025-08-14T21:24:12.7397543Z * [new branch] gh/jamesjwu/176/base -> origin/gh/jamesjwu/176/base 2025-08-14T21:24:12.7397689Z * [new branch] gh/jamesjwu/176/head -> origin/gh/jamesjwu/176/head 2025-08-14T21:24:12.7401658Z * [new branch] gh/jamesjwu/176/orig -> origin/gh/jamesjwu/176/orig 2025-08-14T21:24:12.7401834Z * [new branch] gh/jamesjwu/177/base -> origin/gh/jamesjwu/177/base 2025-08-14T21:24:12.7402110Z * [new branch] gh/jamesjwu/177/head -> origin/gh/jamesjwu/177/head 2025-08-14T21:24:12.7402262Z * [new branch] gh/jamesjwu/177/orig -> origin/gh/jamesjwu/177/orig 2025-08-14T21:24:12.7402399Z * [new branch] gh/jamesjwu/178/base -> origin/gh/jamesjwu/178/base 2025-08-14T21:24:12.7402585Z * [new branch] gh/jamesjwu/178/head -> origin/gh/jamesjwu/178/head 2025-08-14T21:24:12.7402728Z * [new branch] gh/jamesjwu/178/orig -> origin/gh/jamesjwu/178/orig 2025-08-14T21:24:12.7403023Z * [new branch] gh/jamesjwu/179/base -> origin/gh/jamesjwu/179/base 2025-08-14T21:24:12.7403737Z * [new branch] gh/jamesjwu/179/head -> origin/gh/jamesjwu/179/head 2025-08-14T21:24:12.7404331Z * [new branch] gh/jamesjwu/179/orig -> origin/gh/jamesjwu/179/orig 2025-08-14T21:24:12.7405599Z * [new branch] gh/jamesjwu/180/base -> origin/gh/jamesjwu/180/base 2025-08-14T21:24:12.7406141Z * [new branch] gh/jamesjwu/180/head -> origin/gh/jamesjwu/180/head 2025-08-14T21:24:12.7406857Z * [new branch] gh/jamesjwu/180/orig -> origin/gh/jamesjwu/180/orig 2025-08-14T21:24:12.7407919Z * [new branch] gh/jamesjwu/181/base -> origin/gh/jamesjwu/181/base 2025-08-14T21:24:12.7408346Z * [new branch] gh/jamesjwu/181/head -> origin/gh/jamesjwu/181/head 2025-08-14T21:24:12.7409135Z * [new branch] gh/jamesjwu/181/orig -> origin/gh/jamesjwu/181/orig 2025-08-14T21:24:12.7410159Z * [new branch] gh/jamesjwu/182/base -> origin/gh/jamesjwu/182/base 2025-08-14T21:24:12.7410532Z * [new branch] gh/jamesjwu/182/head -> origin/gh/jamesjwu/182/head 2025-08-14T21:24:12.7411555Z * [new branch] gh/jamesjwu/182/orig -> origin/gh/jamesjwu/182/orig 2025-08-14T21:24:12.7413153Z * [new branch] gh/jamesjwu/183/base -> origin/gh/jamesjwu/183/base 2025-08-14T21:24:12.7413330Z * [new branch] gh/jamesjwu/183/head -> origin/gh/jamesjwu/183/head 2025-08-14T21:24:12.7413907Z * [new branch] gh/jamesjwu/183/orig -> origin/gh/jamesjwu/183/orig 2025-08-14T21:24:12.7415148Z * [new branch] gh/jamesjwu/184/base -> origin/gh/jamesjwu/184/base 2025-08-14T21:24:12.7415406Z * [new branch] gh/jamesjwu/184/head -> origin/gh/jamesjwu/184/head 2025-08-14T21:24:12.7416517Z * [new branch] gh/jamesjwu/184/orig -> origin/gh/jamesjwu/184/orig 2025-08-14T21:24:12.7417811Z * [new branch] gh/jamesjwu/52/base -> origin/gh/jamesjwu/52/base 2025-08-14T21:24:12.7417954Z * [new branch] gh/jamesjwu/52/head -> origin/gh/jamesjwu/52/head 2025-08-14T21:24:12.7419087Z * [new branch] gh/jamesjwu/53/base -> origin/gh/jamesjwu/53/base 2025-08-14T21:24:12.7419241Z * [new branch] gh/jamesjwu/53/head -> origin/gh/jamesjwu/53/head 2025-08-14T21:24:12.7420575Z * [new branch] gh/jamesjwu/54/base -> origin/gh/jamesjwu/54/base 2025-08-14T21:24:12.7420864Z * [new branch] gh/jamesjwu/54/head -> origin/gh/jamesjwu/54/head 2025-08-14T21:24:12.7423108Z * [new branch] gh/jamesjwu/55/base -> origin/gh/jamesjwu/55/base 2025-08-14T21:24:12.7423308Z * [new branch] gh/jamesjwu/55/head -> origin/gh/jamesjwu/55/head 2025-08-14T21:24:12.7423459Z * [new branch] gh/jamesjwu/56/base -> origin/gh/jamesjwu/56/base 2025-08-14T21:24:12.7424137Z * [new branch] gh/jamesjwu/56/head -> origin/gh/jamesjwu/56/head 2025-08-14T21:24:12.7424975Z * [new branch] gh/jamesjwu/57/base -> origin/gh/jamesjwu/57/base 2025-08-14T21:24:12.7425511Z * [new branch] gh/jamesjwu/57/head -> origin/gh/jamesjwu/57/head 2025-08-14T21:24:12.7430298Z * [new branch] gh/jamesjwu/58/base -> origin/gh/jamesjwu/58/base 2025-08-14T21:24:12.7430488Z * [new branch] gh/jamesjwu/58/head -> origin/gh/jamesjwu/58/head 2025-08-14T21:24:12.7430631Z * [new branch] gh/jamesjwu/59/base -> origin/gh/jamesjwu/59/base 2025-08-14T21:24:12.7430785Z * [new branch] gh/jamesjwu/59/head -> origin/gh/jamesjwu/59/head 2025-08-14T21:24:12.7430940Z * [new branch] gh/jamesjwu/60/base -> origin/gh/jamesjwu/60/base 2025-08-14T21:24:12.7431083Z * [new branch] gh/jamesjwu/60/head -> origin/gh/jamesjwu/60/head 2025-08-14T21:24:12.7431763Z * [new branch] gh/jamesjwu/61/base -> origin/gh/jamesjwu/61/base 2025-08-14T21:24:12.7432293Z * [new branch] gh/jamesjwu/61/head -> origin/gh/jamesjwu/61/head 2025-08-14T21:24:12.7433730Z * [new branch] gh/jamesjwu/62/base -> origin/gh/jamesjwu/62/base 2025-08-14T21:24:12.7433876Z * [new branch] gh/jamesjwu/62/head -> origin/gh/jamesjwu/62/head 2025-08-14T21:24:12.7434778Z * [new branch] gh/jamesjwu/63/base -> origin/gh/jamesjwu/63/base 2025-08-14T21:24:12.7435267Z * [new branch] gh/jamesjwu/63/head -> origin/gh/jamesjwu/63/head 2025-08-14T21:24:12.7439614Z * [new branch] gh/jamesjwu/64/base -> origin/gh/jamesjwu/64/base 2025-08-14T21:24:12.7439808Z * [new branch] gh/jamesjwu/64/head -> origin/gh/jamesjwu/64/head 2025-08-14T21:24:12.7439949Z * [new branch] gh/jamesjwu/65/base -> origin/gh/jamesjwu/65/base 2025-08-14T21:24:12.7440086Z * [new branch] gh/jamesjwu/65/head -> origin/gh/jamesjwu/65/head 2025-08-14T21:24:12.7440246Z * [new branch] gh/janeyx99/165/base -> origin/gh/janeyx99/165/base 2025-08-14T21:24:12.7442173Z * [new branch] gh/janeyx99/165/head -> origin/gh/janeyx99/165/head 2025-08-14T21:24:12.7442517Z * [new branch] gh/janeyx99/165/orig -> origin/gh/janeyx99/165/orig 2025-08-14T21:24:12.7442693Z * [new branch] gh/janeyx99/201/base -> origin/gh/janeyx99/201/base 2025-08-14T21:24:12.7443002Z * [new branch] gh/janeyx99/201/head -> origin/gh/janeyx99/201/head 2025-08-14T21:24:12.7444488Z * [new branch] gh/janeyx99/201/orig -> origin/gh/janeyx99/201/orig 2025-08-14T21:24:12.7445109Z * [new branch] gh/janeyx99/225/base -> origin/gh/janeyx99/225/base 2025-08-14T21:24:12.7445978Z * [new branch] gh/janeyx99/225/head -> origin/gh/janeyx99/225/head 2025-08-14T21:24:12.7446554Z * [new branch] gh/janeyx99/225/orig -> origin/gh/janeyx99/225/orig 2025-08-14T21:24:12.7451006Z * [new branch] gh/janeyx99/256/base -> origin/gh/janeyx99/256/base 2025-08-14T21:24:12.7451340Z * [new branch] gh/janeyx99/256/head -> origin/gh/janeyx99/256/head 2025-08-14T21:24:12.7451537Z * [new branch] gh/janeyx99/256/orig -> origin/gh/janeyx99/256/orig 2025-08-14T21:24:12.7451763Z * [new branch] gh/janeyx99/268/base -> origin/gh/janeyx99/268/base 2025-08-14T21:24:12.7452453Z * [new branch] gh/janeyx99/268/head -> origin/gh/janeyx99/268/head 2025-08-14T21:24:12.7452761Z * [new branch] gh/janeyx99/268/orig -> origin/gh/janeyx99/268/orig 2025-08-14T21:24:12.7453430Z * [new branch] gh/janeyx99/269/base -> origin/gh/janeyx99/269/base 2025-08-14T21:24:12.7453654Z * [new branch] gh/janeyx99/269/head -> origin/gh/janeyx99/269/head 2025-08-14T21:24:12.7453899Z * [new branch] gh/janeyx99/269/orig -> origin/gh/janeyx99/269/orig 2025-08-14T21:24:12.7460990Z * [new branch] gh/janeyx99/274/base -> origin/gh/janeyx99/274/base 2025-08-14T21:24:12.7461331Z * [new branch] gh/janeyx99/274/head -> origin/gh/janeyx99/274/head 2025-08-14T21:24:12.7461559Z * [new branch] gh/janeyx99/274/orig -> origin/gh/janeyx99/274/orig 2025-08-14T21:24:12.7461739Z * [new branch] gh/janeyx99/276/base -> origin/gh/janeyx99/276/base 2025-08-14T21:24:12.7461969Z * [new branch] gh/janeyx99/276/head -> origin/gh/janeyx99/276/head 2025-08-14T21:24:12.7462119Z * [new branch] gh/janeyx99/276/orig -> origin/gh/janeyx99/276/orig 2025-08-14T21:24:12.7462741Z * [new branch] gh/janeyx99/277/base -> origin/gh/janeyx99/277/base 2025-08-14T21:24:12.7462942Z * [new branch] gh/janeyx99/277/head -> origin/gh/janeyx99/277/head 2025-08-14T21:24:12.7463094Z * [new branch] gh/janeyx99/277/orig -> origin/gh/janeyx99/277/orig 2025-08-14T21:24:12.7463247Z * [new branch] gh/janeyx99/278/base -> origin/gh/janeyx99/278/base 2025-08-14T21:24:12.7463395Z * [new branch] gh/janeyx99/278/head -> origin/gh/janeyx99/278/head 2025-08-14T21:24:12.7463544Z * [new branch] gh/janeyx99/278/orig -> origin/gh/janeyx99/278/orig 2025-08-14T21:24:12.7463863Z * [new branch] gh/janeyx99/279/base -> origin/gh/janeyx99/279/base 2025-08-14T21:24:12.7464381Z * [new branch] gh/janeyx99/279/head -> origin/gh/janeyx99/279/head 2025-08-14T21:24:12.7465496Z * [new branch] gh/janeyx99/279/orig -> origin/gh/janeyx99/279/orig 2025-08-14T21:24:12.7465969Z * [new branch] gh/janeyx99/280/base -> origin/gh/janeyx99/280/base 2025-08-14T21:24:12.7467947Z * [new branch] gh/janeyx99/280/head -> origin/gh/janeyx99/280/head 2025-08-14T21:24:12.7468295Z * [new branch] gh/janeyx99/280/orig -> origin/gh/janeyx99/280/orig 2025-08-14T21:24:12.7468498Z * [new branch] gh/janeyx99/281/base -> origin/gh/janeyx99/281/base 2025-08-14T21:24:12.7468748Z * [new branch] gh/janeyx99/281/head -> origin/gh/janeyx99/281/head 2025-08-14T21:24:12.7470652Z * [new branch] gh/janeyx99/281/orig -> origin/gh/janeyx99/281/orig 2025-08-14T21:24:12.7470844Z * [new branch] gh/janeyx99/282/base -> origin/gh/janeyx99/282/base 2025-08-14T21:24:12.7471020Z * [new branch] gh/janeyx99/282/head -> origin/gh/janeyx99/282/head 2025-08-14T21:24:12.7472841Z * [new branch] gh/janeyx99/282/orig -> origin/gh/janeyx99/282/orig 2025-08-14T21:24:12.7473165Z * [new branch] gh/janeyx99/283/base -> origin/gh/janeyx99/283/base 2025-08-14T21:24:12.7473324Z * [new branch] gh/janeyx99/283/head -> origin/gh/janeyx99/283/head 2025-08-14T21:24:12.7475373Z * [new branch] gh/janeyx99/283/orig -> origin/gh/janeyx99/283/orig 2025-08-14T21:24:12.7475692Z * [new branch] gh/janeyx99/284/base -> origin/gh/janeyx99/284/base 2025-08-14T21:24:12.7475852Z * [new branch] gh/janeyx99/284/head -> origin/gh/janeyx99/284/head 2025-08-14T21:24:12.7476180Z * [new branch] gh/janeyx99/284/orig -> origin/gh/janeyx99/284/orig 2025-08-14T21:24:12.7481360Z * [new branch] gh/janeyx99/285/base -> origin/gh/janeyx99/285/base 2025-08-14T21:24:12.7481679Z * [new branch] gh/janeyx99/285/head -> origin/gh/janeyx99/285/head 2025-08-14T21:24:12.7481840Z * [new branch] gh/janeyx99/285/orig -> origin/gh/janeyx99/285/orig 2025-08-14T21:24:12.7482040Z * [new branch] gh/janeyx99/286/base -> origin/gh/janeyx99/286/base 2025-08-14T21:24:12.7482188Z * [new branch] gh/janeyx99/286/head -> origin/gh/janeyx99/286/head 2025-08-14T21:24:12.7482331Z * [new branch] gh/janeyx99/286/orig -> origin/gh/janeyx99/286/orig 2025-08-14T21:24:12.7482994Z * [new branch] gh/janeyx99/287/base -> origin/gh/janeyx99/287/base 2025-08-14T21:24:12.7483410Z * [new branch] gh/janeyx99/287/head -> origin/gh/janeyx99/287/head 2025-08-14T21:24:12.7483730Z * [new branch] gh/janeyx99/287/orig -> origin/gh/janeyx99/287/orig 2025-08-14T21:24:12.7484506Z * [new branch] gh/janeyx99/288/base -> origin/gh/janeyx99/288/base 2025-08-14T21:24:12.7485746Z * [new branch] gh/janeyx99/288/head -> origin/gh/janeyx99/288/head 2025-08-14T21:24:12.7486053Z * [new branch] gh/janeyx99/288/orig -> origin/gh/janeyx99/288/orig 2025-08-14T21:24:12.7490663Z * [new branch] gh/janeyx99/289/base -> origin/gh/janeyx99/289/base 2025-08-14T21:24:12.7490836Z * [new branch] gh/janeyx99/289/head -> origin/gh/janeyx99/289/head 2025-08-14T21:24:12.7490968Z * [new branch] gh/janeyx99/289/orig -> origin/gh/janeyx99/289/orig 2025-08-14T21:24:12.7491105Z * [new branch] gh/janeyx99/290/base -> origin/gh/janeyx99/290/base 2025-08-14T21:24:12.7491238Z * [new branch] gh/janeyx99/290/head -> origin/gh/janeyx99/290/head 2025-08-14T21:24:12.7491384Z * [new branch] gh/janeyx99/290/orig -> origin/gh/janeyx99/290/orig 2025-08-14T21:24:12.7492385Z * [new branch] gh/janeyx99/291/base -> origin/gh/janeyx99/291/base 2025-08-14T21:24:12.7492816Z * [new branch] gh/janeyx99/291/head -> origin/gh/janeyx99/291/head 2025-08-14T21:24:12.7496658Z * [new branch] gh/janeyx99/291/orig -> origin/gh/janeyx99/291/orig 2025-08-14T21:24:12.7497001Z * [new branch] gh/janeyx99/292/base -> origin/gh/janeyx99/292/base 2025-08-14T21:24:12.7497344Z * [new branch] gh/janeyx99/292/head -> origin/gh/janeyx99/292/head 2025-08-14T21:24:12.7497508Z * [new branch] gh/janeyx99/292/orig -> origin/gh/janeyx99/292/orig 2025-08-14T21:24:12.7497654Z * [new branch] gh/janeyx99/293/base -> origin/gh/janeyx99/293/base 2025-08-14T21:24:12.7497805Z * [new branch] gh/janeyx99/293/head -> origin/gh/janeyx99/293/head 2025-08-14T21:24:12.7498272Z * [new branch] gh/janeyx99/293/orig -> origin/gh/janeyx99/293/orig 2025-08-14T21:24:12.7499459Z * [new branch] gh/janeyx99/294/base -> origin/gh/janeyx99/294/base 2025-08-14T21:24:12.7499850Z * [new branch] gh/janeyx99/294/head -> origin/gh/janeyx99/294/head 2025-08-14T21:24:12.7502190Z * [new branch] gh/janeyx99/294/orig -> origin/gh/janeyx99/294/orig 2025-08-14T21:24:12.7502361Z * [new branch] gh/janeyx99/295/base -> origin/gh/janeyx99/295/base 2025-08-14T21:24:12.7502658Z * [new branch] gh/janeyx99/295/head -> origin/gh/janeyx99/295/head 2025-08-14T21:24:12.7502961Z * [new branch] gh/janeyx99/295/orig -> origin/gh/janeyx99/295/orig 2025-08-14T21:24:12.7503266Z * [new branch] gh/janeyx99/296/base -> origin/gh/janeyx99/296/base 2025-08-14T21:24:12.7504129Z * [new branch] gh/janeyx99/296/head -> origin/gh/janeyx99/296/head 2025-08-14T21:24:12.7507441Z * [new branch] gh/janeyx99/296/orig -> origin/gh/janeyx99/296/orig 2025-08-14T21:24:12.7507625Z * [new branch] gh/janeyx99/297/base -> origin/gh/janeyx99/297/base 2025-08-14T21:24:12.7507755Z * [new branch] gh/janeyx99/297/head -> origin/gh/janeyx99/297/head 2025-08-14T21:24:12.7507891Z * [new branch] gh/janeyx99/297/orig -> origin/gh/janeyx99/297/orig 2025-08-14T21:24:12.7508051Z * [new branch] gh/janeyx99/298/base -> origin/gh/janeyx99/298/base 2025-08-14T21:24:12.7508779Z * [new branch] gh/janeyx99/298/head -> origin/gh/janeyx99/298/head 2025-08-14T21:24:12.7509262Z * [new branch] gh/janeyx99/298/orig -> origin/gh/janeyx99/298/orig 2025-08-14T21:24:12.7512796Z * [new branch] gh/janeyx99/299/base -> origin/gh/janeyx99/299/base 2025-08-14T21:24:12.7512978Z * [new branch] gh/janeyx99/299/head -> origin/gh/janeyx99/299/head 2025-08-14T21:24:12.7513125Z * [new branch] gh/janeyx99/299/orig -> origin/gh/janeyx99/299/orig 2025-08-14T21:24:12.7513268Z * [new branch] gh/janeyx99/300/base -> origin/gh/janeyx99/300/base 2025-08-14T21:24:12.7513397Z * [new branch] gh/janeyx99/300/head -> origin/gh/janeyx99/300/head 2025-08-14T21:24:12.7514441Z * [new branch] gh/janeyx99/300/orig -> origin/gh/janeyx99/300/orig 2025-08-14T21:24:12.7515684Z * [new branch] gh/janeyx99/88/base -> origin/gh/janeyx99/88/base 2025-08-14T21:24:12.7515823Z * [new branch] gh/janeyx99/88/head -> origin/gh/janeyx99/88/head 2025-08-14T21:24:12.7516722Z * [new branch] gh/janeyx99/88/orig -> origin/gh/janeyx99/88/orig 2025-08-14T21:24:12.7521816Z * [new branch] gh/jansel/360/base -> origin/gh/jansel/360/base 2025-08-14T21:24:12.7522002Z * [new branch] gh/jansel/360/head -> origin/gh/jansel/360/head 2025-08-14T21:24:12.7522158Z * [new branch] gh/jansel/451/base -> origin/gh/jansel/451/base 2025-08-14T21:24:12.7522443Z * [new branch] gh/jansel/451/head -> origin/gh/jansel/451/head 2025-08-14T21:24:12.7522587Z * [new branch] gh/jansel/451/orig -> origin/gh/jansel/451/orig 2025-08-14T21:24:12.7522719Z * [new branch] gh/jansel/462/base -> origin/gh/jansel/462/base 2025-08-14T21:24:12.7522863Z * [new branch] gh/jansel/462/head -> origin/gh/jansel/462/head 2025-08-14T21:24:12.7523000Z * [new branch] gh/jansel/462/orig -> origin/gh/jansel/462/orig 2025-08-14T21:24:12.7523435Z * [new branch] gh/jansel/531/base -> origin/gh/jansel/531/base 2025-08-14T21:24:12.7524229Z * [new branch] gh/jansel/531/head -> origin/gh/jansel/531/head 2025-08-14T21:24:12.7524782Z * [new branch] gh/jansel/531/orig -> origin/gh/jansel/531/orig 2025-08-14T21:24:12.7526602Z * [new branch] gh/jansel/534/base -> origin/gh/jansel/534/base 2025-08-14T21:24:12.7532029Z * [new branch] gh/jansel/534/head -> origin/gh/jansel/534/head 2025-08-14T21:24:12.7536914Z * [new branch] gh/jansel/534/orig -> origin/gh/jansel/534/orig 2025-08-14T21:24:12.7541287Z * [new branch] gh/jbschlosser/226/base -> origin/gh/jbschlosser/226/base 2025-08-14T21:24:12.7543332Z * [new branch] gh/jbschlosser/226/head -> origin/gh/jbschlosser/226/head 2025-08-14T21:24:12.7543583Z * [new branch] gh/jbschlosser/226/orig -> origin/gh/jbschlosser/226/orig 2025-08-14T21:24:12.7543774Z * [new branch] gh/jbschlosser/239/base -> origin/gh/jbschlosser/239/base 2025-08-14T21:24:12.7544125Z * [new branch] gh/jbschlosser/239/head -> origin/gh/jbschlosser/239/head 2025-08-14T21:24:12.7544566Z * [new branch] gh/jbschlosser/239/orig -> origin/gh/jbschlosser/239/orig 2025-08-14T21:24:12.7544842Z * [new branch] gh/jbschlosser/247/base -> origin/gh/jbschlosser/247/base 2025-08-14T21:24:12.7549067Z * [new branch] gh/jbschlosser/247/head -> origin/gh/jbschlosser/247/head 2025-08-14T21:24:12.7551428Z * [new branch] gh/jbschlosser/247/orig -> origin/gh/jbschlosser/247/orig 2025-08-14T21:24:12.7551772Z * [new branch] gh/jbschlosser/248/base -> origin/gh/jbschlosser/248/base 2025-08-14T21:24:12.7557634Z * [new branch] gh/jbschlosser/248/head -> origin/gh/jbschlosser/248/head 2025-08-14T21:24:12.7557822Z * [new branch] gh/jbschlosser/248/orig -> origin/gh/jbschlosser/248/orig 2025-08-14T21:24:12.7557971Z * [new branch] gh/jbschlosser/249/base -> origin/gh/jbschlosser/249/base 2025-08-14T21:24:12.7558107Z * [new branch] gh/jbschlosser/249/head -> origin/gh/jbschlosser/249/head 2025-08-14T21:24:12.7558261Z * [new branch] gh/jbschlosser/249/orig -> origin/gh/jbschlosser/249/orig 2025-08-14T21:24:12.7558405Z * [new branch] gh/jbschlosser/250/base -> origin/gh/jbschlosser/250/base 2025-08-14T21:24:12.7558540Z * [new branch] gh/jbschlosser/250/head -> origin/gh/jbschlosser/250/head 2025-08-14T21:24:12.7558683Z * [new branch] gh/jbschlosser/250/orig -> origin/gh/jbschlosser/250/orig 2025-08-14T21:24:12.7558822Z * [new branch] gh/jiayisunx/57/base -> origin/gh/jiayisunx/57/base 2025-08-14T21:24:12.7558956Z * [new branch] gh/jiayisunx/57/head -> origin/gh/jiayisunx/57/head 2025-08-14T21:24:12.7559092Z * [new branch] gh/jiayisunx/57/orig -> origin/gh/jiayisunx/57/orig 2025-08-14T21:24:12.7559219Z * [new branch] gh/jiayisunx/59/base -> origin/gh/jiayisunx/59/base 2025-08-14T21:24:12.7559355Z * [new branch] gh/jiayisunx/59/head -> origin/gh/jiayisunx/59/head 2025-08-14T21:24:12.7559694Z * [new branch] gh/jiayisunx/59/orig -> origin/gh/jiayisunx/59/orig 2025-08-14T21:24:12.7559826Z * [new branch] gh/jiayisunx/61/base -> origin/gh/jiayisunx/61/base 2025-08-14T21:24:12.7559964Z * [new branch] gh/jiayisunx/61/head -> origin/gh/jiayisunx/61/head 2025-08-14T21:24:12.7560092Z * [new branch] gh/jiayisunx/61/orig -> origin/gh/jiayisunx/61/orig 2025-08-14T21:24:12.7560228Z * [new branch] gh/jiayisunx/63/base -> origin/gh/jiayisunx/63/base 2025-08-14T21:24:12.7560355Z * [new branch] gh/jiayisunx/63/head -> origin/gh/jiayisunx/63/head 2025-08-14T21:24:12.7560482Z * [new branch] gh/jiayisunx/63/orig -> origin/gh/jiayisunx/63/orig 2025-08-14T21:24:12.7560635Z * [new branch] gh/jiayisunx/64/base -> origin/gh/jiayisunx/64/base 2025-08-14T21:24:12.7560766Z * [new branch] gh/jiayisunx/64/head -> origin/gh/jiayisunx/64/head 2025-08-14T21:24:12.7560905Z * [new branch] gh/jiayisunx/64/orig -> origin/gh/jiayisunx/64/orig 2025-08-14T21:24:12.7561036Z * [new branch] gh/jiayisunx/65/base -> origin/gh/jiayisunx/65/base 2025-08-14T21:24:12.7561164Z * [new branch] gh/jiayisunx/65/head -> origin/gh/jiayisunx/65/head 2025-08-14T21:24:12.7561300Z * [new branch] gh/jiayisunx/65/orig -> origin/gh/jiayisunx/65/orig 2025-08-14T21:24:12.7561428Z * [new branch] gh/jiayisunx/66/base -> origin/gh/jiayisunx/66/base 2025-08-14T21:24:12.7561565Z * [new branch] gh/jiayisunx/66/head -> origin/gh/jiayisunx/66/head 2025-08-14T21:24:12.7561692Z * [new branch] gh/jiayisunx/66/orig -> origin/gh/jiayisunx/66/orig 2025-08-14T21:24:12.7561821Z * [new branch] gh/jiayisunx/67/base -> origin/gh/jiayisunx/67/base 2025-08-14T21:24:12.7561998Z * [new branch] gh/jiayisunx/67/head -> origin/gh/jiayisunx/67/head 2025-08-14T21:24:12.7562137Z * [new branch] gh/jiayisunx/67/orig -> origin/gh/jiayisunx/67/orig 2025-08-14T21:24:12.7562274Z * [new branch] gh/jiayisunx/68/base -> origin/gh/jiayisunx/68/base 2025-08-14T21:24:12.7562421Z * [new branch] gh/jiayisunx/68/head -> origin/gh/jiayisunx/68/head 2025-08-14T21:24:12.7562567Z * [new branch] gh/jiayisunx/68/orig -> origin/gh/jiayisunx/68/orig 2025-08-14T21:24:12.7562729Z * [new branch] gh/jjwu@meta.com/1/base -> origin/gh/jjwu@meta.com/1/base 2025-08-14T21:24:12.7562885Z * [new branch] gh/jjwu@meta.com/1/head -> origin/gh/jjwu@meta.com/1/head 2025-08-14T21:24:12.7563041Z * [new branch] gh/justinchuby/111/base -> origin/gh/justinchuby/111/base 2025-08-14T21:24:12.7563204Z * [new branch] gh/justinchuby/111/head -> origin/gh/justinchuby/111/head 2025-08-14T21:24:12.7563362Z * [new branch] gh/justinchuby/111/orig -> origin/gh/justinchuby/111/orig 2025-08-14T21:24:12.7563564Z * [new branch] gh/kurtamohler/32/base -> origin/gh/kurtamohler/32/base 2025-08-14T21:24:12.7564470Z * [new branch] gh/kurtamohler/32/head -> origin/gh/kurtamohler/32/head 2025-08-14T21:24:12.7565020Z * [new branch] gh/kurtamohler/32/orig -> origin/gh/kurtamohler/32/orig 2025-08-14T21:24:12.7566523Z * [new branch] gh/kurtamohler/33/base -> origin/gh/kurtamohler/33/base 2025-08-14T21:24:12.7566959Z * [new branch] gh/kurtamohler/33/head -> origin/gh/kurtamohler/33/head 2025-08-14T21:24:12.7567365Z * [new branch] gh/kurtamohler/33/orig -> origin/gh/kurtamohler/33/orig 2025-08-14T21:24:12.7571599Z * [new branch] gh/kurtamohler/34/base -> origin/gh/kurtamohler/34/base 2025-08-14T21:24:12.7571804Z * [new branch] gh/kurtamohler/34/head -> origin/gh/kurtamohler/34/head 2025-08-14T21:24:12.7572314Z * [new branch] gh/kurtamohler/34/orig -> origin/gh/kurtamohler/34/orig 2025-08-14T21:24:12.7572467Z * [new branch] gh/kurtamohler/40/base -> origin/gh/kurtamohler/40/base 2025-08-14T21:24:12.7572608Z * [new branch] gh/kurtamohler/40/head -> origin/gh/kurtamohler/40/head 2025-08-14T21:24:12.7572745Z * [new branch] gh/kurtamohler/40/orig -> origin/gh/kurtamohler/40/orig 2025-08-14T21:24:12.7573052Z * [new branch] gh/kurtamohler/41/base -> origin/gh/kurtamohler/41/base 2025-08-14T21:24:12.7573391Z * [new branch] gh/kurtamohler/41/head -> origin/gh/kurtamohler/41/head 2025-08-14T21:24:12.7574854Z * [new branch] gh/kurtamohler/41/orig -> origin/gh/kurtamohler/41/orig 2025-08-14T21:24:12.7575136Z * [new branch] gh/kurtamohler/42/base -> origin/gh/kurtamohler/42/base 2025-08-14T21:24:12.7577700Z * [new branch] gh/kurtamohler/42/head -> origin/gh/kurtamohler/42/head 2025-08-14T21:24:12.7578031Z * [new branch] gh/kurtamohler/42/orig -> origin/gh/kurtamohler/42/orig 2025-08-14T21:24:12.7578368Z * [new branch] gh/kurtamohler/43/base -> origin/gh/kurtamohler/43/base 2025-08-14T21:24:12.7578541Z * [new branch] gh/kurtamohler/43/head -> origin/gh/kurtamohler/43/head 2025-08-14T21:24:12.7578682Z * [new branch] gh/kurtamohler/43/orig -> origin/gh/kurtamohler/43/orig 2025-08-14T21:24:12.7580015Z * [new branch] gh/kurtamohler/44/base -> origin/gh/kurtamohler/44/base 2025-08-14T21:24:12.7580336Z * [new branch] gh/kurtamohler/44/head -> origin/gh/kurtamohler/44/head 2025-08-14T21:24:12.7580704Z * [new branch] gh/kurtamohler/44/orig -> origin/gh/kurtamohler/44/orig 2025-08-14T21:24:12.7583103Z * [new branch] gh/kurtamohler/45/base -> origin/gh/kurtamohler/45/base 2025-08-14T21:24:12.7583307Z * [new branch] gh/kurtamohler/45/head -> origin/gh/kurtamohler/45/head 2025-08-14T21:24:12.7583465Z * [new branch] gh/kurtamohler/45/orig -> origin/gh/kurtamohler/45/orig 2025-08-14T21:24:12.7588537Z * [new branch] gh/kurtamohler/46/base -> origin/gh/kurtamohler/46/base 2025-08-14T21:24:12.7592857Z * [new branch] gh/kurtamohler/46/head -> origin/gh/kurtamohler/46/head 2025-08-14T21:24:12.7597115Z * [new branch] gh/kurtamohler/46/orig -> origin/gh/kurtamohler/46/orig 2025-08-14T21:24:12.7598979Z * [new branch] gh/kwen2501/130/base -> origin/gh/kwen2501/130/base 2025-08-14T21:24:12.7599152Z * [new branch] gh/kwen2501/130/head -> origin/gh/kwen2501/130/head 2025-08-14T21:24:12.7599285Z * [new branch] gh/kwen2501/130/orig -> origin/gh/kwen2501/130/orig 2025-08-14T21:24:12.7599443Z * [new branch] gh/kwen2501/142/base -> origin/gh/kwen2501/142/base 2025-08-14T21:24:12.7599586Z * [new branch] gh/kwen2501/142/head -> origin/gh/kwen2501/142/head 2025-08-14T21:24:12.7599721Z * [new branch] gh/kwen2501/142/orig -> origin/gh/kwen2501/142/orig 2025-08-14T21:24:12.7599871Z * [new branch] gh/kwen2501/15/base -> origin/gh/kwen2501/15/base 2025-08-14T21:24:12.7600004Z * [new branch] gh/kwen2501/15/head -> origin/gh/kwen2501/15/head 2025-08-14T21:24:12.7600145Z * [new branch] gh/kwen2501/156/base -> origin/gh/kwen2501/156/base 2025-08-14T21:24:12.7600276Z * [new branch] gh/kwen2501/156/head -> origin/gh/kwen2501/156/head 2025-08-14T21:24:12.7600409Z * [new branch] gh/kwen2501/156/orig -> origin/gh/kwen2501/156/orig 2025-08-14T21:24:12.7600548Z * [new branch] gh/kwen2501/170/base -> origin/gh/kwen2501/170/base 2025-08-14T21:24:12.7600680Z * [new branch] gh/kwen2501/170/head -> origin/gh/kwen2501/170/head 2025-08-14T21:24:12.7600990Z * [new branch] gh/kwen2501/179/base -> origin/gh/kwen2501/179/base 2025-08-14T21:24:12.7601133Z * [new branch] gh/kwen2501/179/head -> origin/gh/kwen2501/179/head 2025-08-14T21:24:12.7601264Z * [new branch] gh/kwen2501/179/orig -> origin/gh/kwen2501/179/orig 2025-08-14T21:24:12.7601411Z * [new branch] gh/kwen2501/181/base -> origin/gh/kwen2501/181/base 2025-08-14T21:24:12.7601546Z * [new branch] gh/kwen2501/181/head -> origin/gh/kwen2501/181/head 2025-08-14T21:24:12.7601676Z * [new branch] gh/kwen2501/181/orig -> origin/gh/kwen2501/181/orig 2025-08-14T21:24:12.7601815Z * [new branch] gh/kwen2501/183/base -> origin/gh/kwen2501/183/base 2025-08-14T21:24:12.7602147Z * [new branch] gh/kwen2501/183/head -> origin/gh/kwen2501/183/head 2025-08-14T21:24:12.7602491Z * [new branch] gh/kwen2501/183/orig -> origin/gh/kwen2501/183/orig 2025-08-14T21:24:12.7603643Z * [new branch] gh/kwen2501/184/base -> origin/gh/kwen2501/184/base 2025-08-14T21:24:12.7603946Z * [new branch] gh/kwen2501/184/head -> origin/gh/kwen2501/184/head 2025-08-14T21:24:12.7604882Z * [new branch] gh/kwen2501/184/orig -> origin/gh/kwen2501/184/orig 2025-08-14T21:24:12.7606203Z * [new branch] gh/kwen2501/186/base -> origin/gh/kwen2501/186/base 2025-08-14T21:24:12.7606933Z * [new branch] gh/kwen2501/186/head -> origin/gh/kwen2501/186/head 2025-08-14T21:24:12.7607666Z * [new branch] gh/kwen2501/186/orig -> origin/gh/kwen2501/186/orig 2025-08-14T21:24:12.7608590Z * [new branch] gh/kwen2501/187/base -> origin/gh/kwen2501/187/base 2025-08-14T21:24:12.7609437Z * [new branch] gh/kwen2501/187/head -> origin/gh/kwen2501/187/head 2025-08-14T21:24:12.7609883Z * [new branch] gh/kwen2501/187/orig -> origin/gh/kwen2501/187/orig 2025-08-14T21:24:12.7611121Z * [new branch] gh/kwen2501/188/base -> origin/gh/kwen2501/188/base 2025-08-14T21:24:12.7611357Z * [new branch] gh/kwen2501/188/head -> origin/gh/kwen2501/188/head 2025-08-14T21:24:12.7612079Z * [new branch] gh/kwen2501/188/orig -> origin/gh/kwen2501/188/orig 2025-08-14T21:24:12.7613142Z * [new branch] gh/kwen2501/194/base -> origin/gh/kwen2501/194/base 2025-08-14T21:24:12.7613447Z * [new branch] gh/kwen2501/194/head -> origin/gh/kwen2501/194/head 2025-08-14T21:24:12.7614326Z * [new branch] gh/kwen2501/194/orig -> origin/gh/kwen2501/194/orig 2025-08-14T21:24:12.7615536Z * [new branch] gh/kwen2501/195/base -> origin/gh/kwen2501/195/base 2025-08-14T21:24:12.7615694Z * [new branch] gh/kwen2501/195/head -> origin/gh/kwen2501/195/head 2025-08-14T21:24:12.7616722Z * [new branch] gh/kwen2501/195/orig -> origin/gh/kwen2501/195/orig 2025-08-14T21:24:12.7617376Z * [new branch] gh/kwen2501/196/base -> origin/gh/kwen2501/196/base 2025-08-14T21:24:12.7617972Z * [new branch] gh/kwen2501/196/head -> origin/gh/kwen2501/196/head 2025-08-14T21:24:12.7618933Z * [new branch] gh/kwen2501/196/orig -> origin/gh/kwen2501/196/orig 2025-08-14T21:24:12.7619942Z * [new branch] gh/kwen2501/197/base -> origin/gh/kwen2501/197/base 2025-08-14T21:24:12.7620186Z * [new branch] gh/kwen2501/197/head -> origin/gh/kwen2501/197/head 2025-08-14T21:24:12.7622545Z * [new branch] gh/kwen2501/197/orig -> origin/gh/kwen2501/197/orig 2025-08-14T21:24:12.7623144Z * [new branch] gh/kwen2501/198/base -> origin/gh/kwen2501/198/base 2025-08-14T21:24:12.7623338Z * [new branch] gh/kwen2501/198/head -> origin/gh/kwen2501/198/head 2025-08-14T21:24:12.7623694Z * [new branch] gh/kwen2501/198/orig -> origin/gh/kwen2501/198/orig 2025-08-14T21:24:12.7624255Z * [new branch] gh/kwen2501/199/base -> origin/gh/kwen2501/199/base 2025-08-14T21:24:12.7627333Z * [new branch] gh/kwen2501/199/head -> origin/gh/kwen2501/199/head 2025-08-14T21:24:12.7627510Z * [new branch] gh/kwen2501/199/orig -> origin/gh/kwen2501/199/orig 2025-08-14T21:24:12.7627666Z * [new branch] gh/kwen2501/200/base -> origin/gh/kwen2501/200/base 2025-08-14T21:24:12.7627812Z * [new branch] gh/kwen2501/200/head -> origin/gh/kwen2501/200/head 2025-08-14T21:24:12.7628175Z * [new branch] gh/kwen2501/200/orig -> origin/gh/kwen2501/200/orig 2025-08-14T21:24:12.7632781Z * [new branch] gh/kwen2501/201/base -> origin/gh/kwen2501/201/base 2025-08-14T21:24:12.7633087Z * [new branch] gh/kwen2501/201/head -> origin/gh/kwen2501/201/head 2025-08-14T21:24:12.7633254Z * [new branch] gh/kwen2501/201/orig -> origin/gh/kwen2501/201/orig 2025-08-14T21:24:12.7633382Z * [new branch] gh/kwen2501/202/base -> origin/gh/kwen2501/202/base 2025-08-14T21:24:12.7633648Z * [new branch] gh/kwen2501/202/head -> origin/gh/kwen2501/202/head 2025-08-14T21:24:12.7633965Z * [new branch] gh/kwen2501/202/orig -> origin/gh/kwen2501/202/orig 2025-08-14T21:24:12.7634098Z * [new branch] gh/kwen2501/203/base -> origin/gh/kwen2501/203/base 2025-08-14T21:24:12.7634233Z * [new branch] gh/kwen2501/203/head -> origin/gh/kwen2501/203/head 2025-08-14T21:24:12.7634884Z * [new branch] gh/kwen2501/203/orig -> origin/gh/kwen2501/203/orig 2025-08-14T21:24:12.7638968Z * [new branch] gh/laithsakka/152/base -> origin/gh/laithsakka/152/base 2025-08-14T21:24:12.7639312Z * [new branch] gh/laithsakka/152/head -> origin/gh/laithsakka/152/head 2025-08-14T21:24:12.7639473Z * [new branch] gh/laithsakka/152/orig -> origin/gh/laithsakka/152/orig 2025-08-14T21:24:12.7639621Z * [new branch] gh/laithsakka/156/base -> origin/gh/laithsakka/156/base 2025-08-14T21:24:12.7639878Z * [new branch] gh/laithsakka/156/head -> origin/gh/laithsakka/156/head 2025-08-14T21:24:12.7640097Z * [new branch] gh/laithsakka/156/orig -> origin/gh/laithsakka/156/orig 2025-08-14T21:24:12.7641006Z * [new branch] gh/laithsakka/159/base -> origin/gh/laithsakka/159/base 2025-08-14T21:24:12.7641450Z * [new branch] gh/laithsakka/159/head -> origin/gh/laithsakka/159/head 2025-08-14T21:24:12.7644307Z * [new branch] gh/laithsakka/159/orig -> origin/gh/laithsakka/159/orig 2025-08-14T21:24:12.7644674Z * [new branch] gh/laithsakka/160/base -> origin/gh/laithsakka/160/base 2025-08-14T21:24:12.7644859Z * [new branch] gh/laithsakka/160/head -> origin/gh/laithsakka/160/head 2025-08-14T21:24:12.7645037Z * [new branch] gh/laithsakka/160/orig -> origin/gh/laithsakka/160/orig 2025-08-14T21:24:12.7645486Z * [new branch] gh/laithsakka/178/base -> origin/gh/laithsakka/178/base 2025-08-14T21:24:12.7646429Z * [new branch] gh/laithsakka/178/head -> origin/gh/laithsakka/178/head 2025-08-14T21:24:12.7649610Z * [new branch] gh/laithsakka/178/orig -> origin/gh/laithsakka/178/orig 2025-08-14T21:24:12.7649956Z * [new branch] gh/laithsakka/191/base -> origin/gh/laithsakka/191/base 2025-08-14T21:24:12.7650133Z * [new branch] gh/laithsakka/191/head -> origin/gh/laithsakka/191/head 2025-08-14T21:24:12.7650288Z * [new branch] gh/laithsakka/191/orig -> origin/gh/laithsakka/191/orig 2025-08-14T21:24:12.7650811Z * [new branch] gh/laithsakka/234/base -> origin/gh/laithsakka/234/base 2025-08-14T21:24:12.7651132Z * [new branch] gh/laithsakka/234/head -> origin/gh/laithsakka/234/head 2025-08-14T21:24:12.7651961Z * [new branch] gh/laithsakka/234/orig -> origin/gh/laithsakka/234/orig 2025-08-14T21:24:12.7656377Z * [new branch] gh/laithsakka/237/base -> origin/gh/laithsakka/237/base 2025-08-14T21:24:12.7656725Z * [new branch] gh/laithsakka/237/head -> origin/gh/laithsakka/237/head 2025-08-14T21:24:12.7656981Z * [new branch] gh/laithsakka/237/orig -> origin/gh/laithsakka/237/orig 2025-08-14T21:24:12.7657150Z * [new branch] gh/laithsakka/238/base -> origin/gh/laithsakka/238/base 2025-08-14T21:24:12.7657390Z * [new branch] gh/laithsakka/238/head -> origin/gh/laithsakka/238/head 2025-08-14T21:24:12.7657581Z * [new branch] gh/laithsakka/238/orig -> origin/gh/laithsakka/238/orig 2025-08-14T21:24:12.7658138Z * [new branch] gh/laithsakka/239/base -> origin/gh/laithsakka/239/base 2025-08-14T21:24:12.7658339Z * [new branch] gh/laithsakka/239/head -> origin/gh/laithsakka/239/head 2025-08-14T21:24:12.7658482Z * [new branch] gh/laithsakka/239/orig -> origin/gh/laithsakka/239/orig 2025-08-14T21:24:12.7660372Z * [new branch] gh/laithsakka/240/base -> origin/gh/laithsakka/240/base 2025-08-14T21:24:12.7660728Z * [new branch] gh/laithsakka/240/head -> origin/gh/laithsakka/240/head 2025-08-14T21:24:12.7660984Z * [new branch] gh/laithsakka/240/orig -> origin/gh/laithsakka/240/orig 2025-08-14T21:24:12.7661159Z * [new branch] gh/laithsakka/242/base -> origin/gh/laithsakka/242/base 2025-08-14T21:24:12.7661687Z * [new branch] gh/laithsakka/242/head -> origin/gh/laithsakka/242/head 2025-08-14T21:24:12.7663410Z * [new branch] gh/laithsakka/242/orig -> origin/gh/laithsakka/242/orig 2025-08-14T21:24:12.7663754Z * [new branch] gh/laithsakka/243/base -> origin/gh/laithsakka/243/base 2025-08-14T21:24:12.7663929Z * [new branch] gh/laithsakka/243/head -> origin/gh/laithsakka/243/head 2025-08-14T21:24:12.7666152Z * [new branch] gh/laithsakka/243/orig -> origin/gh/laithsakka/243/orig 2025-08-14T21:24:12.7666349Z * [new branch] gh/laithsakka/244/base -> origin/gh/laithsakka/244/base 2025-08-14T21:24:12.7666498Z * [new branch] gh/laithsakka/244/head -> origin/gh/laithsakka/244/head 2025-08-14T21:24:12.7666883Z * [new branch] gh/laithsakka/244/orig -> origin/gh/laithsakka/244/orig 2025-08-14T21:24:12.7671209Z * [new branch] gh/laithsakka/245/base -> origin/gh/laithsakka/245/base 2025-08-14T21:24:12.7671558Z * [new branch] gh/laithsakka/245/head -> origin/gh/laithsakka/245/head 2025-08-14T21:24:12.7671815Z * [new branch] gh/laithsakka/245/orig -> origin/gh/laithsakka/245/orig 2025-08-14T21:24:12.7672508Z * [new branch] gh/laithsakka/246/base -> origin/gh/laithsakka/246/base 2025-08-14T21:24:12.7672706Z * [new branch] gh/laithsakka/246/head -> origin/gh/laithsakka/246/head 2025-08-14T21:24:12.7672876Z * [new branch] gh/laithsakka/246/orig -> origin/gh/laithsakka/246/orig 2025-08-14T21:24:12.7673028Z * [new branch] gh/laithsakka/247/base -> origin/gh/laithsakka/247/base 2025-08-14T21:24:12.7673352Z * [new branch] gh/laithsakka/247/head -> origin/gh/laithsakka/247/head 2025-08-14T21:24:12.7675420Z * [new branch] gh/laithsakka/247/orig -> origin/gh/laithsakka/247/orig 2025-08-14T21:24:12.7675598Z * [new branch] gh/laithsakka/248/base -> origin/gh/laithsakka/248/base 2025-08-14T21:24:12.7675752Z * [new branch] gh/laithsakka/248/head -> origin/gh/laithsakka/248/head 2025-08-14T21:24:12.7676087Z * [new branch] gh/laithsakka/248/orig -> origin/gh/laithsakka/248/orig 2025-08-14T21:24:12.7677732Z * [new branch] gh/laithsakka/249/base -> origin/gh/laithsakka/249/base 2025-08-14T21:24:12.7677909Z * [new branch] gh/laithsakka/249/head -> origin/gh/laithsakka/249/head 2025-08-14T21:24:12.7678374Z * [new branch] gh/laithsakka/249/orig -> origin/gh/laithsakka/249/orig 2025-08-14T21:24:12.7680227Z * [new branch] gh/laithsakka/250/base -> origin/gh/laithsakka/250/base 2025-08-14T21:24:12.7680409Z * [new branch] gh/laithsakka/250/head -> origin/gh/laithsakka/250/head 2025-08-14T21:24:12.7680706Z * [new branch] gh/laithsakka/250/orig -> origin/gh/laithsakka/250/orig 2025-08-14T21:24:12.7681514Z * [new branch] gh/laithsakka/251/base -> origin/gh/laithsakka/251/base 2025-08-14T21:24:12.7682017Z * [new branch] gh/laithsakka/251/head -> origin/gh/laithsakka/251/head 2025-08-14T21:24:12.7682987Z * [new branch] gh/laithsakka/251/orig -> origin/gh/laithsakka/251/orig 2025-08-14T21:24:12.7684274Z * [new branch] gh/laithsakka/252/base -> origin/gh/laithsakka/252/base 2025-08-14T21:24:12.7684659Z * [new branch] gh/laithsakka/252/head -> origin/gh/laithsakka/252/head 2025-08-14T21:24:12.7685986Z * [new branch] gh/laithsakka/252/orig -> origin/gh/laithsakka/252/orig 2025-08-14T21:24:12.7686288Z * [new branch] gh/laithsakka/253/base -> origin/gh/laithsakka/253/base 2025-08-14T21:24:12.7687369Z * [new branch] gh/laithsakka/253/head -> origin/gh/laithsakka/253/head 2025-08-14T21:24:12.7687546Z * [new branch] gh/laithsakka/253/orig -> origin/gh/laithsakka/253/orig 2025-08-14T21:24:12.7690469Z * [new branch] gh/laithsakka/254/base -> origin/gh/laithsakka/254/base 2025-08-14T21:24:12.7690679Z * [new branch] gh/laithsakka/254/head -> origin/gh/laithsakka/254/head 2025-08-14T21:24:12.7690834Z * [new branch] gh/laithsakka/254/orig -> origin/gh/laithsakka/254/orig 2025-08-14T21:24:12.7690982Z * [new branch] gh/laithsakka/255/base -> origin/gh/laithsakka/255/base 2025-08-14T21:24:12.7691449Z * [new branch] gh/laithsakka/255/head -> origin/gh/laithsakka/255/head 2025-08-14T21:24:12.7692233Z * [new branch] gh/laithsakka/255/orig -> origin/gh/laithsakka/255/orig 2025-08-14T21:24:12.7693294Z * [new branch] gh/laithsakka/256/base -> origin/gh/laithsakka/256/base 2025-08-14T21:24:12.7693787Z * [new branch] gh/laithsakka/256/head -> origin/gh/laithsakka/256/head 2025-08-14T21:24:12.7694490Z * [new branch] gh/laithsakka/256/orig -> origin/gh/laithsakka/256/orig 2025-08-14T21:24:12.7695614Z * [new branch] gh/laithsakka/257/base -> origin/gh/laithsakka/257/base 2025-08-14T21:24:12.7695938Z * [new branch] gh/laithsakka/257/head -> origin/gh/laithsakka/257/head 2025-08-14T21:24:12.7697020Z * [new branch] gh/laithsakka/257/orig -> origin/gh/laithsakka/257/orig 2025-08-14T21:24:12.7697867Z * [new branch] gh/laithsakka/258/base -> origin/gh/laithsakka/258/base 2025-08-14T21:24:12.7698083Z * [new branch] gh/laithsakka/258/head -> origin/gh/laithsakka/258/head 2025-08-14T21:24:12.7699173Z * [new branch] gh/laithsakka/258/orig -> origin/gh/laithsakka/258/orig 2025-08-14T21:24:12.7700059Z * [new branch] gh/laithsakka/259/base -> origin/gh/laithsakka/259/base 2025-08-14T21:24:12.7700309Z * [new branch] gh/laithsakka/259/head -> origin/gh/laithsakka/259/head 2025-08-14T21:24:12.7701283Z * [new branch] gh/laithsakka/259/orig -> origin/gh/laithsakka/259/orig 2025-08-14T21:24:12.7701905Z * [new branch] gh/laithsakka/260/base -> origin/gh/laithsakka/260/base 2025-08-14T21:24:12.7702576Z * [new branch] gh/laithsakka/260/head -> origin/gh/laithsakka/260/head 2025-08-14T21:24:12.7703239Z * [new branch] gh/laithsakka/260/orig -> origin/gh/laithsakka/260/orig 2025-08-14T21:24:12.7704189Z * [new branch] gh/laithsakka/261/base -> origin/gh/laithsakka/261/base 2025-08-14T21:24:12.7704550Z * [new branch] gh/laithsakka/261/head -> origin/gh/laithsakka/261/head 2025-08-14T21:24:12.7705489Z * [new branch] gh/laithsakka/261/orig -> origin/gh/laithsakka/261/orig 2025-08-14T21:24:12.7706061Z * [new branch] gh/laithsakka/262/base -> origin/gh/laithsakka/262/base 2025-08-14T21:24:12.7707041Z * [new branch] gh/laithsakka/262/head -> origin/gh/laithsakka/262/head 2025-08-14T21:24:12.7707295Z * [new branch] gh/laithsakka/262/orig -> origin/gh/laithsakka/262/orig 2025-08-14T21:24:12.7708607Z * [new branch] gh/laithsakka/28/base -> origin/gh/laithsakka/28/base 2025-08-14T21:24:12.7709132Z * [new branch] gh/laithsakka/29/base -> origin/gh/laithsakka/29/base 2025-08-14T21:24:12.7710116Z * [new branch] gh/laithsakka/30/base -> origin/gh/laithsakka/30/base 2025-08-14T21:24:12.7710500Z * [new branch] gh/laithsakka/30/head -> origin/gh/laithsakka/30/head 2025-08-14T21:24:12.7711513Z * [new branch] gh/laithsakka/31/base -> origin/gh/laithsakka/31/base 2025-08-14T21:24:12.7711793Z * [new branch] gh/laithsakka/31/head -> origin/gh/laithsakka/31/head 2025-08-14T21:24:12.7712877Z * [new branch] gh/laithsakka/32/base -> origin/gh/laithsakka/32/base 2025-08-14T21:24:12.7713272Z * [new branch] gh/laithsakka/32/head -> origin/gh/laithsakka/32/head 2025-08-14T21:24:12.7716401Z * [new branch] gh/lucaskabela/1/base -> origin/gh/lucaskabela/1/base 2025-08-14T21:24:12.7716700Z * [new branch] gh/lucaskabela/1/head -> origin/gh/lucaskabela/1/head 2025-08-14T21:24:12.7718369Z * [new branch] gh/lucaskabela/10/base -> origin/gh/lucaskabela/10/base 2025-08-14T21:24:12.7718633Z * [new branch] gh/lucaskabela/10/head -> origin/gh/lucaskabela/10/head 2025-08-14T21:24:12.7719734Z * [new branch] gh/lucaskabela/10/orig -> origin/gh/lucaskabela/10/orig 2025-08-14T21:24:12.7719974Z * [new branch] gh/lucaskabela/11/base -> origin/gh/lucaskabela/11/base 2025-08-14T21:24:12.7720998Z * [new branch] gh/lucaskabela/11/head -> origin/gh/lucaskabela/11/head 2025-08-14T21:24:12.7721329Z * [new branch] gh/lucaskabela/11/orig -> origin/gh/lucaskabela/11/orig 2025-08-14T21:24:12.7722559Z * [new branch] gh/lucaskabela/12/base -> origin/gh/lucaskabela/12/base 2025-08-14T21:24:12.7722814Z * [new branch] gh/lucaskabela/12/head -> origin/gh/lucaskabela/12/head 2025-08-14T21:24:12.7729978Z * [new branch] gh/lucaskabela/12/orig -> origin/gh/lucaskabela/12/orig 2025-08-14T21:24:12.7730147Z * [new branch] gh/lucaskabela/13/base -> origin/gh/lucaskabela/13/base 2025-08-14T21:24:12.7730298Z * [new branch] gh/lucaskabela/13/head -> origin/gh/lucaskabela/13/head 2025-08-14T21:24:12.7730440Z * [new branch] gh/lucaskabela/13/orig -> origin/gh/lucaskabela/13/orig 2025-08-14T21:24:12.7730580Z * [new branch] gh/lucaskabela/14/base -> origin/gh/lucaskabela/14/base 2025-08-14T21:24:12.7730865Z * [new branch] gh/lucaskabela/14/head -> origin/gh/lucaskabela/14/head 2025-08-14T21:24:12.7735811Z * [new branch] gh/lucaskabela/14/orig -> origin/gh/lucaskabela/14/orig 2025-08-14T21:24:12.7740001Z * [new branch] gh/lucaskabela/15/base -> origin/gh/lucaskabela/15/base 2025-08-14T21:24:12.7744384Z * [new branch] gh/lucaskabela/15/head -> origin/gh/lucaskabela/15/head 2025-08-14T21:24:12.7749688Z * [new branch] gh/lucaskabela/15/orig -> origin/gh/lucaskabela/15/orig 2025-08-14T21:24:12.7755119Z * [new branch] gh/lucaskabela/16/base -> origin/gh/lucaskabela/16/base 2025-08-14T21:24:12.7760079Z * [new branch] gh/lucaskabela/16/head -> origin/gh/lucaskabela/16/head 2025-08-14T21:24:12.7762534Z * [new branch] gh/lucaskabela/16/orig -> origin/gh/lucaskabela/16/orig 2025-08-14T21:24:12.7762760Z * [new branch] gh/lucaskabela/17/base -> origin/gh/lucaskabela/17/base 2025-08-14T21:24:12.7762912Z * [new branch] gh/lucaskabela/17/head -> origin/gh/lucaskabela/17/head 2025-08-14T21:24:12.7763108Z * [new branch] gh/lucaskabela/17/orig -> origin/gh/lucaskabela/17/orig 2025-08-14T21:24:12.7763274Z * [new branch] gh/lucaskabela/2/base -> origin/gh/lucaskabela/2/base 2025-08-14T21:24:12.7763419Z * [new branch] gh/lucaskabela/2/head -> origin/gh/lucaskabela/2/head 2025-08-14T21:24:12.7763567Z * [new branch] gh/lucaskabela/2/orig -> origin/gh/lucaskabela/2/orig 2025-08-14T21:24:12.7763702Z * [new branch] gh/lucaskabela/3/base -> origin/gh/lucaskabela/3/base 2025-08-14T21:24:12.7763847Z * [new branch] gh/lucaskabela/3/head -> origin/gh/lucaskabela/3/head 2025-08-14T21:24:12.7763986Z * [new branch] gh/lucaskabela/3/orig -> origin/gh/lucaskabela/3/orig 2025-08-14T21:24:12.7764134Z * [new branch] gh/lucaskabela/4/base -> origin/gh/lucaskabela/4/base 2025-08-14T21:24:12.7764286Z * [new branch] gh/lucaskabela/4/head -> origin/gh/lucaskabela/4/head 2025-08-14T21:24:12.7764667Z * [new branch] gh/lucaskabela/4/orig -> origin/gh/lucaskabela/4/orig 2025-08-14T21:24:12.7764834Z * [new branch] gh/lucaskabela/5/base -> origin/gh/lucaskabela/5/base 2025-08-14T21:24:12.7764995Z * [new branch] gh/lucaskabela/5/head -> origin/gh/lucaskabela/5/head 2025-08-14T21:24:12.7765143Z * [new branch] gh/lucaskabela/5/orig -> origin/gh/lucaskabela/5/orig 2025-08-14T21:24:12.7765298Z * [new branch] gh/lucaskabela/6/base -> origin/gh/lucaskabela/6/base 2025-08-14T21:24:12.7765446Z * [new branch] gh/lucaskabela/6/head -> origin/gh/lucaskabela/6/head 2025-08-14T21:24:12.7765787Z * [new branch] gh/lucaskabela/6/orig -> origin/gh/lucaskabela/6/orig 2025-08-14T21:24:12.7765950Z * [new branch] gh/lucaskabela/7/base -> origin/gh/lucaskabela/7/base 2025-08-14T21:24:12.7766085Z * [new branch] gh/lucaskabela/7/head -> origin/gh/lucaskabela/7/head 2025-08-14T21:24:12.7766232Z * [new branch] gh/lucaskabela/7/orig -> origin/gh/lucaskabela/7/orig 2025-08-14T21:24:12.7766374Z * [new branch] gh/lucaskabela/8/base -> origin/gh/lucaskabela/8/base 2025-08-14T21:24:12.7766509Z * [new branch] gh/lucaskabela/8/head -> origin/gh/lucaskabela/8/head 2025-08-14T21:24:12.7766654Z * [new branch] gh/lucaskabela/8/orig -> origin/gh/lucaskabela/8/orig 2025-08-14T21:24:12.7766790Z * [new branch] gh/lucaskabela/9/base -> origin/gh/lucaskabela/9/base 2025-08-14T21:24:12.7766933Z * [new branch] gh/lucaskabela/9/head -> origin/gh/lucaskabela/9/head 2025-08-14T21:24:12.7767074Z * [new branch] gh/lucaskabela/9/orig -> origin/gh/lucaskabela/9/orig 2025-08-14T21:24:12.7767198Z * [new branch] gh/lw/1/base -> origin/gh/lw/1/base 2025-08-14T21:24:12.7767320Z * [new branch] gh/lw/1/head -> origin/gh/lw/1/head 2025-08-14T21:24:12.7767433Z * [new branch] gh/lw/1/orig -> origin/gh/lw/1/orig 2025-08-14T21:24:12.7767595Z * [new branch] gh/lw/2/base -> origin/gh/lw/2/base 2025-08-14T21:24:12.7767714Z * [new branch] gh/lw/2/head -> origin/gh/lw/2/head 2025-08-14T21:24:12.7767824Z * [new branch] gh/lw/2/orig -> origin/gh/lw/2/orig 2025-08-14T21:24:12.7767942Z * [new branch] gh/lw/3/base -> origin/gh/lw/3/base 2025-08-14T21:24:12.7768052Z * [new branch] gh/lw/3/head -> origin/gh/lw/3/head 2025-08-14T21:24:12.7768163Z * [new branch] gh/lw/3/orig -> origin/gh/lw/3/orig 2025-08-14T21:24:12.7768303Z * [new branch] gh/malfet/14/base -> origin/gh/malfet/14/base 2025-08-14T21:24:12.7768437Z * [new branch] gh/malfet/330/base -> origin/gh/malfet/330/base 2025-08-14T21:24:12.7768575Z * [new branch] gh/malfet/330/head -> origin/gh/malfet/330/head 2025-08-14T21:24:12.7768706Z * [new branch] gh/malfet/330/orig -> origin/gh/malfet/330/orig 2025-08-14T21:24:12.7768845Z * [new branch] gh/malfet/396/base -> origin/gh/malfet/396/base 2025-08-14T21:24:12.7768981Z * [new branch] gh/malfet/396/head -> origin/gh/malfet/396/head 2025-08-14T21:24:12.7769107Z * [new branch] gh/malfet/396/orig -> origin/gh/malfet/396/orig 2025-08-14T21:24:12.7769238Z * [new branch] gh/malfet/397/base -> origin/gh/malfet/397/base 2025-08-14T21:24:12.7769390Z * [new branch] gh/malfet/397/head -> origin/gh/malfet/397/head 2025-08-14T21:24:12.7769514Z * [new branch] gh/malfet/397/orig -> origin/gh/malfet/397/orig 2025-08-14T21:24:12.7769643Z * [new branch] gh/malfet/398/base -> origin/gh/malfet/398/base 2025-08-14T21:24:12.7769807Z * [new branch] gh/malfet/398/head -> origin/gh/malfet/398/head 2025-08-14T21:24:12.7769946Z * [new branch] gh/malfet/398/orig -> origin/gh/malfet/398/orig 2025-08-14T21:24:12.7770073Z * [new branch] gh/malfet/399/base -> origin/gh/malfet/399/base 2025-08-14T21:24:12.7770600Z * [new branch] gh/malfet/399/head -> origin/gh/malfet/399/head 2025-08-14T21:24:12.7772670Z * [new branch] gh/malfet/399/orig -> origin/gh/malfet/399/orig 2025-08-14T21:24:12.7772967Z * [new branch] gh/malfet/414/base -> origin/gh/malfet/414/base 2025-08-14T21:24:12.7773121Z * [new branch] gh/malfet/414/head -> origin/gh/malfet/414/head 2025-08-14T21:24:12.7773417Z * [new branch] gh/malfet/414/orig -> origin/gh/malfet/414/orig 2025-08-14T21:24:12.7774892Z * [new branch] gh/malfet/417/base -> origin/gh/malfet/417/base 2025-08-14T21:24:12.7775218Z * [new branch] gh/malfet/417/head -> origin/gh/malfet/417/head 2025-08-14T21:24:12.7775486Z * [new branch] gh/malfet/417/orig -> origin/gh/malfet/417/orig 2025-08-14T21:24:12.7777567Z * [new branch] gh/malfet/418/base -> origin/gh/malfet/418/base 2025-08-14T21:24:12.7777881Z * [new branch] gh/malfet/418/head -> origin/gh/malfet/418/head 2025-08-14T21:24:12.7778172Z * [new branch] gh/malfet/418/orig -> origin/gh/malfet/418/orig 2025-08-14T21:24:12.7778387Z * [new branch] gh/malfet/422/base -> origin/gh/malfet/422/base 2025-08-14T21:24:12.7779765Z * [new branch] gh/malfet/422/head -> origin/gh/malfet/422/head 2025-08-14T21:24:12.7780086Z * [new branch] gh/malfet/422/orig -> origin/gh/malfet/422/orig 2025-08-14T21:24:12.7782890Z * [new branch] gh/malfet/438/base -> origin/gh/malfet/438/base 2025-08-14T21:24:12.7783225Z * [new branch] gh/malfet/438/head -> origin/gh/malfet/438/head 2025-08-14T21:24:12.7783629Z * [new branch] gh/malfet/438/orig -> origin/gh/malfet/438/orig 2025-08-14T21:24:12.7783775Z * [new branch] gh/malfet/439/base -> origin/gh/malfet/439/base 2025-08-14T21:24:12.7783983Z * [new branch] gh/malfet/439/head -> origin/gh/malfet/439/head 2025-08-14T21:24:12.7784123Z * [new branch] gh/malfet/439/orig -> origin/gh/malfet/439/orig 2025-08-14T21:24:12.7785472Z * [new branch] gh/malfet/440/base -> origin/gh/malfet/440/base 2025-08-14T21:24:12.7785672Z * [new branch] gh/malfet/440/head -> origin/gh/malfet/440/head 2025-08-14T21:24:12.7788021Z * [new branch] gh/malfet/440/orig -> origin/gh/malfet/440/orig 2025-08-14T21:24:12.7788341Z * [new branch] gh/malfet/441/base -> origin/gh/malfet/441/base 2025-08-14T21:24:12.7788515Z * [new branch] gh/malfet/441/head -> origin/gh/malfet/441/head 2025-08-14T21:24:12.7789689Z * [new branch] gh/malfet/441/orig -> origin/gh/malfet/441/orig 2025-08-14T21:24:12.7790069Z * [new branch] gh/malfet/442/base -> origin/gh/malfet/442/base 2025-08-14T21:24:12.7792690Z * [new branch] gh/malfet/442/head -> origin/gh/malfet/442/head 2025-08-14T21:24:12.7793012Z * [new branch] gh/malfet/442/orig -> origin/gh/malfet/442/orig 2025-08-14T21:24:12.7793183Z * [new branch] gh/malfet/443/base -> origin/gh/malfet/443/base 2025-08-14T21:24:12.7793345Z * [new branch] gh/malfet/443/head -> origin/gh/malfet/443/head 2025-08-14T21:24:12.7793480Z * [new branch] gh/malfet/443/orig -> origin/gh/malfet/443/orig 2025-08-14T21:24:12.7797699Z * [new branch] gh/malfet/444/base -> origin/gh/malfet/444/base 2025-08-14T21:24:12.7798042Z * [new branch] gh/malfet/444/head -> origin/gh/malfet/444/head 2025-08-14T21:24:12.7798265Z * [new branch] gh/malfet/444/orig -> origin/gh/malfet/444/orig 2025-08-14T21:24:12.7798431Z * [new branch] gh/malfet/445/base -> origin/gh/malfet/445/base 2025-08-14T21:24:12.7798560Z * [new branch] gh/malfet/445/head -> origin/gh/malfet/445/head 2025-08-14T21:24:12.7831818Z * [new branch] gh/malfet/445/orig -> origin/gh/malfet/445/orig 2025-08-14T21:24:12.7832086Z * [new branch] gh/malfet/446/base -> origin/gh/malfet/446/base 2025-08-14T21:24:12.7835054Z * [new branch] gh/malfet/446/head -> origin/gh/malfet/446/head 2025-08-14T21:24:12.7838167Z * [new branch] gh/malfet/446/orig -> origin/gh/malfet/446/orig 2025-08-14T21:24:12.7838352Z * [new branch] gh/malfet/447/base -> origin/gh/malfet/447/base 2025-08-14T21:24:12.7838484Z * [new branch] gh/malfet/447/head -> origin/gh/malfet/447/head 2025-08-14T21:24:12.7838617Z * [new branch] gh/malfet/448/base -> origin/gh/malfet/448/base 2025-08-14T21:24:12.7838745Z * [new branch] gh/malfet/448/head -> origin/gh/malfet/448/head 2025-08-14T21:24:12.7838870Z * [new branch] gh/malfet/449/base -> origin/gh/malfet/449/base 2025-08-14T21:24:12.7839006Z * [new branch] gh/malfet/449/head -> origin/gh/malfet/449/head 2025-08-14T21:24:12.7839146Z * [new branch] gh/malfet/450/base -> origin/gh/malfet/450/base 2025-08-14T21:24:12.7839291Z * [new branch] gh/malfet/450/head -> origin/gh/malfet/450/head 2025-08-14T21:24:12.7839431Z * [new branch] gh/malfet/451/base -> origin/gh/malfet/451/base 2025-08-14T21:24:12.7839578Z * [new branch] gh/malfet/451/head -> origin/gh/malfet/451/head 2025-08-14T21:24:12.7839942Z * [new branch] gh/malfet/452/base -> origin/gh/malfet/452/base 2025-08-14T21:24:12.7840085Z * [new branch] gh/malfet/452/head -> origin/gh/malfet/452/head 2025-08-14T21:24:12.7840225Z * [new branch] gh/malfet/452/orig -> origin/gh/malfet/452/orig 2025-08-14T21:24:12.7840376Z * [new branch] gh/malfet/453/base -> origin/gh/malfet/453/base 2025-08-14T21:24:12.7840517Z * [new branch] gh/malfet/453/head -> origin/gh/malfet/453/head 2025-08-14T21:24:12.7840661Z * [new branch] gh/malfet/453/orig -> origin/gh/malfet/453/orig 2025-08-14T21:24:12.7840801Z * [new branch] gh/malfet/454/base -> origin/gh/malfet/454/base 2025-08-14T21:24:12.7840942Z * [new branch] gh/malfet/454/head -> origin/gh/malfet/454/head 2025-08-14T21:24:12.7841093Z * [new branch] gh/malfet/454/orig -> origin/gh/malfet/454/orig 2025-08-14T21:24:12.7841227Z * [new branch] gh/malfet/455/base -> origin/gh/malfet/455/base 2025-08-14T21:24:12.7841373Z * [new branch] gh/malfet/455/head -> origin/gh/malfet/455/head 2025-08-14T21:24:12.7841515Z * [new branch] gh/malfet/455/orig -> origin/gh/malfet/455/orig 2025-08-14T21:24:12.7841657Z * [new branch] gh/malfet/456/base -> origin/gh/malfet/456/base 2025-08-14T21:24:12.7841801Z * [new branch] gh/malfet/456/head -> origin/gh/malfet/456/head 2025-08-14T21:24:12.7841943Z * [new branch] gh/malfet/456/orig -> origin/gh/malfet/456/orig 2025-08-14T21:24:12.7842073Z * [new branch] gh/malfet/457/base -> origin/gh/malfet/457/base 2025-08-14T21:24:12.7842221Z * [new branch] gh/malfet/457/head -> origin/gh/malfet/457/head 2025-08-14T21:24:12.7842422Z * [new branch] gh/malfet/457/orig -> origin/gh/malfet/457/orig 2025-08-14T21:24:12.7842581Z * [new branch] gh/malfet/458/base -> origin/gh/malfet/458/base 2025-08-14T21:24:12.7842724Z * [new branch] gh/malfet/458/head -> origin/gh/malfet/458/head 2025-08-14T21:24:12.7842870Z * [new branch] gh/malfet/458/orig -> origin/gh/malfet/458/orig 2025-08-14T21:24:12.7843013Z * [new branch] gh/malfet/459/base -> origin/gh/malfet/459/base 2025-08-14T21:24:12.7843153Z * [new branch] gh/malfet/459/head -> origin/gh/malfet/459/head 2025-08-14T21:24:12.7843289Z * [new branch] gh/malfet/459/orig -> origin/gh/malfet/459/orig 2025-08-14T21:24:12.7843431Z * [new branch] gh/malfet/460/base -> origin/gh/malfet/460/base 2025-08-14T21:24:12.7843567Z * [new branch] gh/malfet/460/head -> origin/gh/malfet/460/head 2025-08-14T21:24:12.7843720Z * [new branch] gh/malfet/460/orig -> origin/gh/malfet/460/orig 2025-08-14T21:24:12.7843852Z * [new branch] gh/malfet/461/base -> origin/gh/malfet/461/base 2025-08-14T21:24:12.7843997Z * [new branch] gh/malfet/461/head -> origin/gh/malfet/461/head 2025-08-14T21:24:12.7844134Z * [new branch] gh/malfet/461/orig -> origin/gh/malfet/461/orig 2025-08-14T21:24:12.7844273Z * [new branch] gh/malfet/462/base -> origin/gh/malfet/462/base 2025-08-14T21:24:12.7844420Z * [new branch] gh/malfet/462/head -> origin/gh/malfet/462/head 2025-08-14T21:24:12.7844561Z * [new branch] gh/malfet/462/orig -> origin/gh/malfet/462/orig 2025-08-14T21:24:12.7844704Z * [new branch] gh/malfet/463/base -> origin/gh/malfet/463/base 2025-08-14T21:24:12.7844844Z * [new branch] gh/malfet/463/head -> origin/gh/malfet/463/head 2025-08-14T21:24:12.7844987Z * [new branch] gh/malfet/463/orig -> origin/gh/malfet/463/orig 2025-08-14T21:24:12.7845160Z * [new branch] gh/malfet/464/base -> origin/gh/malfet/464/base 2025-08-14T21:24:12.7845303Z * [new branch] gh/malfet/464/head -> origin/gh/malfet/464/head 2025-08-14T21:24:12.7845447Z * [new branch] gh/malfet/464/orig -> origin/gh/malfet/464/orig 2025-08-14T21:24:12.7845764Z * [new branch] gh/malfet/465/base -> origin/gh/malfet/465/base 2025-08-14T21:24:12.7845915Z * [new branch] gh/malfet/465/head -> origin/gh/malfet/465/head 2025-08-14T21:24:12.7846063Z * [new branch] gh/malfet/465/orig -> origin/gh/malfet/465/orig 2025-08-14T21:24:12.7846205Z * [new branch] gh/malfet/466/base -> origin/gh/malfet/466/base 2025-08-14T21:24:12.7846344Z * [new branch] gh/malfet/466/head -> origin/gh/malfet/466/head 2025-08-14T21:24:12.7846501Z * [new branch] gh/malfet/466/orig -> origin/gh/malfet/466/orig 2025-08-14T21:24:12.7846636Z * [new branch] gh/malfet/467/base -> origin/gh/malfet/467/base 2025-08-14T21:24:12.7846780Z * [new branch] gh/malfet/467/head -> origin/gh/malfet/467/head 2025-08-14T21:24:12.7846925Z * [new branch] gh/malfet/467/orig -> origin/gh/malfet/467/orig 2025-08-14T21:24:12.7852181Z * [new branch] gh/malfet/468/base -> origin/gh/malfet/468/base 2025-08-14T21:24:12.7857334Z * [new branch] gh/malfet/468/head -> origin/gh/malfet/468/head 2025-08-14T21:24:12.7862294Z * [new branch] gh/malfet/468/orig -> origin/gh/malfet/468/orig 2025-08-14T21:24:12.7864552Z * [new branch] gh/malfet/469/base -> origin/gh/malfet/469/base 2025-08-14T21:24:12.7870487Z * [new branch] gh/malfet/469/head -> origin/gh/malfet/469/head 2025-08-14T21:24:12.7876289Z * [new branch] gh/malfet/469/orig -> origin/gh/malfet/469/orig 2025-08-14T21:24:12.7878016Z * [new branch] gh/malfet/470/base -> origin/gh/malfet/470/base 2025-08-14T21:24:12.7878238Z * [new branch] gh/malfet/470/head -> origin/gh/malfet/470/head 2025-08-14T21:24:12.7878455Z * [new branch] gh/malfet/470/orig -> origin/gh/malfet/470/orig 2025-08-14T21:24:12.7878579Z * [new branch] gh/malfet/471/base -> origin/gh/malfet/471/base 2025-08-14T21:24:12.7878825Z * [new branch] gh/malfet/471/head -> origin/gh/malfet/471/head 2025-08-14T21:24:12.7878963Z * [new branch] gh/malfet/471/orig -> origin/gh/malfet/471/orig 2025-08-14T21:24:12.7879171Z * [new branch] gh/malfet/472/base -> origin/gh/malfet/472/base 2025-08-14T21:24:12.7879306Z * [new branch] gh/malfet/472/head -> origin/gh/malfet/472/head 2025-08-14T21:24:12.7879457Z * [new branch] gh/malfet/472/orig -> origin/gh/malfet/472/orig 2025-08-14T21:24:12.7879660Z * [new branch] gh/malfet/473/base -> origin/gh/malfet/473/base 2025-08-14T21:24:12.7879818Z * [new branch] gh/malfet/473/head -> origin/gh/malfet/473/head 2025-08-14T21:24:12.7880330Z * [new branch] gh/malfet/473/orig -> origin/gh/malfet/473/orig 2025-08-14T21:24:12.7880504Z * [new branch] gh/malfet/474/base -> origin/gh/malfet/474/base 2025-08-14T21:24:12.7880640Z * [new branch] gh/malfet/474/head -> origin/gh/malfet/474/head 2025-08-14T21:24:12.7880765Z * [new branch] gh/malfet/474/orig -> origin/gh/malfet/474/orig 2025-08-14T21:24:12.7880891Z * [new branch] gh/malfet/475/base -> origin/gh/malfet/475/base 2025-08-14T21:24:12.7881025Z * [new branch] gh/malfet/475/head -> origin/gh/malfet/475/head 2025-08-14T21:24:12.7881346Z * [new branch] gh/malfet/475/orig -> origin/gh/malfet/475/orig 2025-08-14T21:24:12.7881481Z * [new branch] gh/malfet/476/base -> origin/gh/malfet/476/base 2025-08-14T21:24:12.7881613Z * [new branch] gh/malfet/476/head -> origin/gh/malfet/476/head 2025-08-14T21:24:12.7881737Z * [new branch] gh/malfet/476/orig -> origin/gh/malfet/476/orig 2025-08-14T21:24:12.7881870Z * [new branch] gh/malfet/477/base -> origin/gh/malfet/477/base 2025-08-14T21:24:12.7881996Z * [new branch] gh/malfet/477/head -> origin/gh/malfet/477/head 2025-08-14T21:24:12.7882127Z * [new branch] gh/malfet/477/orig -> origin/gh/malfet/477/orig 2025-08-14T21:24:12.7882263Z * [new branch] gh/malfet/478/base -> origin/gh/malfet/478/base 2025-08-14T21:24:12.7882403Z * [new branch] gh/malfet/478/head -> origin/gh/malfet/478/head 2025-08-14T21:24:12.7882543Z * [new branch] gh/malfet/478/orig -> origin/gh/malfet/478/orig 2025-08-14T21:24:12.7882677Z * [new branch] gh/malfet/479/base -> origin/gh/malfet/479/base 2025-08-14T21:24:12.7882817Z * [new branch] gh/malfet/479/head -> origin/gh/malfet/479/head 2025-08-14T21:24:12.7882954Z * [new branch] gh/malfet/479/orig -> origin/gh/malfet/479/orig 2025-08-14T21:24:12.7883091Z * [new branch] gh/malfet/480/base -> origin/gh/malfet/480/base 2025-08-14T21:24:12.7883227Z * [new branch] gh/malfet/480/head -> origin/gh/malfet/480/head 2025-08-14T21:24:12.7883362Z * [new branch] gh/malfet/480/orig -> origin/gh/malfet/480/orig 2025-08-14T21:24:12.7883496Z * [new branch] gh/malfet/481/base -> origin/gh/malfet/481/base 2025-08-14T21:24:12.7883707Z * [new branch] gh/malfet/481/head -> origin/gh/malfet/481/head 2025-08-14T21:24:12.7883854Z * [new branch] gh/malfet/481/orig -> origin/gh/malfet/481/orig 2025-08-14T21:24:12.7883996Z * [new branch] gh/malfet/482/base -> origin/gh/malfet/482/base 2025-08-14T21:24:12.7884134Z * [new branch] gh/malfet/482/head -> origin/gh/malfet/482/head 2025-08-14T21:24:12.7884275Z * [new branch] gh/malfet/482/orig -> origin/gh/malfet/482/orig 2025-08-14T21:24:12.7884431Z * [new branch] gh/malfet/483/base -> origin/gh/malfet/483/base 2025-08-14T21:24:12.7884574Z * [new branch] gh/malfet/483/head -> origin/gh/malfet/483/head 2025-08-14T21:24:12.7884721Z * [new branch] gh/malfet/483/orig -> origin/gh/malfet/483/orig 2025-08-14T21:24:12.7884864Z * [new branch] gh/malfet/484/base -> origin/gh/malfet/484/base 2025-08-14T21:24:12.7885008Z * [new branch] gh/malfet/484/head -> origin/gh/malfet/484/head 2025-08-14T21:24:12.7885149Z * [new branch] gh/malfet/484/orig -> origin/gh/malfet/484/orig 2025-08-14T21:24:12.7885290Z * [new branch] gh/malfet/485/base -> origin/gh/malfet/485/base 2025-08-14T21:24:12.7885428Z * [new branch] gh/malfet/485/head -> origin/gh/malfet/485/head 2025-08-14T21:24:12.7885860Z * [new branch] gh/malfet/485/orig -> origin/gh/malfet/485/orig 2025-08-14T21:24:12.7886022Z * [new branch] gh/malfet/486/base -> origin/gh/malfet/486/base 2025-08-14T21:24:12.7886165Z * [new branch] gh/malfet/486/head -> origin/gh/malfet/486/head 2025-08-14T21:24:12.7886312Z * [new branch] gh/malfet/486/orig -> origin/gh/malfet/486/orig 2025-08-14T21:24:12.7886443Z * [new branch] gh/malfet/487/base -> origin/gh/malfet/487/base 2025-08-14T21:24:12.7886591Z * [new branch] gh/malfet/487/head -> origin/gh/malfet/487/head 2025-08-14T21:24:12.7886766Z * [new branch] gh/malfet/487/orig -> origin/gh/malfet/487/orig 2025-08-14T21:24:12.7886910Z * [new branch] gh/malfet/488/base -> origin/gh/malfet/488/base 2025-08-14T21:24:12.7892278Z * [new branch] gh/malfet/488/head -> origin/gh/malfet/488/head 2025-08-14T21:24:12.7892448Z * [new branch] gh/malfet/488/orig -> origin/gh/malfet/488/orig 2025-08-14T21:24:12.7892603Z * [new branch] gh/malfet/489/base -> origin/gh/malfet/489/base 2025-08-14T21:24:12.7892744Z * [new branch] gh/malfet/489/head -> origin/gh/malfet/489/head 2025-08-14T21:24:12.7892885Z * [new branch] gh/malfet/489/orig -> origin/gh/malfet/489/orig 2025-08-14T21:24:12.7893025Z * [new branch] gh/malfet/490/base -> origin/gh/malfet/490/base 2025-08-14T21:24:12.7893183Z * [new branch] gh/malfet/490/head -> origin/gh/malfet/490/head 2025-08-14T21:24:12.7893329Z * [new branch] gh/malfet/490/orig -> origin/gh/malfet/490/orig 2025-08-14T21:24:12.7893465Z * [new branch] gh/malfet/64/base -> origin/gh/malfet/64/base 2025-08-14T21:24:12.7899644Z * [new branch] gh/malfet/64/head -> origin/gh/malfet/64/head 2025-08-14T21:24:12.7899868Z * [new branch] gh/manuelcandales/10/base -> origin/gh/manuelcandales/10/base 2025-08-14T21:24:12.7900040Z * [new branch] gh/manuelcandales/10/head -> origin/gh/manuelcandales/10/head 2025-08-14T21:24:12.7900215Z * [new branch] gh/manuelcandales/10/orig -> origin/gh/manuelcandales/10/orig 2025-08-14T21:24:12.7900385Z * [new branch] gh/manuelcandales/9/base -> origin/gh/manuelcandales/9/base 2025-08-14T21:24:12.7900549Z * [new branch] gh/manuelcandales/9/head -> origin/gh/manuelcandales/9/head 2025-08-14T21:24:12.7900853Z * [new branch] gh/manuelcandales/9/orig -> origin/gh/manuelcandales/9/orig 2025-08-14T21:24:12.7901015Z * [new branch] gh/markkm/1/base -> origin/gh/markkm/1/base 2025-08-14T21:24:12.7902204Z * [new branch] gh/masnesral/204/base -> origin/gh/masnesral/204/base 2025-08-14T21:24:12.7907330Z * [new branch] gh/masnesral/204/head -> origin/gh/masnesral/204/head 2025-08-14T21:24:12.7912769Z * [new branch] gh/masnesral/204/orig -> origin/gh/masnesral/204/orig 2025-08-14T21:24:12.7919850Z * [new branch] gh/masnesral/223/base -> origin/gh/masnesral/223/base 2025-08-14T21:24:12.7920234Z * [new branch] gh/masnesral/223/head -> origin/gh/masnesral/223/head 2025-08-14T21:24:12.7920402Z * [new branch] gh/masnesral/223/orig -> origin/gh/masnesral/223/orig 2025-08-14T21:24:12.7920670Z * [new branch] gh/masnesral/224/base -> origin/gh/masnesral/224/base 2025-08-14T21:24:12.7920885Z * [new branch] gh/masnesral/224/head -> origin/gh/masnesral/224/head 2025-08-14T21:24:12.7921036Z * [new branch] gh/masnesral/224/orig -> origin/gh/masnesral/224/orig 2025-08-14T21:24:12.7921329Z * [new branch] gh/masnesral/225/base -> origin/gh/masnesral/225/base 2025-08-14T21:24:12.7921820Z * [new branch] gh/masnesral/225/head -> origin/gh/masnesral/225/head 2025-08-14T21:24:12.7922007Z * [new branch] gh/masnesral/225/orig -> origin/gh/masnesral/225/orig 2025-08-14T21:24:12.7922168Z * [new branch] gh/masnesral/226/base -> origin/gh/masnesral/226/base 2025-08-14T21:24:12.7922358Z * [new branch] gh/masnesral/226/head -> origin/gh/masnesral/226/head 2025-08-14T21:24:12.7922514Z * [new branch] gh/masnesral/226/orig -> origin/gh/masnesral/226/orig 2025-08-14T21:24:12.7922685Z * [new branch] gh/masnesral/227/base -> origin/gh/masnesral/227/base 2025-08-14T21:24:12.7922995Z * [new branch] gh/masnesral/227/head -> origin/gh/masnesral/227/head 2025-08-14T21:24:12.7923148Z * [new branch] gh/masnesral/227/orig -> origin/gh/masnesral/227/orig 2025-08-14T21:24:12.7923303Z * [new branch] gh/masnesral/228/base -> origin/gh/masnesral/228/base 2025-08-14T21:24:12.7923461Z * [new branch] gh/masnesral/228/head -> origin/gh/masnesral/228/head 2025-08-14T21:24:12.7923614Z * [new branch] gh/masnesral/228/orig -> origin/gh/masnesral/228/orig 2025-08-14T21:24:12.7923770Z * [new branch] gh/masnesral/229/base -> origin/gh/masnesral/229/base 2025-08-14T21:24:12.7923919Z * [new branch] gh/masnesral/229/head -> origin/gh/masnesral/229/head 2025-08-14T21:24:12.7924068Z * [new branch] gh/masnesral/229/orig -> origin/gh/masnesral/229/orig 2025-08-14T21:24:12.7924230Z * [new branch] gh/masnesral/230/base -> origin/gh/masnesral/230/base 2025-08-14T21:24:12.7924384Z * [new branch] gh/masnesral/230/head -> origin/gh/masnesral/230/head 2025-08-14T21:24:12.7924540Z * [new branch] gh/masnesral/230/orig -> origin/gh/masnesral/230/orig 2025-08-14T21:24:12.7924700Z * [new branch] gh/masnesral/231/base -> origin/gh/masnesral/231/base 2025-08-14T21:24:12.7924848Z * [new branch] gh/masnesral/231/head -> origin/gh/masnesral/231/head 2025-08-14T21:24:12.7925011Z * [new branch] gh/masnesral/231/orig -> origin/gh/masnesral/231/orig 2025-08-14T21:24:12.7925162Z * [new branch] gh/masnesral/232/base -> origin/gh/masnesral/232/base 2025-08-14T21:24:12.7925323Z * [new branch] gh/masnesral/232/head -> origin/gh/masnesral/232/head 2025-08-14T21:24:12.7926172Z * [new branch] gh/masnesral/232/orig -> origin/gh/masnesral/232/orig 2025-08-14T21:24:12.7930615Z * [new branch] gh/masnesral/233/base -> origin/gh/masnesral/233/base 2025-08-14T21:24:12.7930776Z * [new branch] gh/masnesral/233/head -> origin/gh/masnesral/233/head 2025-08-14T21:24:12.7931154Z * [new branch] gh/masnesral/233/orig -> origin/gh/masnesral/233/orig 2025-08-14T21:24:12.7931320Z * [new branch] gh/masnesral/234/base -> origin/gh/masnesral/234/base 2025-08-14T21:24:12.7931482Z * [new branch] gh/masnesral/234/head -> origin/gh/masnesral/234/head 2025-08-14T21:24:12.7931640Z * [new branch] gh/masnesral/234/orig -> origin/gh/masnesral/234/orig 2025-08-14T21:24:12.7935062Z * [new branch] gh/masnesral/235/base -> origin/gh/masnesral/235/base 2025-08-14T21:24:12.7935310Z * [new branch] gh/masnesral/235/head -> origin/gh/masnesral/235/head 2025-08-14T21:24:12.7935999Z * [new branch] gh/masnesral/235/orig -> origin/gh/masnesral/235/orig 2025-08-14T21:24:12.7941075Z * [new branch] gh/masnesral/236/base -> origin/gh/masnesral/236/base 2025-08-14T21:24:12.7941343Z * [new branch] gh/masnesral/236/head -> origin/gh/masnesral/236/head 2025-08-14T21:24:12.7941657Z * [new branch] gh/masnesral/236/orig -> origin/gh/masnesral/236/orig 2025-08-14T21:24:12.7941825Z * [new branch] gh/masnesral/34/base -> origin/gh/masnesral/34/base 2025-08-14T21:24:12.7942066Z * [new branch] gh/mhorowitz/0/base -> origin/gh/mhorowitz/0/base 2025-08-14T21:24:12.7942235Z * [new branch] gh/mhorowitz/0/head -> origin/gh/mhorowitz/0/head 2025-08-14T21:24:12.7942372Z * [new branch] gh/mhorowitz/1/base -> origin/gh/mhorowitz/1/base 2025-08-14T21:24:12.7948021Z * [new branch] gh/mhorowitz/1/head -> origin/gh/mhorowitz/1/head 2025-08-14T21:24:12.7948237Z * [new branch] gh/mhorowitz/2/base -> origin/gh/mhorowitz/2/base 2025-08-14T21:24:12.7952945Z * [new branch] gh/mhorowitz/2/head -> origin/gh/mhorowitz/2/head 2025-08-14T21:24:12.7958191Z * [new branch] gh/mhorowitz/3/base -> origin/gh/mhorowitz/3/base 2025-08-14T21:24:12.7960244Z * [new branch] gh/mhorowitz/3/head -> origin/gh/mhorowitz/3/head 2025-08-14T21:24:12.7960523Z * [new branch] gh/mhorowitz/4/base -> origin/gh/mhorowitz/4/base 2025-08-14T21:24:12.7960712Z * [new branch] gh/mhorowitz/4/head -> origin/gh/mhorowitz/4/head 2025-08-14T21:24:12.7960862Z * [new branch] gh/mhorowitz/5/base -> origin/gh/mhorowitz/5/base 2025-08-14T21:24:12.7961021Z * [new branch] gh/mhorowitz/5/head -> origin/gh/mhorowitz/5/head 2025-08-14T21:24:12.7961206Z * [new branch] gh/mhorowitz/6/base -> origin/gh/mhorowitz/6/base 2025-08-14T21:24:12.7961370Z * [new branch] gh/mhorowitz/6/head -> origin/gh/mhorowitz/6/head 2025-08-14T21:24:12.7961569Z * [new branch] gh/mikaylagawarecki/234/base -> origin/gh/mikaylagawarecki/234/base 2025-08-14T21:24:12.7961746Z * [new branch] gh/mikaylagawarecki/234/head -> origin/gh/mikaylagawarecki/234/head 2025-08-14T21:24:12.7961917Z * [new branch] gh/mikaylagawarecki/235/base -> origin/gh/mikaylagawarecki/235/base 2025-08-14T21:24:12.7962086Z * [new branch] gh/mikaylagawarecki/235/head -> origin/gh/mikaylagawarecki/235/head 2025-08-14T21:24:12.7962263Z * [new branch] gh/mikaylagawarecki/236/base -> origin/gh/mikaylagawarecki/236/base 2025-08-14T21:24:12.7962431Z * [new branch] gh/mikaylagawarecki/236/head -> origin/gh/mikaylagawarecki/236/head 2025-08-14T21:24:12.7962597Z * [new branch] gh/mikaylagawarecki/237/base -> origin/gh/mikaylagawarecki/237/base 2025-08-14T21:24:12.7963222Z * [new branch] gh/mikaylagawarecki/237/head -> origin/gh/mikaylagawarecki/237/head 2025-08-14T21:24:12.7963402Z * [new branch] gh/mikaylagawarecki/238/base -> origin/gh/mikaylagawarecki/238/base 2025-08-14T21:24:12.7963574Z * [new branch] gh/mikaylagawarecki/238/head -> origin/gh/mikaylagawarecki/238/head 2025-08-14T21:24:12.7963737Z * [new branch] gh/mikaylagawarecki/313/base -> origin/gh/mikaylagawarecki/313/base 2025-08-14T21:24:12.7963906Z * [new branch] gh/mikaylagawarecki/313/head -> origin/gh/mikaylagawarecki/313/head 2025-08-14T21:24:12.7964083Z * [new branch] gh/mikaylagawarecki/313/orig -> origin/gh/mikaylagawarecki/313/orig 2025-08-14T21:24:12.7964243Z * [new branch] gh/mikaylagawarecki/317/base -> origin/gh/mikaylagawarecki/317/base 2025-08-14T21:24:12.7964415Z * [new branch] gh/mikaylagawarecki/317/head -> origin/gh/mikaylagawarecki/317/head 2025-08-14T21:24:12.7964580Z * [new branch] gh/mikaylagawarecki/317/orig -> origin/gh/mikaylagawarecki/317/orig 2025-08-14T21:24:12.7964746Z * [new branch] gh/mikaylagawarecki/318/base -> origin/gh/mikaylagawarecki/318/base 2025-08-14T21:24:12.7964918Z * [new branch] gh/mikaylagawarecki/318/head -> origin/gh/mikaylagawarecki/318/head 2025-08-14T21:24:12.7965092Z * [new branch] gh/mikaylagawarecki/318/orig -> origin/gh/mikaylagawarecki/318/orig 2025-08-14T21:24:12.7965273Z * [new branch] gh/mikaylagawarecki/319/base -> origin/gh/mikaylagawarecki/319/base 2025-08-14T21:24:12.7965445Z * [new branch] gh/mikaylagawarecki/319/head -> origin/gh/mikaylagawarecki/319/head 2025-08-14T21:24:12.7965841Z * [new branch] gh/mikaylagawarecki/319/orig -> origin/gh/mikaylagawarecki/319/orig 2025-08-14T21:24:12.7966025Z * [new branch] gh/mikaylagawarecki/320/base -> origin/gh/mikaylagawarecki/320/base 2025-08-14T21:24:12.7966210Z * [new branch] gh/mikaylagawarecki/320/head -> origin/gh/mikaylagawarecki/320/head 2025-08-14T21:24:12.7966454Z * [new branch] gh/mikaylagawarecki/320/orig -> origin/gh/mikaylagawarecki/320/orig 2025-08-14T21:24:12.7966636Z * [new branch] gh/mikaylagawarecki/321/base -> origin/gh/mikaylagawarecki/321/base 2025-08-14T21:24:12.7966820Z * [new branch] gh/mikaylagawarecki/321/head -> origin/gh/mikaylagawarecki/321/head 2025-08-14T21:24:12.7967005Z * [new branch] gh/mikaylagawarecki/321/orig -> origin/gh/mikaylagawarecki/321/orig 2025-08-14T21:24:12.7970468Z * [new branch] gh/mikaylagawarecki/322/base -> origin/gh/mikaylagawarecki/322/base 2025-08-14T21:24:12.7971012Z * [new branch] gh/mikaylagawarecki/322/head -> origin/gh/mikaylagawarecki/322/head 2025-08-14T21:24:12.7971211Z * [new branch] gh/mikaylagawarecki/322/orig -> origin/gh/mikaylagawarecki/322/orig 2025-08-14T21:24:12.7971398Z * [new branch] gh/mikaylagawarecki/323/base -> origin/gh/mikaylagawarecki/323/base 2025-08-14T21:24:12.7971571Z * [new branch] gh/mikaylagawarecki/323/head -> origin/gh/mikaylagawarecki/323/head 2025-08-14T21:24:12.7971738Z * [new branch] gh/mikaylagawarecki/323/orig -> origin/gh/mikaylagawarecki/323/orig 2025-08-14T21:24:12.7971898Z * [new branch] gh/mikaylagawarecki/324/base -> origin/gh/mikaylagawarecki/324/base 2025-08-14T21:24:12.7974863Z * [new branch] gh/mikaylagawarecki/324/head -> origin/gh/mikaylagawarecki/324/head 2025-08-14T21:24:12.7975056Z * [new branch] gh/mikaylagawarecki/324/orig -> origin/gh/mikaylagawarecki/324/orig 2025-08-14T21:24:12.7975233Z * [new branch] gh/mikaylagawarecki/325/base -> origin/gh/mikaylagawarecki/325/base 2025-08-14T21:24:12.7975410Z * [new branch] gh/mikaylagawarecki/325/head -> origin/gh/mikaylagawarecki/325/head 2025-08-14T21:24:12.7975740Z * [new branch] gh/mikaylagawarecki/325/orig -> origin/gh/mikaylagawarecki/325/orig 2025-08-14T21:24:12.7979849Z * [new branch] gh/mikaylagawarecki/326/base -> origin/gh/mikaylagawarecki/326/base 2025-08-14T21:24:12.7980021Z * [new branch] gh/mikaylagawarecki/326/head -> origin/gh/mikaylagawarecki/326/head 2025-08-14T21:24:12.7980233Z * [new branch] gh/mikaylagawarecki/326/orig -> origin/gh/mikaylagawarecki/326/orig 2025-08-14T21:24:12.7980438Z * [new branch] gh/mikaylagawarecki/327/base -> origin/gh/mikaylagawarecki/327/base 2025-08-14T21:24:12.7980614Z * [new branch] gh/mikaylagawarecki/327/head -> origin/gh/mikaylagawarecki/327/head 2025-08-14T21:24:12.7980831Z * [new branch] gh/mikaylagawarecki/327/orig -> origin/gh/mikaylagawarecki/327/orig 2025-08-14T21:24:12.7987202Z * [new branch] gh/mikaylagawarecki/328/base -> origin/gh/mikaylagawarecki/328/base 2025-08-14T21:24:12.7989414Z * [new branch] gh/mikaylagawarecki/328/head -> origin/gh/mikaylagawarecki/328/head 2025-08-14T21:24:12.7989742Z * [new branch] gh/mikaylagawarecki/328/orig -> origin/gh/mikaylagawarecki/328/orig 2025-08-14T21:24:12.7995258Z * [new branch] gh/mikaylagawarecki/329/base -> origin/gh/mikaylagawarecki/329/base 2025-08-14T21:24:12.7997375Z * [new branch] gh/mikaylagawarecki/329/head -> origin/gh/mikaylagawarecki/329/head 2025-08-14T21:24:12.7997567Z * [new branch] gh/mikaylagawarecki/329/orig -> origin/gh/mikaylagawarecki/329/orig 2025-08-14T21:24:12.7998051Z * [new branch] gh/mikaylagawarecki/330/base -> origin/gh/mikaylagawarecki/330/base 2025-08-14T21:24:12.7998241Z * [new branch] gh/mikaylagawarecki/330/head -> origin/gh/mikaylagawarecki/330/head 2025-08-14T21:24:12.7998431Z * [new branch] gh/mikaylagawarecki/330/orig -> origin/gh/mikaylagawarecki/330/orig 2025-08-14T21:24:12.7998611Z * [new branch] gh/mikaylagawarecki/331/base -> origin/gh/mikaylagawarecki/331/base 2025-08-14T21:24:12.7998799Z * [new branch] gh/mikaylagawarecki/331/head -> origin/gh/mikaylagawarecki/331/head 2025-08-14T21:24:12.7999130Z * [new branch] gh/mikaylagawarecki/331/orig -> origin/gh/mikaylagawarecki/331/orig 2025-08-14T21:24:12.7999309Z * [new branch] gh/mikaylagawarecki/332/base -> origin/gh/mikaylagawarecki/332/base 2025-08-14T21:24:12.7999496Z * [new branch] gh/mikaylagawarecki/332/head -> origin/gh/mikaylagawarecki/332/head 2025-08-14T21:24:12.7999672Z * [new branch] gh/mikaylagawarecki/332/orig -> origin/gh/mikaylagawarecki/332/orig 2025-08-14T21:24:12.7999847Z * [new branch] gh/mikaylagawarecki/333/base -> origin/gh/mikaylagawarecki/333/base 2025-08-14T21:24:12.8000027Z * [new branch] gh/mikaylagawarecki/333/head -> origin/gh/mikaylagawarecki/333/head 2025-08-14T21:24:12.8000207Z * [new branch] gh/mikaylagawarecki/333/orig -> origin/gh/mikaylagawarecki/333/orig 2025-08-14T21:24:12.8000405Z * [new branch] gh/mikaylagawarecki/334/base -> origin/gh/mikaylagawarecki/334/base 2025-08-14T21:24:12.8000592Z * [new branch] gh/mikaylagawarecki/334/head -> origin/gh/mikaylagawarecki/334/head 2025-08-14T21:24:12.8000770Z * [new branch] gh/mikaylagawarecki/334/orig -> origin/gh/mikaylagawarecki/334/orig 2025-08-14T21:24:12.8000935Z * [new branch] gh/mlazos/1/base -> origin/gh/mlazos/1/base 2025-08-14T21:24:12.8001086Z * [new branch] gh/mlazos/1/head -> origin/gh/mlazos/1/head 2025-08-14T21:24:12.8001226Z * [new branch] gh/mlazos/1/orig -> origin/gh/mlazos/1/orig 2025-08-14T21:24:12.8001379Z * [new branch] gh/mlazos/10/base -> origin/gh/mlazos/10/base 2025-08-14T21:24:12.8001519Z * [new branch] gh/mlazos/10/head -> origin/gh/mlazos/10/head 2025-08-14T21:24:12.8001713Z * [new branch] gh/mlazos/10/orig -> origin/gh/mlazos/10/orig 2025-08-14T21:24:12.8002103Z * [new branch] gh/mlazos/11/base -> origin/gh/mlazos/11/base 2025-08-14T21:24:12.8003843Z * [new branch] gh/mlazos/11/head -> origin/gh/mlazos/11/head 2025-08-14T21:24:12.8004297Z * [new branch] gh/mlazos/11/orig -> origin/gh/mlazos/11/orig 2025-08-14T21:24:12.8004647Z * [new branch] gh/mlazos/12/base -> origin/gh/mlazos/12/base 2025-08-14T21:24:12.8006108Z * [new branch] gh/mlazos/12/head -> origin/gh/mlazos/12/head 2025-08-14T21:24:12.8006449Z * [new branch] gh/mlazos/12/orig -> origin/gh/mlazos/12/orig 2025-08-14T21:24:12.8011694Z * [new branch] gh/mlazos/13/base -> origin/gh/mlazos/13/base 2025-08-14T21:24:12.8017084Z * [new branch] gh/mlazos/13/head -> origin/gh/mlazos/13/head 2025-08-14T21:24:12.8017262Z * [new branch] gh/mlazos/13/orig -> origin/gh/mlazos/13/orig 2025-08-14T21:24:12.8017419Z * [new branch] gh/mlazos/2/base -> origin/gh/mlazos/2/base 2025-08-14T21:24:12.8017562Z * [new branch] gh/mlazos/2/head -> origin/gh/mlazos/2/head 2025-08-14T21:24:12.8017681Z * [new branch] gh/mlazos/2/orig -> origin/gh/mlazos/2/orig 2025-08-14T21:24:12.8017811Z * [new branch] gh/mlazos/3/base -> origin/gh/mlazos/3/base 2025-08-14T21:24:12.8017932Z * [new branch] gh/mlazos/3/head -> origin/gh/mlazos/3/head 2025-08-14T21:24:12.8018059Z * [new branch] gh/mlazos/3/orig -> origin/gh/mlazos/3/orig 2025-08-14T21:24:12.8018177Z * [new branch] gh/mlazos/4/base -> origin/gh/mlazos/4/base 2025-08-14T21:24:12.8018293Z * [new branch] gh/mlazos/4/head -> origin/gh/mlazos/4/head 2025-08-14T21:24:12.8018420Z * [new branch] gh/mlazos/4/orig -> origin/gh/mlazos/4/orig 2025-08-14T21:24:12.8018701Z * [new branch] gh/mlazos/5/base -> origin/gh/mlazos/5/base 2025-08-14T21:24:12.8018834Z * [new branch] gh/mlazos/5/head -> origin/gh/mlazos/5/head 2025-08-14T21:24:12.8018955Z * [new branch] gh/mlazos/5/orig -> origin/gh/mlazos/5/orig 2025-08-14T21:24:12.8019573Z * [new branch] gh/mlazos/6/base -> origin/gh/mlazos/6/base 2025-08-14T21:24:12.8019909Z * [new branch] gh/mlazos/6/head -> origin/gh/mlazos/6/head 2025-08-14T21:24:12.8024254Z * [new branch] gh/mlazos/6/orig -> origin/gh/mlazos/6/orig 2025-08-14T21:24:12.8024418Z * [new branch] gh/mlazos/7/base -> origin/gh/mlazos/7/base 2025-08-14T21:24:12.8024548Z * [new branch] gh/mlazos/7/head -> origin/gh/mlazos/7/head 2025-08-14T21:24:12.8024679Z * [new branch] gh/mlazos/7/orig -> origin/gh/mlazos/7/orig 2025-08-14T21:24:12.8024818Z * [new branch] gh/mlazos/8/base -> origin/gh/mlazos/8/base 2025-08-14T21:24:12.8029979Z * [new branch] gh/mlazos/8/head -> origin/gh/mlazos/8/head 2025-08-14T21:24:12.8030140Z * [new branch] gh/mlazos/8/orig -> origin/gh/mlazos/8/orig 2025-08-14T21:24:12.8030270Z * [new branch] gh/mlazos/9/base -> origin/gh/mlazos/9/base 2025-08-14T21:24:12.8030390Z * [new branch] gh/mlazos/9/head -> origin/gh/mlazos/9/head 2025-08-14T21:24:12.8030510Z * [new branch] gh/mlazos/9/orig -> origin/gh/mlazos/9/orig 2025-08-14T21:24:12.8030666Z * [new branch] gh/mrmiywj/1/base -> origin/gh/mrmiywj/1/base 2025-08-14T21:24:12.8030793Z * [new branch] gh/mrmiywj/1/head -> origin/gh/mrmiywj/1/head 2025-08-14T21:24:12.8030947Z * [new branch] gh/muchulee8/62/base -> origin/gh/muchulee8/62/base 2025-08-14T21:24:12.8031230Z * [new branch] gh/muchulee8/62/head -> origin/gh/muchulee8/62/head 2025-08-14T21:24:12.8031376Z * [new branch] gh/muchulee8/62/orig -> origin/gh/muchulee8/62/orig 2025-08-14T21:24:12.8031515Z * [new branch] gh/muchulee8/63/base -> origin/gh/muchulee8/63/base 2025-08-14T21:24:12.8031652Z * [new branch] gh/muchulee8/63/head -> origin/gh/muchulee8/63/head 2025-08-14T21:24:12.8031793Z * [new branch] gh/muchulee8/63/orig -> origin/gh/muchulee8/63/orig 2025-08-14T21:24:12.8035465Z * [new branch] gh/muchulee8/64/base -> origin/gh/muchulee8/64/base 2025-08-14T21:24:12.8035642Z * [new branch] gh/muchulee8/64/head -> origin/gh/muchulee8/64/head 2025-08-14T21:24:12.8035786Z * [new branch] gh/muchulee8/64/orig -> origin/gh/muchulee8/64/orig 2025-08-14T21:24:12.8035934Z * [new branch] gh/muchulee8/65/base -> origin/gh/muchulee8/65/base 2025-08-14T21:24:12.8041471Z * [new branch] gh/muchulee8/65/head -> origin/gh/muchulee8/65/head 2025-08-14T21:24:12.8041805Z * [new branch] gh/muchulee8/65/orig -> origin/gh/muchulee8/65/orig 2025-08-14T21:24:12.8041975Z * [new branch] gh/oulgen/35/base -> origin/gh/oulgen/35/base 2025-08-14T21:24:12.8042120Z * [new branch] gh/oulgen/35/head -> origin/gh/oulgen/35/head 2025-08-14T21:24:12.8042381Z * [new branch] gh/oulgen/35/orig -> origin/gh/oulgen/35/orig 2025-08-14T21:24:12.8042565Z * [new branch] gh/oulgen/44/base -> origin/gh/oulgen/44/base 2025-08-14T21:24:12.8042707Z * [new branch] gh/oulgen/44/head -> origin/gh/oulgen/44/head 2025-08-14T21:24:12.8042850Z * [new branch] gh/oulgen/44/orig -> origin/gh/oulgen/44/orig 2025-08-14T21:24:12.8042995Z * [new branch] gh/oulgen/45/base -> origin/gh/oulgen/45/base 2025-08-14T21:24:12.8043707Z * [new branch] gh/oulgen/45/head -> origin/gh/oulgen/45/head 2025-08-14T21:24:12.8044418Z * [new branch] gh/oulgen/45/orig -> origin/gh/oulgen/45/orig 2025-08-14T21:24:12.8045517Z * [new branch] gh/oulgen/46/base -> origin/gh/oulgen/46/base 2025-08-14T21:24:12.8045958Z * [new branch] gh/oulgen/46/head -> origin/gh/oulgen/46/head 2025-08-14T21:24:12.8051647Z * [new branch] gh/oulgen/46/orig -> origin/gh/oulgen/46/orig 2025-08-14T21:24:12.8056201Z * [new branch] gh/oulgen/47/base -> origin/gh/oulgen/47/base 2025-08-14T21:24:12.8061413Z * [new branch] gh/oulgen/47/head -> origin/gh/oulgen/47/head 2025-08-14T21:24:12.8063529Z * [new branch] gh/oulgen/47/orig -> origin/gh/oulgen/47/orig 2025-08-14T21:24:12.8063835Z * [new branch] gh/pearu/108/base -> origin/gh/pearu/108/base 2025-08-14T21:24:12.8069864Z * [new branch] gh/pearu/108/head -> origin/gh/pearu/108/head 2025-08-14T21:24:12.8075254Z * [new branch] gh/pearu/108/orig -> origin/gh/pearu/108/orig 2025-08-14T21:24:12.8079258Z * [new branch] gh/pearu/56/base -> origin/gh/pearu/56/base 2025-08-14T21:24:12.8080950Z * [new branch] gh/pearu/56/head -> origin/gh/pearu/56/head 2025-08-14T21:24:12.8081093Z * [new branch] gh/pearu/56/orig -> origin/gh/pearu/56/orig 2025-08-14T21:24:12.8081321Z * [new branch] gh/pearu/97/base -> origin/gh/pearu/97/base 2025-08-14T21:24:12.8081462Z * [new branch] gh/pearu/97/head -> origin/gh/pearu/97/head 2025-08-14T21:24:12.8081580Z * [new branch] gh/pearu/97/orig -> origin/gh/pearu/97/orig 2025-08-14T21:24:12.8081925Z * [new branch] gh/qqaatw/29/base -> origin/gh/qqaatw/29/base 2025-08-14T21:24:12.8082085Z * [new branch] gh/qqaatw/29/head -> origin/gh/qqaatw/29/head 2025-08-14T21:24:12.8082209Z * [new branch] gh/qqaatw/29/orig -> origin/gh/qqaatw/29/orig 2025-08-14T21:24:12.8082413Z * [new branch] gh/raymo/cleanup-dynamo-logging -> origin/gh/raymo/cleanup-dynamo-logging 2025-08-14T21:24:12.8082582Z * [new branch] gh/raymo/refresh-script -> origin/gh/raymo/refresh-script 2025-08-14T21:24:12.8082713Z * [new branch] gh/rec/141/base -> origin/gh/rec/141/base 2025-08-14T21:24:12.8082845Z * [new branch] gh/rec/141/head -> origin/gh/rec/141/head 2025-08-14T21:24:12.8082962Z * [new branch] gh/rec/153/base -> origin/gh/rec/153/base 2025-08-14T21:24:12.8083076Z * [new branch] gh/rec/153/head -> origin/gh/rec/153/head 2025-08-14T21:24:12.8083203Z * [new branch] gh/rec/153/orig -> origin/gh/rec/153/orig 2025-08-14T21:24:12.8083321Z * [new branch] gh/rec/154/base -> origin/gh/rec/154/base 2025-08-14T21:24:12.8083444Z * [new branch] gh/rec/154/head -> origin/gh/rec/154/head 2025-08-14T21:24:12.8083559Z * [new branch] gh/rec/154/orig -> origin/gh/rec/154/orig 2025-08-14T21:24:12.8083695Z * [new branch] gh/rec/156/base -> origin/gh/rec/156/base 2025-08-14T21:24:12.8083818Z * [new branch] gh/rec/156/head -> origin/gh/rec/156/head 2025-08-14T21:24:12.8083932Z * [new branch] gh/rec/156/orig -> origin/gh/rec/156/orig 2025-08-14T21:24:12.8084053Z * [new branch] gh/rec/158/base -> origin/gh/rec/158/base 2025-08-14T21:24:12.8084169Z * [new branch] gh/rec/158/head -> origin/gh/rec/158/head 2025-08-14T21:24:12.8084286Z * [new branch] gh/rec/158/orig -> origin/gh/rec/158/orig 2025-08-14T21:24:12.8084453Z * [new branch] gh/rec/159/base -> origin/gh/rec/159/base 2025-08-14T21:24:12.8084566Z * [new branch] gh/rec/159/head -> origin/gh/rec/159/head 2025-08-14T21:24:12.8084685Z * [new branch] gh/rec/160/base -> origin/gh/rec/160/base 2025-08-14T21:24:12.8084798Z * [new branch] gh/rec/160/head -> origin/gh/rec/160/head 2025-08-14T21:24:12.8084912Z * [new branch] gh/rec/160/orig -> origin/gh/rec/160/orig 2025-08-14T21:24:12.8085034Z * [new branch] gh/rec/161/base -> origin/gh/rec/161/base 2025-08-14T21:24:12.8085155Z * [new branch] gh/rec/161/head -> origin/gh/rec/161/head 2025-08-14T21:24:12.8085274Z * [new branch] gh/rec/161/orig -> origin/gh/rec/161/orig 2025-08-14T21:24:12.8085404Z * [new branch] gh/rec/162/base -> origin/gh/rec/162/base 2025-08-14T21:24:12.8085680Z * [new branch] gh/rec/162/head -> origin/gh/rec/162/head 2025-08-14T21:24:12.8085828Z * [new branch] gh/rec/162/orig -> origin/gh/rec/162/orig 2025-08-14T21:24:12.8085951Z * [new branch] gh/rec/163/base -> origin/gh/rec/163/base 2025-08-14T21:24:12.8086077Z * [new branch] gh/rec/163/head -> origin/gh/rec/163/head 2025-08-14T21:24:12.8086211Z * [new branch] gh/rec/163/orig -> origin/gh/rec/163/orig 2025-08-14T21:24:12.8086338Z * [new branch] gh/rec/164/base -> origin/gh/rec/164/base 2025-08-14T21:24:12.8086471Z * [new branch] gh/rec/164/head -> origin/gh/rec/164/head 2025-08-14T21:24:12.8086598Z * [new branch] gh/rec/164/orig -> origin/gh/rec/164/orig 2025-08-14T21:24:12.8086822Z * [new branch] gh/robert-hardwick/1/base -> origin/gh/robert-hardwick/1/base 2025-08-14T21:24:12.8087009Z * [new branch] gh/robert-hardwick/1/head -> origin/gh/robert-hardwick/1/head 2025-08-14T21:24:12.8087164Z * [new branch] gh/robert-hardwick/1/orig -> origin/gh/robert-hardwick/1/orig 2025-08-14T21:24:12.8087333Z * [new branch] gh/robert-hardwick/2/base -> origin/gh/robert-hardwick/2/base 2025-08-14T21:24:12.8087484Z * [new branch] gh/robert-hardwick/2/head -> origin/gh/robert-hardwick/2/head 2025-08-14T21:24:12.8087638Z * [new branch] gh/robert-hardwick/2/orig -> origin/gh/robert-hardwick/2/orig 2025-08-14T21:24:12.8087787Z * [new branch] gh/robert-hardwick/3/base -> origin/gh/robert-hardwick/3/base 2025-08-14T21:24:12.8089790Z * [new branch] gh/robert-hardwick/3/head -> origin/gh/robert-hardwick/3/head 2025-08-14T21:24:12.8089994Z * [new branch] gh/robert-hardwick/3/orig -> origin/gh/robert-hardwick/3/orig 2025-08-14T21:24:12.8090179Z * [new branch] gh/robert-hardwick/4/base -> origin/gh/robert-hardwick/4/base 2025-08-14T21:24:12.8090354Z * [new branch] gh/robert-hardwick/4/head -> origin/gh/robert-hardwick/4/head 2025-08-14T21:24:12.8090504Z * [new branch] gh/robert-hardwick/4/orig -> origin/gh/robert-hardwick/4/orig 2025-08-14T21:24:12.8095792Z * [new branch] gh/rtimpe/1/base -> origin/gh/rtimpe/1/base 2025-08-14T21:24:12.8095973Z * [new branch] gh/rtimpe/1/head -> origin/gh/rtimpe/1/head 2025-08-14T21:24:12.8096112Z * [new branch] gh/rtimpe/10/base -> origin/gh/rtimpe/10/base 2025-08-14T21:24:12.8096245Z * [new branch] gh/rtimpe/10/head -> origin/gh/rtimpe/10/head 2025-08-14T21:24:12.8096379Z * [new branch] gh/rtimpe/10/orig -> origin/gh/rtimpe/10/orig 2025-08-14T21:24:12.8098218Z * [new branch] gh/rtimpe/11/base -> origin/gh/rtimpe/11/base 2025-08-14T21:24:12.8098579Z * [new branch] gh/rtimpe/11/head -> origin/gh/rtimpe/11/head 2025-08-14T21:24:12.8099032Z * [new branch] gh/rtimpe/11/orig -> origin/gh/rtimpe/11/orig 2025-08-14T21:24:12.8099180Z * [new branch] gh/rtimpe/12/base -> origin/gh/rtimpe/12/base 2025-08-14T21:24:12.8099405Z * [new branch] gh/rtimpe/12/head -> origin/gh/rtimpe/12/head 2025-08-14T21:24:12.8099551Z * [new branch] gh/rtimpe/12/orig -> origin/gh/rtimpe/12/orig 2025-08-14T21:24:12.8104884Z * [new branch] gh/rtimpe/2/base -> origin/gh/rtimpe/2/base 2025-08-14T21:24:12.8105177Z * [new branch] gh/rtimpe/2/head -> origin/gh/rtimpe/2/head 2025-08-14T21:24:12.8105408Z * [new branch] gh/rtimpe/3/base -> origin/gh/rtimpe/3/base 2025-08-14T21:24:12.8105559Z * [new branch] gh/rtimpe/3/head -> origin/gh/rtimpe/3/head 2025-08-14T21:24:12.8105724Z * [new branch] gh/rtimpe/4/base -> origin/gh/rtimpe/4/base 2025-08-14T21:24:12.8105868Z * [new branch] gh/rtimpe/4/head -> origin/gh/rtimpe/4/head 2025-08-14T21:24:12.8110896Z * [new branch] gh/rtimpe/5/base -> origin/gh/rtimpe/5/base 2025-08-14T21:24:12.8115335Z * [new branch] gh/rtimpe/5/head -> origin/gh/rtimpe/5/head 2025-08-14T21:24:12.8119893Z * [new branch] gh/rtimpe/5/orig -> origin/gh/rtimpe/5/orig 2025-08-14T21:24:12.8120075Z * [new branch] gh/rtimpe/6/base -> origin/gh/rtimpe/6/base 2025-08-14T21:24:12.8120523Z * [new branch] gh/rtimpe/6/head -> origin/gh/rtimpe/6/head 2025-08-14T21:24:12.8120677Z * [new branch] gh/rtimpe/6/orig -> origin/gh/rtimpe/6/orig 2025-08-14T21:24:12.8120812Z * [new branch] gh/rtimpe/7/base -> origin/gh/rtimpe/7/base 2025-08-14T21:24:12.8121103Z * [new branch] gh/rtimpe/7/head -> origin/gh/rtimpe/7/head 2025-08-14T21:24:12.8121257Z * [new branch] gh/rtimpe/7/orig -> origin/gh/rtimpe/7/orig 2025-08-14T21:24:12.8121425Z * [new branch] gh/rtimpe/8/base -> origin/gh/rtimpe/8/base 2025-08-14T21:24:12.8121560Z * [new branch] gh/rtimpe/8/head -> origin/gh/rtimpe/8/head 2025-08-14T21:24:12.8121705Z * [new branch] gh/rtimpe/8/orig -> origin/gh/rtimpe/8/orig 2025-08-14T21:24:12.8121844Z * [new branch] gh/rtimpe/9/base -> origin/gh/rtimpe/9/base 2025-08-14T21:24:12.8121979Z * [new branch] gh/rtimpe/9/head -> origin/gh/rtimpe/9/head 2025-08-14T21:24:12.8122116Z * [new branch] gh/rtimpe/9/orig -> origin/gh/rtimpe/9/orig 2025-08-14T21:24:12.8122298Z * [new branch] gh/ruisizhang123/1/base -> origin/gh/ruisizhang123/1/base 2025-08-14T21:24:12.8122466Z * [new branch] gh/ruisizhang123/1/head -> origin/gh/ruisizhang123/1/head 2025-08-14T21:24:12.8122633Z * [new branch] gh/ruisizhang123/1/orig -> origin/gh/ruisizhang123/1/orig 2025-08-14T21:24:12.8122789Z * [new branch] gh/ruisizhang123/4/base -> origin/gh/ruisizhang123/4/base 2025-08-14T21:24:12.8122945Z * [new branch] gh/ruisizhang123/4/head -> origin/gh/ruisizhang123/4/head 2025-08-14T21:24:12.8123099Z * [new branch] gh/ruisizhang123/4/orig -> origin/gh/ruisizhang123/4/orig 2025-08-14T21:24:12.8123251Z * [new branch] gh/ruisizhang123/5/base -> origin/gh/ruisizhang123/5/base 2025-08-14T21:24:12.8123410Z * [new branch] gh/ruisizhang123/5/head -> origin/gh/ruisizhang123/5/head 2025-08-14T21:24:12.8123562Z * [new branch] gh/ruisizhang123/5/orig -> origin/gh/ruisizhang123/5/orig 2025-08-14T21:24:12.8123724Z * [new branch] gh/ruisizhang123/6/base -> origin/gh/ruisizhang123/6/base 2025-08-14T21:24:12.8123940Z * [new branch] gh/ruisizhang123/6/head -> origin/gh/ruisizhang123/6/head 2025-08-14T21:24:12.8124092Z * [new branch] gh/ruisizhang123/6/orig -> origin/gh/ruisizhang123/6/orig 2025-08-14T21:24:12.8124252Z * [new branch] gh/ruisizhang123/7/base -> origin/gh/ruisizhang123/7/base 2025-08-14T21:24:12.8130235Z * [new branch] gh/ruisizhang123/7/head -> origin/gh/ruisizhang123/7/head 2025-08-14T21:24:12.8130490Z * [new branch] gh/ruisizhang123/7/orig -> origin/gh/ruisizhang123/7/orig 2025-08-14T21:24:12.8130705Z * [new branch] gh/ruisizhang123/8/base -> origin/gh/ruisizhang123/8/base 2025-08-14T21:24:12.8130886Z * [new branch] gh/ruisizhang123/8/head -> origin/gh/ruisizhang123/8/head 2025-08-14T21:24:12.8131170Z * [new branch] gh/ruisizhang123/8/orig -> origin/gh/ruisizhang123/8/orig 2025-08-14T21:24:12.8131398Z * [new branch] gh/sarckk/2/base -> origin/gh/sarckk/2/base 2025-08-14T21:24:12.8131639Z * [new branch] gh/sarckk/2/head -> origin/gh/sarckk/2/head 2025-08-14T21:24:12.8131999Z * [new branch] gh/sarckk/2/orig -> origin/gh/sarckk/2/orig 2025-08-14T21:24:12.8132557Z * [new branch] gh/seemethere/23/head -> origin/gh/seemethere/23/head 2025-08-14T21:24:12.8132744Z * [new branch] gh/seemethere/24/base -> origin/gh/seemethere/24/base 2025-08-14T21:24:12.8138461Z * [new branch] gh/seemethere/24/head -> origin/gh/seemethere/24/head 2025-08-14T21:24:12.8138647Z * [new branch] gh/seemethere/24/orig -> origin/gh/seemethere/24/orig 2025-08-14T21:24:12.8138812Z * [new branch] gh/seemethere/30/base -> origin/gh/seemethere/30/base 2025-08-14T21:24:12.8138971Z * [new branch] gh/seemethere/30/head -> origin/gh/seemethere/30/head 2025-08-14T21:24:12.8139355Z * [new branch] gh/seemethere/30/orig -> origin/gh/seemethere/30/orig 2025-08-14T21:24:12.8139509Z * [new branch] gh/seemethere/32/base -> origin/gh/seemethere/32/base 2025-08-14T21:24:12.8139671Z * [new branch] gh/seemethere/32/head -> origin/gh/seemethere/32/head 2025-08-14T21:24:12.8139829Z * [new branch] gh/seemethere/32/orig -> origin/gh/seemethere/32/orig 2025-08-14T21:24:12.8145159Z * [new branch] gh/seemethere/33/base -> origin/gh/seemethere/33/base 2025-08-14T21:24:12.8145498Z * [new branch] gh/seemethere/33/head -> origin/gh/seemethere/33/head 2025-08-14T21:24:12.8145674Z * [new branch] gh/seemethere/33/orig -> origin/gh/seemethere/33/orig 2025-08-14T21:24:12.8145920Z * [new branch] gh/seemethere/34/base -> origin/gh/seemethere/34/base 2025-08-14T21:24:12.8146314Z * [new branch] gh/seemethere/34/head -> origin/gh/seemethere/34/head 2025-08-14T21:24:12.8146520Z * [new branch] gh/seemethere/34/orig -> origin/gh/seemethere/34/orig 2025-08-14T21:24:12.8146660Z * [new branch] gh/seemethere/35/base -> origin/gh/seemethere/35/base 2025-08-14T21:24:12.8147093Z * [new branch] gh/seemethere/35/head -> origin/gh/seemethere/35/head 2025-08-14T21:24:12.8147545Z * [new branch] gh/seemethere/35/orig -> origin/gh/seemethere/35/orig 2025-08-14T21:24:12.8151639Z * [new branch] gh/seemethere/37/base -> origin/gh/seemethere/37/base 2025-08-14T21:24:12.8151820Z * [new branch] gh/seemethere/37/head -> origin/gh/seemethere/37/head 2025-08-14T21:24:12.8151967Z * [new branch] gh/seemethere/37/orig -> origin/gh/seemethere/37/orig 2025-08-14T21:24:12.8152104Z * [new branch] gh/seemethere/39/base -> origin/gh/seemethere/39/base 2025-08-14T21:24:12.8152262Z * [new branch] gh/seemethere/39/head -> origin/gh/seemethere/39/head 2025-08-14T21:24:12.8152619Z * [new branch] gh/seemethere/39/orig -> origin/gh/seemethere/39/orig 2025-08-14T21:24:12.8152828Z * [new branch] gh/seemethere/40/base -> origin/gh/seemethere/40/base 2025-08-14T21:24:12.8153234Z * [new branch] gh/seemethere/40/head -> origin/gh/seemethere/40/head 2025-08-14T21:24:12.8157123Z * [new branch] gh/seemethere/40/orig -> origin/gh/seemethere/40/orig 2025-08-14T21:24:12.8157458Z * [new branch] gh/seemethere/41/base -> origin/gh/seemethere/41/base 2025-08-14T21:24:12.8157618Z * [new branch] gh/seemethere/41/head -> origin/gh/seemethere/41/head 2025-08-14T21:24:12.8157764Z * [new branch] gh/seemethere/41/orig -> origin/gh/seemethere/41/orig 2025-08-14T21:24:12.8158016Z * [new branch] gh/seemethere/42/base -> origin/gh/seemethere/42/base 2025-08-14T21:24:12.8158236Z * [new branch] gh/seemethere/42/head -> origin/gh/seemethere/42/head 2025-08-14T21:24:12.8158480Z * [new branch] gh/seemethere/42/orig -> origin/gh/seemethere/42/orig 2025-08-14T21:24:12.8159026Z * [new branch] gh/seemethere/43/base -> origin/gh/seemethere/43/base 2025-08-14T21:24:12.8159736Z * [new branch] gh/seemethere/43/head -> origin/gh/seemethere/43/head 2025-08-14T21:24:12.8160514Z * [new branch] gh/seemethere/43/orig -> origin/gh/seemethere/43/orig 2025-08-14T21:24:12.8163375Z * [new branch] gh/seemethere/44/base -> origin/gh/seemethere/44/base 2025-08-14T21:24:12.8164015Z * [new branch] gh/seemethere/44/head -> origin/gh/seemethere/44/head 2025-08-14T21:24:12.8164202Z * [new branch] gh/seemethere/44/orig -> origin/gh/seemethere/44/orig 2025-08-14T21:24:12.8164497Z * [new branch] gh/seemethere/45/base -> origin/gh/seemethere/45/base 2025-08-14T21:24:12.8164669Z * [new branch] gh/seemethere/45/head -> origin/gh/seemethere/45/head 2025-08-14T21:24:12.8169009Z * [new branch] gh/seemethere/45/orig -> origin/gh/seemethere/45/orig 2025-08-14T21:24:12.8174824Z * [new branch] gh/seemethere/46/base -> origin/gh/seemethere/46/base 2025-08-14T21:24:12.8180417Z * [new branch] gh/seemethere/46/head -> origin/gh/seemethere/46/head 2025-08-14T21:24:12.8185374Z * [new branch] gh/seemethere/46/orig -> origin/gh/seemethere/46/orig 2025-08-14T21:24:12.8190276Z * [new branch] gh/seemethere/47/base -> origin/gh/seemethere/47/base 2025-08-14T21:24:12.8195694Z * [new branch] gh/seemethere/47/head -> origin/gh/seemethere/47/head 2025-08-14T21:24:12.8200705Z * [new branch] gh/seemethere/47/orig -> origin/gh/seemethere/47/orig 2025-08-14T21:24:12.8202603Z * [new branch] gh/seemethere/48/base -> origin/gh/seemethere/48/base 2025-08-14T21:24:12.8202774Z * [new branch] gh/seemethere/48/head -> origin/gh/seemethere/48/head 2025-08-14T21:24:12.8203169Z * [new branch] gh/seemethere/48/orig -> origin/gh/seemethere/48/orig 2025-08-14T21:24:12.8203328Z * [new branch] gh/seemethere/49/base -> origin/gh/seemethere/49/base 2025-08-14T21:24:12.8203480Z * [new branch] gh/seemethere/49/head -> origin/gh/seemethere/49/head 2025-08-14T21:24:12.8203623Z * [new branch] gh/seemethere/49/orig -> origin/gh/seemethere/49/orig 2025-08-14T21:24:12.8203766Z * [new branch] gh/seemethere/50/base -> origin/gh/seemethere/50/base 2025-08-14T21:24:12.8203919Z * [new branch] gh/seemethere/50/head -> origin/gh/seemethere/50/head 2025-08-14T21:24:12.8204060Z * [new branch] gh/seemethere/50/orig -> origin/gh/seemethere/50/orig 2025-08-14T21:24:12.8204233Z * [new branch] gh/seemethere/51/base -> origin/gh/seemethere/51/base 2025-08-14T21:24:12.8204538Z * [new branch] gh/seemethere/51/head -> origin/gh/seemethere/51/head 2025-08-14T21:24:12.8204678Z * [new branch] gh/seemethere/51/orig -> origin/gh/seemethere/51/orig 2025-08-14T21:24:12.8204826Z * [new branch] gh/seemethere/52/base -> origin/gh/seemethere/52/base 2025-08-14T21:24:12.8204966Z * [new branch] gh/seemethere/52/head -> origin/gh/seemethere/52/head 2025-08-14T21:24:12.8205116Z * [new branch] gh/seemethere/52/orig -> origin/gh/seemethere/52/orig 2025-08-14T21:24:12.8205257Z * [new branch] gh/seemethere/53/base -> origin/gh/seemethere/53/base 2025-08-14T21:24:12.8205397Z * [new branch] gh/seemethere/53/head -> origin/gh/seemethere/53/head 2025-08-14T21:24:12.8205614Z * [new branch] gh/seemethere/53/orig -> origin/gh/seemethere/53/orig 2025-08-14T21:24:12.8205785Z * [new branch] gh/seemethere/54/base -> origin/gh/seemethere/54/base 2025-08-14T21:24:12.8205929Z * [new branch] gh/seemethere/54/head -> origin/gh/seemethere/54/head 2025-08-14T21:24:12.8206067Z * [new branch] gh/seemethere/54/orig -> origin/gh/seemethere/54/orig 2025-08-14T21:24:12.8206208Z * [new branch] gh/seemethere/55/base -> origin/gh/seemethere/55/base 2025-08-14T21:24:12.8206355Z * [new branch] gh/seemethere/55/head -> origin/gh/seemethere/55/head 2025-08-14T21:24:12.8206503Z * [new branch] gh/seemethere/55/orig -> origin/gh/seemethere/55/orig 2025-08-14T21:24:12.8206653Z * [new branch] gh/seemethere/56/base -> origin/gh/seemethere/56/base 2025-08-14T21:24:12.8206809Z * [new branch] gh/seemethere/56/head -> origin/gh/seemethere/56/head 2025-08-14T21:24:12.8207012Z * [new branch] gh/seemethere/56/orig -> origin/gh/seemethere/56/orig 2025-08-14T21:24:12.8207166Z * [new branch] gh/seemethere/57/base -> origin/gh/seemethere/57/base 2025-08-14T21:24:12.8207306Z * [new branch] gh/seemethere/57/head -> origin/gh/seemethere/57/head 2025-08-14T21:24:12.8207442Z * [new branch] gh/seemethere/57/orig -> origin/gh/seemethere/57/orig 2025-08-14T21:24:12.8207588Z * [new branch] gh/seemethere/58/base -> origin/gh/seemethere/58/base 2025-08-14T21:24:12.8207724Z * [new branch] gh/seemethere/58/head -> origin/gh/seemethere/58/head 2025-08-14T21:24:12.8207869Z * [new branch] gh/seemethere/58/orig -> origin/gh/seemethere/58/orig 2025-08-14T21:24:12.8208016Z * [new branch] gh/seemethere/59/base -> origin/gh/seemethere/59/base 2025-08-14T21:24:12.8208153Z * [new branch] gh/seemethere/59/head -> origin/gh/seemethere/59/head 2025-08-14T21:24:12.8208303Z * [new branch] gh/seemethere/59/orig -> origin/gh/seemethere/59/orig 2025-08-14T21:24:12.8208463Z * [new branch] gh/seemethere/7/head -> origin/gh/seemethere/7/head 2025-08-14T21:24:12.8208627Z * [new branch] gh/shunting314/145/base -> origin/gh/shunting314/145/base 2025-08-14T21:24:12.8208785Z * [new branch] gh/shunting314/145/head -> origin/gh/shunting314/145/head 2025-08-14T21:24:12.8208920Z * [new branch] gh/shunting314/145/orig -> origin/gh/shunting314/145/orig 2025-08-14T21:24:12.8209061Z * [new branch] gh/shunting314/176/base -> origin/gh/shunting314/176/base 2025-08-14T21:24:12.8209192Z * [new branch] gh/shunting314/176/head -> origin/gh/shunting314/176/head 2025-08-14T21:24:12.8209325Z * [new branch] gh/shunting314/176/orig -> origin/gh/shunting314/176/orig 2025-08-14T21:24:12.8209582Z * [new branch] gh/shunting314/211/base -> origin/gh/shunting314/211/base 2025-08-14T21:24:12.8209790Z * [new branch] gh/shunting314/211/head -> origin/gh/shunting314/211/head 2025-08-14T21:24:12.8209946Z * [new branch] gh/shunting314/211/orig -> origin/gh/shunting314/211/orig 2025-08-14T21:24:12.8210098Z * [new branch] gh/shunting314/212/base -> origin/gh/shunting314/212/base 2025-08-14T21:24:12.8210251Z * [new branch] gh/shunting314/212/head -> origin/gh/shunting314/212/head 2025-08-14T21:24:12.8210411Z * [new branch] gh/shunting314/212/orig -> origin/gh/shunting314/212/orig 2025-08-14T21:24:12.8210572Z * [new branch] gh/shunting314/213/base -> origin/gh/shunting314/213/base 2025-08-14T21:24:12.8210727Z * [new branch] gh/shunting314/213/head -> origin/gh/shunting314/213/head 2025-08-14T21:24:12.8210881Z * [new branch] gh/shunting314/213/orig -> origin/gh/shunting314/213/orig 2025-08-14T21:24:12.8211035Z * [new branch] gh/silverguo/1/base -> origin/gh/silverguo/1/base 2025-08-14T21:24:12.8211454Z * [new branch] gh/silverguo/1/head -> origin/gh/silverguo/1/head 2025-08-14T21:24:12.8211604Z * [new branch] gh/silverguo/2/base -> origin/gh/silverguo/2/base 2025-08-14T21:24:12.8211753Z * [new branch] gh/silverguo/2/head -> origin/gh/silverguo/2/head 2025-08-14T21:24:12.8211899Z * [new branch] gh/silverguo/3/base -> origin/gh/silverguo/3/base 2025-08-14T21:24:12.8212048Z * [new branch] gh/silverguo/3/head -> origin/gh/silverguo/3/head 2025-08-14T21:24:12.8212209Z * [new branch] gh/silverguo/4/base -> origin/gh/silverguo/4/base 2025-08-14T21:24:12.8216222Z * [new branch] gh/silverguo/4/head -> origin/gh/silverguo/4/head 2025-08-14T21:24:12.8216724Z * [new branch] gh/sinhaanhsul/1/base -> origin/gh/sinhaanhsul/1/base 2025-08-14T21:24:12.8217043Z * [new branch] gh/sinhaanhsul/1/head -> origin/gh/sinhaanhsul/1/head 2025-08-14T21:24:12.8217236Z * [new branch] gh/skarjala/11/base -> origin/gh/skarjala/11/base 2025-08-14T21:24:12.8217370Z * [new branch] gh/skarjala/11/head -> origin/gh/skarjala/11/head 2025-08-14T21:24:12.8217506Z * [new branch] gh/skarjala/11/orig -> origin/gh/skarjala/11/orig 2025-08-14T21:24:12.8222859Z * [new branch] gh/skarjala/13/base -> origin/gh/skarjala/13/base 2025-08-14T21:24:12.8227113Z * [new branch] gh/skarjala/13/head -> origin/gh/skarjala/13/head 2025-08-14T21:24:12.8231195Z * [new branch] gh/skarjala/13/orig -> origin/gh/skarjala/13/orig 2025-08-14T21:24:12.8236007Z * [new branch] gh/skarjala/14/base -> origin/gh/skarjala/14/base 2025-08-14T21:24:12.8239896Z * [new branch] gh/skarjala/14/head -> origin/gh/skarjala/14/head 2025-08-14T21:24:12.8240075Z * [new branch] gh/skarjala/14/orig -> origin/gh/skarjala/14/orig 2025-08-14T21:24:12.8240514Z * [new branch] gh/skarjala/15/base -> origin/gh/skarjala/15/base 2025-08-14T21:24:12.8240662Z * [new branch] gh/skarjala/15/head -> origin/gh/skarjala/15/head 2025-08-14T21:24:12.8240833Z * [new branch] gh/skarjala/15/orig -> origin/gh/skarjala/15/orig 2025-08-14T21:24:12.8240978Z * [new branch] gh/skarjala/16/base -> origin/gh/skarjala/16/base 2025-08-14T21:24:12.8241122Z * [new branch] gh/skarjala/16/head -> origin/gh/skarjala/16/head 2025-08-14T21:24:12.8241268Z * [new branch] gh/skarjala/16/orig -> origin/gh/skarjala/16/orig 2025-08-14T21:24:12.8241411Z * [new branch] gh/skarjala/17/base -> origin/gh/skarjala/17/base 2025-08-14T21:24:12.8241567Z * [new branch] gh/skarjala/17/head -> origin/gh/skarjala/17/head 2025-08-14T21:24:12.8241930Z * [new branch] gh/skarjala/17/orig -> origin/gh/skarjala/17/orig 2025-08-14T21:24:12.8242098Z * [new branch] gh/skarjala/18/base -> origin/gh/skarjala/18/base 2025-08-14T21:24:12.8242246Z * [new branch] gh/skarjala/18/head -> origin/gh/skarjala/18/head 2025-08-14T21:24:12.8242386Z * [new branch] gh/skarjala/18/orig -> origin/gh/skarjala/18/orig 2025-08-14T21:24:12.8242527Z * [new branch] gh/skarjala/19/base -> origin/gh/skarjala/19/base 2025-08-14T21:24:12.8242675Z * [new branch] gh/skarjala/19/head -> origin/gh/skarjala/19/head 2025-08-14T21:24:12.8242814Z * [new branch] gh/skarjala/19/orig -> origin/gh/skarjala/19/orig 2025-08-14T21:24:12.8242978Z * [new branch] gh/soulitzer/269/base -> origin/gh/soulitzer/269/base 2025-08-14T21:24:12.8243131Z * [new branch] gh/soulitzer/269/head -> origin/gh/soulitzer/269/head 2025-08-14T21:24:12.8243279Z * [new branch] gh/soulitzer/269/orig -> origin/gh/soulitzer/269/orig 2025-08-14T21:24:12.8243437Z * [new branch] gh/soulitzer/276/base -> origin/gh/soulitzer/276/base 2025-08-14T21:24:12.8243586Z * [new branch] gh/soulitzer/276/head -> origin/gh/soulitzer/276/head 2025-08-14T21:24:12.8243738Z * [new branch] gh/soulitzer/276/orig -> origin/gh/soulitzer/276/orig 2025-08-14T21:24:12.8243889Z * [new branch] gh/soulitzer/287/base -> origin/gh/soulitzer/287/base 2025-08-14T21:24:12.8244037Z * [new branch] gh/soulitzer/287/head -> origin/gh/soulitzer/287/head 2025-08-14T21:24:12.8244187Z * [new branch] gh/soulitzer/287/orig -> origin/gh/soulitzer/287/orig 2025-08-14T21:24:12.8244332Z * [new branch] gh/soulitzer/296/base -> origin/gh/soulitzer/296/base 2025-08-14T21:24:12.8244569Z * [new branch] gh/soulitzer/296/head -> origin/gh/soulitzer/296/head 2025-08-14T21:24:12.8244730Z * [new branch] gh/soulitzer/296/orig -> origin/gh/soulitzer/296/orig 2025-08-14T21:24:12.8244878Z * [new branch] gh/soulitzer/299/base -> origin/gh/soulitzer/299/base 2025-08-14T21:24:12.8245024Z * [new branch] gh/soulitzer/299/head -> origin/gh/soulitzer/299/head 2025-08-14T21:24:12.8245175Z * [new branch] gh/soulitzer/299/orig -> origin/gh/soulitzer/299/orig 2025-08-14T21:24:12.8245321Z * [new branch] gh/soulitzer/300/base -> origin/gh/soulitzer/300/base 2025-08-14T21:24:12.8245477Z * [new branch] gh/soulitzer/300/head -> origin/gh/soulitzer/300/head 2025-08-14T21:24:12.8245780Z * [new branch] gh/soulitzer/300/orig -> origin/gh/soulitzer/300/orig 2025-08-14T21:24:12.8245966Z * [new branch] gh/soulitzer/301/base -> origin/gh/soulitzer/301/base 2025-08-14T21:24:12.8246118Z * [new branch] gh/soulitzer/301/head -> origin/gh/soulitzer/301/head 2025-08-14T21:24:12.8246277Z * [new branch] gh/soulitzer/301/orig -> origin/gh/soulitzer/301/orig 2025-08-14T21:24:12.8251690Z * [new branch] gh/soulitzer/313/base -> origin/gh/soulitzer/313/base 2025-08-14T21:24:12.8251876Z * [new branch] gh/soulitzer/313/head -> origin/gh/soulitzer/313/head 2025-08-14T21:24:12.8252029Z * [new branch] gh/soulitzer/313/orig -> origin/gh/soulitzer/313/orig 2025-08-14T21:24:12.8252195Z * [new branch] gh/soulitzer/319/base -> origin/gh/soulitzer/319/base 2025-08-14T21:24:12.8252364Z * [new branch] gh/soulitzer/319/head -> origin/gh/soulitzer/319/head 2025-08-14T21:24:12.8252532Z * [new branch] gh/soulitzer/319/orig -> origin/gh/soulitzer/319/orig 2025-08-14T21:24:12.8254279Z * [new branch] gh/soulitzer/320/base -> origin/gh/soulitzer/320/base 2025-08-14T21:24:12.8254819Z * [new branch] gh/soulitzer/320/head -> origin/gh/soulitzer/320/head 2025-08-14T21:24:12.8255068Z * [new branch] gh/soulitzer/320/orig -> origin/gh/soulitzer/320/orig 2025-08-14T21:24:12.8255228Z * [new branch] gh/soulitzer/336/base -> origin/gh/soulitzer/336/base 2025-08-14T21:24:12.8255486Z * [new branch] gh/soulitzer/336/head -> origin/gh/soulitzer/336/head 2025-08-14T21:24:12.8259349Z * [new branch] gh/soulitzer/336/orig -> origin/gh/soulitzer/336/orig 2025-08-14T21:24:12.8259657Z * [new branch] gh/soulitzer/347/base -> origin/gh/soulitzer/347/base 2025-08-14T21:24:12.8259824Z * [new branch] gh/soulitzer/347/head -> origin/gh/soulitzer/347/head 2025-08-14T21:24:12.8259986Z * [new branch] gh/soulitzer/347/orig -> origin/gh/soulitzer/347/orig 2025-08-14T21:24:12.8260159Z * [new branch] gh/soulitzer/349/base -> origin/gh/soulitzer/349/base 2025-08-14T21:24:12.8260802Z * [new branch] gh/soulitzer/349/head -> origin/gh/soulitzer/349/head 2025-08-14T21:24:12.8260968Z * [new branch] gh/soulitzer/349/orig -> origin/gh/soulitzer/349/orig 2025-08-14T21:24:12.8261109Z * [new branch] gh/soulitzer/350/base -> origin/gh/soulitzer/350/base 2025-08-14T21:24:12.8266436Z * [new branch] gh/soulitzer/350/head -> origin/gh/soulitzer/350/head 2025-08-14T21:24:12.8266618Z * [new branch] gh/soulitzer/350/orig -> origin/gh/soulitzer/350/orig 2025-08-14T21:24:12.8266751Z * [new branch] gh/soulitzer/351/base -> origin/gh/soulitzer/351/base 2025-08-14T21:24:12.8266878Z * [new branch] gh/soulitzer/351/head -> origin/gh/soulitzer/351/head 2025-08-14T21:24:12.8267156Z * [new branch] gh/soulitzer/351/orig -> origin/gh/soulitzer/351/orig 2025-08-14T21:24:12.8267299Z * [new branch] gh/soulitzer/353/base -> origin/gh/soulitzer/353/base 2025-08-14T21:24:12.8267424Z * [new branch] gh/soulitzer/353/head -> origin/gh/soulitzer/353/head 2025-08-14T21:24:12.8267555Z * [new branch] gh/soulitzer/353/orig -> origin/gh/soulitzer/353/orig 2025-08-14T21:24:12.8268754Z * [new branch] gh/soulitzer/358/base -> origin/gh/soulitzer/358/base 2025-08-14T21:24:12.8268922Z * [new branch] gh/soulitzer/358/head -> origin/gh/soulitzer/358/head 2025-08-14T21:24:12.8269238Z * [new branch] gh/soulitzer/358/orig -> origin/gh/soulitzer/358/orig 2025-08-14T21:24:12.8269397Z * [new branch] gh/soulitzer/359/base -> origin/gh/soulitzer/359/base 2025-08-14T21:24:12.8269676Z * [new branch] gh/soulitzer/359/head -> origin/gh/soulitzer/359/head 2025-08-14T21:24:12.8269882Z * [new branch] gh/soulitzer/359/orig -> origin/gh/soulitzer/359/orig 2025-08-14T21:24:12.8275280Z * [new branch] gh/soulitzer/362/base -> origin/gh/soulitzer/362/base 2025-08-14T21:24:12.8275615Z * [new branch] gh/soulitzer/362/head -> origin/gh/soulitzer/362/head 2025-08-14T21:24:12.8275834Z * [new branch] gh/soulitzer/362/orig -> origin/gh/soulitzer/362/orig 2025-08-14T21:24:12.8276138Z * [new branch] gh/soulitzer/372/base -> origin/gh/soulitzer/372/base 2025-08-14T21:24:12.8276357Z * [new branch] gh/soulitzer/372/head -> origin/gh/soulitzer/372/head 2025-08-14T21:24:12.8276503Z * [new branch] gh/soulitzer/372/orig -> origin/gh/soulitzer/372/orig 2025-08-14T21:24:12.8276730Z * [new branch] gh/swolchok/728/next -> origin/gh/swolchok/728/next 2025-08-14T21:24:12.8278178Z * [new branch] gh/swolchok/758/base -> origin/gh/swolchok/758/base 2025-08-14T21:24:12.8278497Z * [new branch] gh/swolchok/758/head -> origin/gh/swolchok/758/head 2025-08-14T21:24:12.8278892Z * [new branch] gh/swolchok/758/orig -> origin/gh/swolchok/758/orig 2025-08-14T21:24:12.8279057Z * [new branch] gh/swolchok/767/base -> origin/gh/swolchok/767/base 2025-08-14T21:24:12.8279258Z * [new branch] gh/swolchok/767/head -> origin/gh/swolchok/767/head 2025-08-14T21:24:12.8279471Z * [new branch] gh/swolchok/767/orig -> origin/gh/swolchok/767/orig 2025-08-14T21:24:12.8280750Z * [new branch] gh/swolchok/768/base -> origin/gh/swolchok/768/base 2025-08-14T21:24:12.8280933Z * [new branch] gh/swolchok/768/head -> origin/gh/swolchok/768/head 2025-08-14T21:24:12.8281360Z * [new branch] gh/swolchok/768/orig -> origin/gh/swolchok/768/orig 2025-08-14T21:24:12.8283513Z * [new branch] gh/swolchok/769/base -> origin/gh/swolchok/769/base 2025-08-14T21:24:12.8283729Z * [new branch] gh/swolchok/769/head -> origin/gh/swolchok/769/head 2025-08-14T21:24:12.8283870Z * [new branch] gh/swolchok/769/orig -> origin/gh/swolchok/769/orig 2025-08-14T21:24:12.8289207Z * [new branch] gh/swolchok/771/base -> origin/gh/swolchok/771/base 2025-08-14T21:24:12.8293915Z * [new branch] gh/swolchok/771/head -> origin/gh/swolchok/771/head 2025-08-14T21:24:12.8294128Z * [new branch] gh/swolchok/771/orig -> origin/gh/swolchok/771/orig 2025-08-14T21:24:12.8294261Z * [new branch] gh/swolchok/772/base -> origin/gh/swolchok/772/base 2025-08-14T21:24:12.8294402Z * [new branch] gh/swolchok/772/head -> origin/gh/swolchok/772/head 2025-08-14T21:24:12.8294530Z * [new branch] gh/swolchok/772/orig -> origin/gh/swolchok/772/orig 2025-08-14T21:24:12.8294817Z * [new branch] gh/swolchok/773/base -> origin/gh/swolchok/773/base 2025-08-14T21:24:12.8294967Z * [new branch] gh/swolchok/773/head -> origin/gh/swolchok/773/head 2025-08-14T21:24:12.8295199Z * [new branch] gh/swolchok/773/orig -> origin/gh/swolchok/773/orig 2025-08-14T21:24:12.8297610Z * [new branch] gh/swolchok/786/base -> origin/gh/swolchok/786/base 2025-08-14T21:24:12.8297789Z * [new branch] gh/swolchok/786/head -> origin/gh/swolchok/786/head 2025-08-14T21:24:12.8297941Z * [new branch] gh/swolchok/786/orig -> origin/gh/swolchok/786/orig 2025-08-14T21:24:12.8298082Z * [new branch] gh/swolchok/787/base -> origin/gh/swolchok/787/base 2025-08-14T21:24:12.8298233Z * [new branch] gh/swolchok/787/head -> origin/gh/swolchok/787/head 2025-08-14T21:24:12.8298444Z * [new branch] gh/swolchok/787/orig -> origin/gh/swolchok/787/orig 2025-08-14T21:24:12.8298604Z * [new branch] gh/syed-ahmed/2/base -> origin/gh/syed-ahmed/2/base 2025-08-14T21:24:12.8298784Z * [new branch] gh/syed-ahmed/2/head -> origin/gh/syed-ahmed/2/head 2025-08-14T21:24:12.8298930Z * [new branch] gh/syed-ahmed/2/orig -> origin/gh/syed-ahmed/2/orig 2025-08-14T21:24:12.8299078Z * [new branch] gh/syed-ahmed/3/base -> origin/gh/syed-ahmed/3/base 2025-08-14T21:24:12.8304588Z * [new branch] gh/syed-ahmed/3/head -> origin/gh/syed-ahmed/3/head 2025-08-14T21:24:12.8307796Z * [new branch] gh/syed-ahmed/3/orig -> origin/gh/syed-ahmed/3/orig 2025-08-14T21:24:12.8311439Z * [new branch] gh/syed-ahmed/4/base -> origin/gh/syed-ahmed/4/base 2025-08-14T21:24:12.8315304Z * [new branch] gh/syed-ahmed/4/head -> origin/gh/syed-ahmed/4/head 2025-08-14T21:24:12.8319600Z * [new branch] gh/syed-ahmed/4/orig -> origin/gh/syed-ahmed/4/orig 2025-08-14T21:24:12.8323786Z * [new branch] gh/teja-rao/3/base -> origin/gh/teja-rao/3/base 2025-08-14T21:24:12.8324172Z * [new branch] gh/teja-rao/3/head -> origin/gh/teja-rao/3/head 2025-08-14T21:24:12.8324311Z * [new branch] gh/teja-rao/3/orig -> origin/gh/teja-rao/3/orig 2025-08-14T21:24:12.8324449Z * [new branch] gh/tianyu-l/2/base -> origin/gh/tianyu-l/2/base 2025-08-14T21:24:12.8324590Z * [new branch] gh/tianyu-l/2/head -> origin/gh/tianyu-l/2/head 2025-08-14T21:24:12.8324722Z * [new branch] gh/tianyu-l/2/orig -> origin/gh/tianyu-l/2/orig 2025-08-14T21:24:12.8324880Z * [new branch] gh/titaiwangms/1/base -> origin/gh/titaiwangms/1/base 2025-08-14T21:24:12.8325035Z * [new branch] gh/titaiwangms/1/head -> origin/gh/titaiwangms/1/head 2025-08-14T21:24:12.8325180Z * [new branch] gh/titaiwangms/1/orig -> origin/gh/titaiwangms/1/orig 2025-08-14T21:24:12.8325338Z * [new branch] gh/titaiwangms/2/base -> origin/gh/titaiwangms/2/base 2025-08-14T21:24:12.8325490Z * [new branch] gh/titaiwangms/2/head -> origin/gh/titaiwangms/2/head 2025-08-14T21:24:12.8325915Z * [new branch] gh/titaiwangms/2/orig -> origin/gh/titaiwangms/2/orig 2025-08-14T21:24:12.8326076Z * [new branch] gh/titaiwangms/3/base -> origin/gh/titaiwangms/3/base 2025-08-14T21:24:12.8326225Z * [new branch] gh/titaiwangms/3/head -> origin/gh/titaiwangms/3/head 2025-08-14T21:24:12.8326372Z * [new branch] gh/titaiwangms/3/orig -> origin/gh/titaiwangms/3/orig 2025-08-14T21:24:12.8326532Z * [new branch] gh/titaiwangms/4/base -> origin/gh/titaiwangms/4/base 2025-08-14T21:24:12.8326681Z * [new branch] gh/titaiwangms/4/head -> origin/gh/titaiwangms/4/head 2025-08-14T21:24:12.8326919Z * [new branch] gh/titaiwangms/4/orig -> origin/gh/titaiwangms/4/orig 2025-08-14T21:24:12.8327071Z * [new branch] gh/titaiwangms/5/base -> origin/gh/titaiwangms/5/base 2025-08-14T21:24:12.8327215Z * [new branch] gh/titaiwangms/5/head -> origin/gh/titaiwangms/5/head 2025-08-14T21:24:12.8327367Z * [new branch] gh/titaiwangms/5/orig -> origin/gh/titaiwangms/5/orig 2025-08-14T21:24:12.8327511Z * [new branch] gh/titaiwangms/6/base -> origin/gh/titaiwangms/6/base 2025-08-14T21:24:12.8327660Z * [new branch] gh/titaiwangms/6/head -> origin/gh/titaiwangms/6/head 2025-08-14T21:24:12.8327804Z * [new branch] gh/titaiwangms/6/orig -> origin/gh/titaiwangms/6/orig 2025-08-14T21:24:12.8327948Z * [new branch] gh/titaiwangms/7/base -> origin/gh/titaiwangms/7/base 2025-08-14T21:24:12.8328102Z * [new branch] gh/titaiwangms/7/head -> origin/gh/titaiwangms/7/head 2025-08-14T21:24:12.8328253Z * [new branch] gh/titaiwangms/7/orig -> origin/gh/titaiwangms/7/orig 2025-08-14T21:24:12.8328416Z * [new branch] gh/titaiwangms/8/base -> origin/gh/titaiwangms/8/base 2025-08-14T21:24:12.8328553Z * [new branch] gh/titaiwangms/8/head -> origin/gh/titaiwangms/8/head 2025-08-14T21:24:12.8329687Z * [new branch] gh/titaiwangms/8/orig -> origin/gh/titaiwangms/8/orig 2025-08-14T21:24:12.8330199Z * [new branch] gh/tugsbayasgalan/1/base -> origin/gh/tugsbayasgalan/1/base 2025-08-14T21:24:12.8330389Z * [new branch] gh/tugsbayasgalan/1/head -> origin/gh/tugsbayasgalan/1/head 2025-08-14T21:24:12.8330546Z * [new branch] gh/tugsbayasgalan/1/orig -> origin/gh/tugsbayasgalan/1/orig 2025-08-14T21:24:12.8330678Z * [new branch] gh/v0i0/1/base -> origin/gh/v0i0/1/base 2025-08-14T21:24:12.8330805Z * [new branch] gh/v0i0/1/head -> origin/gh/v0i0/1/head 2025-08-14T21:24:12.8336680Z * [new branch] gh/v0i0/1/orig -> origin/gh/v0i0/1/orig 2025-08-14T21:24:12.8340637Z * [new branch] gh/v0i0/2/base -> origin/gh/v0i0/2/base 2025-08-14T21:24:12.8340962Z * [new branch] gh/v0i0/2/head -> origin/gh/v0i0/2/head 2025-08-14T21:24:12.8341164Z * [new branch] gh/v0i0/2/orig -> origin/gh/v0i0/2/orig 2025-08-14T21:24:12.8341370Z * [new branch] gh/v0i0/3/base -> origin/gh/v0i0/3/base 2025-08-14T21:24:12.8341507Z * [new branch] gh/v0i0/3/head -> origin/gh/v0i0/3/head 2025-08-14T21:24:12.8341632Z * [new branch] gh/v0i0/3/orig -> origin/gh/v0i0/3/orig 2025-08-14T21:24:12.8341891Z * [new branch] gh/v0i0/4/base -> origin/gh/v0i0/4/base 2025-08-14T21:24:12.8342026Z * [new branch] gh/v0i0/4/head -> origin/gh/v0i0/4/head 2025-08-14T21:24:12.8342261Z * [new branch] gh/v0i0/4/orig -> origin/gh/v0i0/4/orig 2025-08-14T21:24:12.8342428Z * [new branch] gh/v0i0/5/base -> origin/gh/v0i0/5/base 2025-08-14T21:24:12.8342552Z * [new branch] gh/v0i0/5/head -> origin/gh/v0i0/5/head 2025-08-14T21:24:12.8342796Z * [new branch] gh/v0i0/5/orig -> origin/gh/v0i0/5/orig 2025-08-14T21:24:12.8342964Z * [new branch] gh/v0i0/6/base -> origin/gh/v0i0/6/base 2025-08-14T21:24:12.8343084Z * [new branch] gh/v0i0/6/head -> origin/gh/v0i0/6/head 2025-08-14T21:24:12.8343193Z * [new branch] gh/v0i0/6/orig -> origin/gh/v0i0/6/orig 2025-08-14T21:24:12.8343324Z * [new branch] gh/vkuzo/1/next -> origin/gh/vkuzo/1/next 2025-08-14T21:24:12.8343448Z * [new branch] gh/vkuzo/2/next -> origin/gh/vkuzo/2/next 2025-08-14T21:24:12.8345086Z * [new branch] gh/vkuzo/3/next -> origin/gh/vkuzo/3/next 2025-08-14T21:24:12.8345502Z * [new branch] gh/wconstab/392/base -> origin/gh/wconstab/392/base 2025-08-14T21:24:12.8347860Z * [new branch] gh/wconstab/392/head -> origin/gh/wconstab/392/head 2025-08-14T21:24:12.8348189Z * [new branch] gh/wconstab/392/orig -> origin/gh/wconstab/392/orig 2025-08-14T21:24:12.8348425Z * [new branch] gh/wconstab/419/base -> origin/gh/wconstab/419/base 2025-08-14T21:24:12.8348578Z * [new branch] gh/wconstab/419/head -> origin/gh/wconstab/419/head 2025-08-14T21:24:12.8348876Z * [new branch] gh/wconstab/419/orig -> origin/gh/wconstab/419/orig 2025-08-14T21:24:12.8352987Z * [new branch] gh/wconstab/424/base -> origin/gh/wconstab/424/base 2025-08-14T21:24:12.8353297Z * [new branch] gh/wconstab/424/head -> origin/gh/wconstab/424/head 2025-08-14T21:24:12.8353489Z * [new branch] gh/wconstab/424/orig -> origin/gh/wconstab/424/orig 2025-08-14T21:24:12.8353698Z * [new branch] gh/wconstab/425/base -> origin/gh/wconstab/425/base 2025-08-14T21:24:12.8353854Z * [new branch] gh/wconstab/425/head -> origin/gh/wconstab/425/head 2025-08-14T21:24:12.8353991Z * [new branch] gh/wconstab/425/orig -> origin/gh/wconstab/425/orig 2025-08-14T21:24:12.8354553Z * [new branch] gh/wconstab/426/base -> origin/gh/wconstab/426/base 2025-08-14T21:24:12.8355481Z * [new branch] gh/wconstab/426/head -> origin/gh/wconstab/426/head 2025-08-14T21:24:12.8355692Z * [new branch] gh/wconstab/426/orig -> origin/gh/wconstab/426/orig 2025-08-14T21:24:12.8359659Z * [new branch] gh/wconstab/427/base -> origin/gh/wconstab/427/base 2025-08-14T21:24:12.8359970Z * [new branch] gh/wconstab/427/head -> origin/gh/wconstab/427/head 2025-08-14T21:24:12.8360148Z * [new branch] gh/wconstab/427/orig -> origin/gh/wconstab/427/orig 2025-08-14T21:24:12.8360630Z * [new branch] gh/wconstab/428/base -> origin/gh/wconstab/428/base 2025-08-14T21:24:12.8361079Z * [new branch] gh/wconstab/428/head -> origin/gh/wconstab/428/head 2025-08-14T21:24:12.8361246Z * [new branch] gh/wconstab/428/orig -> origin/gh/wconstab/428/orig 2025-08-14T21:24:12.8361397Z * [new branch] gh/wconstab/429/base -> origin/gh/wconstab/429/base 2025-08-14T21:24:12.8363413Z * [new branch] gh/wconstab/429/head -> origin/gh/wconstab/429/head 2025-08-14T21:24:12.8363593Z * [new branch] gh/wconstab/429/orig -> origin/gh/wconstab/429/orig 2025-08-14T21:24:12.8363815Z * [new branch] gh/wconstab/430/base -> origin/gh/wconstab/430/base 2025-08-14T21:24:12.8364620Z * [new branch] gh/wconstab/430/head -> origin/gh/wconstab/430/head 2025-08-14T21:24:12.8365142Z * [new branch] gh/wconstab/430/orig -> origin/gh/wconstab/430/orig 2025-08-14T21:24:12.8370682Z * [new branch] gh/wconstab/431/base -> origin/gh/wconstab/431/base 2025-08-14T21:24:12.8370917Z * [new branch] gh/wconstab/431/head -> origin/gh/wconstab/431/head 2025-08-14T21:24:12.8371069Z * [new branch] gh/wconstab/431/orig -> origin/gh/wconstab/431/orig 2025-08-14T21:24:12.8371363Z * [new branch] gh/wconstab/432/base -> origin/gh/wconstab/432/base 2025-08-14T21:24:12.8371526Z * [new branch] gh/wconstab/432/head -> origin/gh/wconstab/432/head 2025-08-14T21:24:12.8371791Z * [new branch] gh/wconstab/432/orig -> origin/gh/wconstab/432/orig 2025-08-14T21:24:12.8371978Z * [new branch] gh/wconstab/433/base -> origin/gh/wconstab/433/base 2025-08-14T21:24:12.8372621Z * [new branch] gh/wconstab/433/head -> origin/gh/wconstab/433/head 2025-08-14T21:24:12.8378458Z * [new branch] gh/wconstab/433/orig -> origin/gh/wconstab/433/orig 2025-08-14T21:24:12.8378807Z * [new branch] gh/wconstab/434/base -> origin/gh/wconstab/434/base 2025-08-14T21:24:12.8378977Z * [new branch] gh/wconstab/434/head -> origin/gh/wconstab/434/head 2025-08-14T21:24:12.8379140Z * [new branch] gh/wconstab/434/orig -> origin/gh/wconstab/434/orig 2025-08-14T21:24:12.8379432Z * [new branch] gh/wconstab/435/base -> origin/gh/wconstab/435/base 2025-08-14T21:24:12.8379607Z * [new branch] gh/wconstab/435/head -> origin/gh/wconstab/435/head 2025-08-14T21:24:12.8379753Z * [new branch] gh/wconstab/435/orig -> origin/gh/wconstab/435/orig 2025-08-14T21:24:12.8379978Z * [new branch] gh/wconstab/436/base -> origin/gh/wconstab/436/base 2025-08-14T21:24:12.8380154Z * [new branch] gh/wconstab/436/head -> origin/gh/wconstab/436/head 2025-08-14T21:24:12.8380306Z * [new branch] gh/wconstab/436/orig -> origin/gh/wconstab/436/orig 2025-08-14T21:24:12.8380451Z * [new branch] gh/wconstab/437/base -> origin/gh/wconstab/437/base 2025-08-14T21:24:12.8380594Z * [new branch] gh/wconstab/437/head -> origin/gh/wconstab/437/head 2025-08-14T21:24:12.8387012Z * [new branch] gh/wconstab/437/orig -> origin/gh/wconstab/437/orig 2025-08-14T21:24:12.8387352Z * [new branch] gh/wconstab/438/base -> origin/gh/wconstab/438/base 2025-08-14T21:24:12.8387514Z * [new branch] gh/wconstab/438/head -> origin/gh/wconstab/438/head 2025-08-14T21:24:12.8387647Z * [new branch] gh/wconstab/438/orig -> origin/gh/wconstab/438/orig 2025-08-14T21:24:12.8387915Z * [new branch] gh/wconstab/439/base -> origin/gh/wconstab/439/base 2025-08-14T21:24:12.8388219Z * [new branch] gh/wconstab/439/head -> origin/gh/wconstab/439/head 2025-08-14T21:24:12.8388536Z * [new branch] gh/wconstab/439/orig -> origin/gh/wconstab/439/orig 2025-08-14T21:24:12.8388668Z * [new branch] gh/wconstab/440/base -> origin/gh/wconstab/440/base 2025-08-14T21:24:12.8388799Z * [new branch] gh/wconstab/440/head -> origin/gh/wconstab/440/head 2025-08-14T21:24:12.8388937Z * [new branch] gh/wconstab/440/orig -> origin/gh/wconstab/440/orig 2025-08-14T21:24:12.8394979Z * [new branch] gh/wconstab/441/base -> origin/gh/wconstab/441/base 2025-08-14T21:24:12.8395327Z * [new branch] gh/wconstab/441/head -> origin/gh/wconstab/441/head 2025-08-14T21:24:12.8395498Z * [new branch] gh/wconstab/441/orig -> origin/gh/wconstab/441/orig 2025-08-14T21:24:12.8395673Z * [new branch] gh/wconstab/442/base -> origin/gh/wconstab/442/base 2025-08-14T21:24:12.8395973Z * [new branch] gh/wconstab/442/head -> origin/gh/wconstab/442/head 2025-08-14T21:24:12.8396151Z * [new branch] gh/wconstab/442/orig -> origin/gh/wconstab/442/orig 2025-08-14T21:24:12.8396388Z * [new branch] gh/weifengpy/27/base -> origin/gh/weifengpy/27/base 2025-08-14T21:24:12.8396593Z * [new branch] gh/weifengpy/27/head -> origin/gh/weifengpy/27/head 2025-08-14T21:24:12.8396746Z * [new branch] gh/weifengpy/27/orig -> origin/gh/weifengpy/27/orig 2025-08-14T21:24:12.8400649Z * [new branch] gh/weifengpy/30/base -> origin/gh/weifengpy/30/base 2025-08-14T21:24:12.8400996Z * [new branch] gh/weifengpy/30/head -> origin/gh/weifengpy/30/head 2025-08-14T21:24:12.8401177Z * [new branch] gh/weifengpy/30/orig -> origin/gh/weifengpy/30/orig 2025-08-14T21:24:12.8401481Z * [new branch] gh/weifengpy/31/base -> origin/gh/weifengpy/31/base 2025-08-14T21:24:12.8401757Z * [new branch] gh/weifengpy/31/head -> origin/gh/weifengpy/31/head 2025-08-14T21:24:12.8401933Z * [new branch] gh/weifengpy/31/orig -> origin/gh/weifengpy/31/orig 2025-08-14T21:24:12.8402067Z * [new branch] gh/weifengpy/32/base -> origin/gh/weifengpy/32/base 2025-08-14T21:24:12.8402204Z * [new branch] gh/weifengpy/32/head -> origin/gh/weifengpy/32/head 2025-08-14T21:24:12.8402343Z * [new branch] gh/weifengpy/32/orig -> origin/gh/weifengpy/32/orig 2025-08-14T21:24:12.8402473Z * [new branch] gh/weifengpy/33/base -> origin/gh/weifengpy/33/base 2025-08-14T21:24:12.8402605Z * [new branch] gh/weifengpy/33/head -> origin/gh/weifengpy/33/head 2025-08-14T21:24:12.8402740Z * [new branch] gh/weifengpy/33/orig -> origin/gh/weifengpy/33/orig 2025-08-14T21:24:12.8403343Z * [new branch] gh/williamwen42/196/base -> origin/gh/williamwen42/196/base 2025-08-14T21:24:12.8404146Z * [new branch] gh/williamwen42/196/head -> origin/gh/williamwen42/196/head 2025-08-14T21:24:12.8409978Z * [new branch] gh/williamwen42/196/orig -> origin/gh/williamwen42/196/orig 2025-08-14T21:24:12.8412816Z * [new branch] gh/williamwen42/209/base -> origin/gh/williamwen42/209/base 2025-08-14T21:24:12.8417779Z * [new branch] gh/williamwen42/209/head -> origin/gh/williamwen42/209/head 2025-08-14T21:24:12.8422292Z * [new branch] gh/williamwen42/209/orig -> origin/gh/williamwen42/209/orig 2025-08-14T21:24:12.8424323Z * [new branch] gh/williamwen42/250/base -> origin/gh/williamwen42/250/base 2025-08-14T21:24:12.8424591Z * [new branch] gh/williamwen42/250/head -> origin/gh/williamwen42/250/head 2025-08-14T21:24:12.8431099Z * [new branch] gh/williamwen42/250/orig -> origin/gh/williamwen42/250/orig 2025-08-14T21:24:12.8435383Z * [new branch] gh/williamwen42/252/base -> origin/gh/williamwen42/252/base 2025-08-14T21:24:12.8440869Z * [new branch] gh/williamwen42/252/head -> origin/gh/williamwen42/252/head 2025-08-14T21:24:12.8441202Z * [new branch] gh/williamwen42/252/orig -> origin/gh/williamwen42/252/orig 2025-08-14T21:24:12.8441368Z * [new branch] gh/williamwen42/256/base -> origin/gh/williamwen42/256/base 2025-08-14T21:24:12.8441627Z * [new branch] gh/williamwen42/256/head -> origin/gh/williamwen42/256/head 2025-08-14T21:24:12.8441818Z * [new branch] gh/williamwen42/256/orig -> origin/gh/williamwen42/256/orig 2025-08-14T21:24:12.8441984Z * [new branch] gh/williamwen42/258/base -> origin/gh/williamwen42/258/base 2025-08-14T21:24:12.8442702Z * [new branch] gh/williamwen42/258/head -> origin/gh/williamwen42/258/head 2025-08-14T21:24:12.8443396Z * [new branch] gh/williamwen42/258/orig -> origin/gh/williamwen42/258/orig 2025-08-14T21:24:12.8443591Z * [new branch] gh/williamwen42/260/base -> origin/gh/williamwen42/260/base 2025-08-14T21:24:12.8443963Z * [new branch] gh/williamwen42/260/head -> origin/gh/williamwen42/260/head 2025-08-14T21:24:12.8444135Z * [new branch] gh/williamwen42/260/orig -> origin/gh/williamwen42/260/orig 2025-08-14T21:24:12.8444323Z * [new branch] gh/williamwen42/261/base -> origin/gh/williamwen42/261/base 2025-08-14T21:24:12.8444492Z * [new branch] gh/williamwen42/261/head -> origin/gh/williamwen42/261/head 2025-08-14T21:24:12.8444651Z * [new branch] gh/williamwen42/261/orig -> origin/gh/williamwen42/261/orig 2025-08-14T21:24:12.8444803Z * [new branch] gh/williamwen42/262/base -> origin/gh/williamwen42/262/base 2025-08-14T21:24:12.8445150Z * [new branch] gh/williamwen42/262/head -> origin/gh/williamwen42/262/head 2025-08-14T21:24:12.8445332Z * [new branch] gh/williamwen42/262/orig -> origin/gh/williamwen42/262/orig 2025-08-14T21:24:12.8445482Z * [new branch] gh/williamwen42/263/base -> origin/gh/williamwen42/263/base 2025-08-14T21:24:12.8445730Z * [new branch] gh/williamwen42/263/head -> origin/gh/williamwen42/263/head 2025-08-14T21:24:12.8445890Z * [new branch] gh/williamwen42/263/orig -> origin/gh/williamwen42/263/orig 2025-08-14T21:24:12.8446036Z * [new branch] gh/williamwen42/264/base -> origin/gh/williamwen42/264/base 2025-08-14T21:24:12.8446190Z * [new branch] gh/williamwen42/264/head -> origin/gh/williamwen42/264/head 2025-08-14T21:24:12.8446337Z * [new branch] gh/williamwen42/264/orig -> origin/gh/williamwen42/264/orig 2025-08-14T21:24:12.8446482Z * [new branch] gh/williamwen42/265/base -> origin/gh/williamwen42/265/base 2025-08-14T21:24:12.8446639Z * [new branch] gh/williamwen42/265/head -> origin/gh/williamwen42/265/head 2025-08-14T21:24:12.8446788Z * [new branch] gh/williamwen42/265/orig -> origin/gh/williamwen42/265/orig 2025-08-14T21:24:12.8446939Z * [new branch] gh/williamwen42/266/base -> origin/gh/williamwen42/266/base 2025-08-14T21:24:12.8447085Z * [new branch] gh/williamwen42/266/head -> origin/gh/williamwen42/266/head 2025-08-14T21:24:12.8447229Z * [new branch] gh/williamwen42/266/orig -> origin/gh/williamwen42/266/orig 2025-08-14T21:24:12.8447380Z * [new branch] gh/williamwen42/267/base -> origin/gh/williamwen42/267/base 2025-08-14T21:24:12.8447526Z * [new branch] gh/williamwen42/267/head -> origin/gh/williamwen42/267/head 2025-08-14T21:24:12.8447678Z * [new branch] gh/williamwen42/267/orig -> origin/gh/williamwen42/267/orig 2025-08-14T21:24:12.8447827Z * [new branch] gh/williamwen42/268/base -> origin/gh/williamwen42/268/base 2025-08-14T21:24:12.8448053Z * [new branch] gh/williamwen42/268/head -> origin/gh/williamwen42/268/head 2025-08-14T21:24:12.8448207Z * [new branch] gh/williamwen42/268/orig -> origin/gh/williamwen42/268/orig 2025-08-14T21:24:12.8448351Z * [new branch] gh/williamwen42/269/base -> origin/gh/williamwen42/269/base 2025-08-14T21:24:12.8448495Z * [new branch] gh/williamwen42/269/head -> origin/gh/williamwen42/269/head 2025-08-14T21:24:12.8448648Z * [new branch] gh/williamwen42/269/orig -> origin/gh/williamwen42/269/orig 2025-08-14T21:24:12.8448793Z * [new branch] gh/williamwen42/270/base -> origin/gh/williamwen42/270/base 2025-08-14T21:24:12.8448947Z * [new branch] gh/williamwen42/270/head -> origin/gh/williamwen42/270/head 2025-08-14T21:24:12.8449093Z * [new branch] gh/williamwen42/270/orig -> origin/gh/williamwen42/270/orig 2025-08-14T21:24:12.8449241Z * [new branch] gh/williamwen42/271/base -> origin/gh/williamwen42/271/base 2025-08-14T21:24:12.8449404Z * [new branch] gh/williamwen42/271/head -> origin/gh/williamwen42/271/head 2025-08-14T21:24:12.8449551Z * [new branch] gh/williamwen42/271/orig -> origin/gh/williamwen42/271/orig 2025-08-14T21:24:12.8449706Z * [new branch] gh/williamwen42/272/base -> origin/gh/williamwen42/272/base 2025-08-14T21:24:12.8449853Z * [new branch] gh/williamwen42/272/head -> origin/gh/williamwen42/272/head 2025-08-14T21:24:12.8449999Z * [new branch] gh/williamwen42/272/orig -> origin/gh/williamwen42/272/orig 2025-08-14T21:24:12.8450165Z * [new branch] gh/williamwen42/273/base -> origin/gh/williamwen42/273/base 2025-08-14T21:24:12.8450309Z * [new branch] gh/williamwen42/273/head -> origin/gh/williamwen42/273/head 2025-08-14T21:24:12.8450517Z * [new branch] gh/williamwen42/273/orig -> origin/gh/williamwen42/273/orig 2025-08-14T21:24:12.8450667Z * [new branch] gh/williamwen42/274/base -> origin/gh/williamwen42/274/base 2025-08-14T21:24:12.8450814Z * [new branch] gh/williamwen42/274/head -> origin/gh/williamwen42/274/head 2025-08-14T21:24:12.8450966Z * [new branch] gh/williamwen42/274/orig -> origin/gh/williamwen42/274/orig 2025-08-14T21:24:12.8451295Z * [new branch] gh/williamwen42/275/base -> origin/gh/williamwen42/275/base 2025-08-14T21:24:12.8451467Z * [new branch] gh/williamwen42/275/head -> origin/gh/williamwen42/275/head 2025-08-14T21:24:12.8451889Z * [new branch] gh/williamwen42/276/base -> origin/gh/williamwen42/276/base 2025-08-14T21:24:12.8453795Z * [new branch] gh/williamwen42/276/head -> origin/gh/williamwen42/276/head 2025-08-14T21:24:12.8454155Z * [new branch] gh/williamwen42/276/orig -> origin/gh/williamwen42/276/orig 2025-08-14T21:24:12.8454415Z * [new branch] gh/williamwen42/277/base -> origin/gh/williamwen42/277/base 2025-08-14T21:24:12.8454809Z * [new branch] gh/williamwen42/277/head -> origin/gh/williamwen42/277/head 2025-08-14T21:24:12.8457512Z * [new branch] gh/williamwen42/277/orig -> origin/gh/williamwen42/277/orig 2025-08-14T21:24:12.8457691Z * [new branch] gh/williamwen42/278/base -> origin/gh/williamwen42/278/base 2025-08-14T21:24:12.8457835Z * [new branch] gh/williamwen42/278/head -> origin/gh/williamwen42/278/head 2025-08-14T21:24:12.8457988Z * [new branch] gh/williamwen42/278/orig -> origin/gh/williamwen42/278/orig 2025-08-14T21:24:12.8458611Z * [new branch] gh/williamwen42/279/base -> origin/gh/williamwen42/279/base 2025-08-14T21:24:12.8459291Z * [new branch] gh/williamwen42/279/head -> origin/gh/williamwen42/279/head 2025-08-14T21:24:12.8459780Z * [new branch] gh/williamwen42/279/orig -> origin/gh/williamwen42/279/orig 2025-08-14T21:24:12.8461555Z * [new branch] gh/xmfan/169/base -> origin/gh/xmfan/169/base 2025-08-14T21:24:12.8461936Z * [new branch] gh/xmfan/169/head -> origin/gh/xmfan/169/head 2025-08-14T21:24:12.8462321Z * [new branch] gh/xmfan/170/base -> origin/gh/xmfan/170/base 2025-08-14T21:24:12.8462865Z * [new branch] gh/xmfan/170/head -> origin/gh/xmfan/170/head 2025-08-14T21:24:12.8464367Z * [new branch] gh/xmfan/18/base -> origin/gh/xmfan/18/base 2025-08-14T21:24:12.8466410Z * [new branch] gh/xmfan/18/head -> origin/gh/xmfan/18/head 2025-08-14T21:24:12.8466797Z * [new branch] gh/xmfan/228/base -> origin/gh/xmfan/228/base 2025-08-14T21:24:12.8467125Z * [new branch] gh/xmfan/228/head -> origin/gh/xmfan/228/head 2025-08-14T21:24:12.8467477Z * [new branch] gh/xmfan/228/orig -> origin/gh/xmfan/228/orig 2025-08-14T21:24:12.8468001Z * [new branch] gh/xmfan/229/base -> origin/gh/xmfan/229/base 2025-08-14T21:24:12.8468508Z * [new branch] gh/xmfan/229/head -> origin/gh/xmfan/229/head 2025-08-14T21:24:12.8469266Z * [new branch] gh/xmfan/229/orig -> origin/gh/xmfan/229/orig 2025-08-14T21:24:12.8470273Z * [new branch] gh/xmfan/237/base -> origin/gh/xmfan/237/base 2025-08-14T21:24:12.8474636Z * [new branch] gh/xmfan/237/head -> origin/gh/xmfan/237/head 2025-08-14T21:24:12.8475151Z * [new branch] gh/xmfan/237/orig -> origin/gh/xmfan/237/orig 2025-08-14T21:24:12.8475595Z * [new branch] gh/xmfan/244/base -> origin/gh/xmfan/244/base 2025-08-14T21:24:12.8476024Z * [new branch] gh/xmfan/244/head -> origin/gh/xmfan/244/head 2025-08-14T21:24:12.8476909Z * [new branch] gh/xmfan/244/orig -> origin/gh/xmfan/244/orig 2025-08-14T21:24:12.8477310Z * [new branch] gh/xmfan/246/base -> origin/gh/xmfan/246/base 2025-08-14T21:24:12.8477619Z * [new branch] gh/xmfan/246/head -> origin/gh/xmfan/246/head 2025-08-14T21:24:12.8477927Z * [new branch] gh/xmfan/246/orig -> origin/gh/xmfan/246/orig 2025-08-14T21:24:12.8478232Z * [new branch] gh/xmfan/253/base -> origin/gh/xmfan/253/base 2025-08-14T21:24:12.8478530Z * [new branch] gh/xmfan/253/head -> origin/gh/xmfan/253/head 2025-08-14T21:24:12.8479003Z * [new branch] gh/xmfan/253/orig -> origin/gh/xmfan/253/orig 2025-08-14T21:24:12.8479446Z * [new branch] gh/xmfan/254/base -> origin/gh/xmfan/254/base 2025-08-14T21:24:12.8479769Z * [new branch] gh/xmfan/254/head -> origin/gh/xmfan/254/head 2025-08-14T21:24:12.8480147Z * [new branch] gh/xmfan/254/orig -> origin/gh/xmfan/254/orig 2025-08-14T21:24:12.8482177Z * [new branch] gh/xmfan/260/base -> origin/gh/xmfan/260/base 2025-08-14T21:24:12.8482560Z * [new branch] gh/xmfan/260/head -> origin/gh/xmfan/260/head 2025-08-14T21:24:12.8482879Z * [new branch] gh/xmfan/260/orig -> origin/gh/xmfan/260/orig 2025-08-14T21:24:12.8483226Z * [new branch] gh/xmfan/262/base -> origin/gh/xmfan/262/base 2025-08-14T21:24:12.8483870Z * [new branch] gh/xmfan/262/head -> origin/gh/xmfan/262/head 2025-08-14T21:24:12.8484468Z * [new branch] gh/xmfan/262/orig -> origin/gh/xmfan/262/orig 2025-08-14T21:24:12.8485387Z * [new branch] gh/xmfan/263/base -> origin/gh/xmfan/263/base 2025-08-14T21:24:12.8492492Z * [new branch] gh/xmfan/263/head -> origin/gh/xmfan/263/head 2025-08-14T21:24:12.8492881Z * [new branch] gh/xmfan/263/orig -> origin/gh/xmfan/263/orig 2025-08-14T21:24:12.8493364Z * [new branch] gh/xmfan/264/base -> origin/gh/xmfan/264/base 2025-08-14T21:24:12.8493672Z * [new branch] gh/xmfan/264/head -> origin/gh/xmfan/264/head 2025-08-14T21:24:12.8493968Z * [new branch] gh/xmfan/264/orig -> origin/gh/xmfan/264/orig 2025-08-14T21:24:12.8494270Z * [new branch] gh/xmfan/268/base -> origin/gh/xmfan/268/base 2025-08-14T21:24:12.8494573Z * [new branch] gh/xmfan/268/head -> origin/gh/xmfan/268/head 2025-08-14T21:24:12.8494872Z * [new branch] gh/xmfan/268/orig -> origin/gh/xmfan/268/orig 2025-08-14T21:24:12.8495169Z * [new branch] gh/xmfan/269/base -> origin/gh/xmfan/269/base 2025-08-14T21:24:12.8500779Z * [new branch] gh/xmfan/269/head -> origin/gh/xmfan/269/head 2025-08-14T21:24:12.8501172Z * [new branch] gh/xmfan/269/orig -> origin/gh/xmfan/269/orig 2025-08-14T21:24:12.8501500Z * [new branch] gh/xmfan/270/base -> origin/gh/xmfan/270/base 2025-08-14T21:24:12.8501800Z * [new branch] gh/xmfan/270/head -> origin/gh/xmfan/270/head 2025-08-14T21:24:12.8502112Z * [new branch] gh/xmfan/270/orig -> origin/gh/xmfan/270/orig 2025-08-14T21:24:12.8502422Z * [new branch] gh/xmfan/271/base -> origin/gh/xmfan/271/base 2025-08-14T21:24:12.8502734Z * [new branch] gh/xmfan/271/head -> origin/gh/xmfan/271/head 2025-08-14T21:24:12.8503035Z * [new branch] gh/xmfan/271/orig -> origin/gh/xmfan/271/orig 2025-08-14T21:24:12.8503352Z * [new branch] gh/xmfan/272/base -> origin/gh/xmfan/272/base 2025-08-14T21:24:12.8503657Z * [new branch] gh/xmfan/272/head -> origin/gh/xmfan/272/head 2025-08-14T21:24:12.8504098Z * [new branch] gh/xmfan/272/orig -> origin/gh/xmfan/272/orig 2025-08-14T21:24:12.8504412Z * [new branch] gh/xmfan/273/base -> origin/gh/xmfan/273/base 2025-08-14T21:24:12.8504722Z * [new branch] gh/xmfan/273/head -> origin/gh/xmfan/273/head 2025-08-14T21:24:12.8505074Z * [new branch] gh/xmfan/273/orig -> origin/gh/xmfan/273/orig 2025-08-14T21:24:12.8506737Z * [new branch] gh/xmfan/274/base -> origin/gh/xmfan/274/base 2025-08-14T21:24:12.8507060Z * [new branch] gh/xmfan/274/head -> origin/gh/xmfan/274/head 2025-08-14T21:24:12.8507386Z * [new branch] gh/xmfan/274/orig -> origin/gh/xmfan/274/orig 2025-08-14T21:24:12.8508830Z * [new branch] gh/xmfan/275/base -> origin/gh/xmfan/275/base 2025-08-14T21:24:12.8509157Z * [new branch] gh/xmfan/275/head -> origin/gh/xmfan/275/head 2025-08-14T21:24:12.8509488Z * [new branch] gh/xmfan/275/orig -> origin/gh/xmfan/275/orig 2025-08-14T21:24:12.8514390Z * [new branch] gh/xmfan/276/base -> origin/gh/xmfan/276/base 2025-08-14T21:24:12.8515045Z * [new branch] gh/xmfan/276/head -> origin/gh/xmfan/276/head 2025-08-14T21:24:12.8515397Z * [new branch] gh/xmfan/276/orig -> origin/gh/xmfan/276/orig 2025-08-14T21:24:12.8515714Z * [new branch] gh/xmfan/277/base -> origin/gh/xmfan/277/base 2025-08-14T21:24:12.8516025Z * [new branch] gh/xmfan/277/head -> origin/gh/xmfan/277/head 2025-08-14T21:24:12.8516326Z * [new branch] gh/xmfan/277/orig -> origin/gh/xmfan/277/orig 2025-08-14T21:24:12.8516684Z * [new branch] gh/xuanzhang816/12/base -> origin/gh/xuanzhang816/12/base 2025-08-14T21:24:12.8517207Z * [new branch] gh/xuanzhang816/12/head -> origin/gh/xuanzhang816/12/head 2025-08-14T21:24:12.8517567Z * [new branch] gh/xuanzhang816/12/orig -> origin/gh/xuanzhang816/12/orig 2025-08-14T21:24:12.8518038Z * [new branch] gh/xuanzhang816/14/base -> origin/gh/xuanzhang816/14/base 2025-08-14T21:24:12.8518378Z * [new branch] gh/xuanzhang816/14/head -> origin/gh/xuanzhang816/14/head 2025-08-14T21:24:12.8518739Z * [new branch] gh/xuanzhang816/14/orig -> origin/gh/xuanzhang816/14/orig 2025-08-14T21:24:12.8519949Z * [new branch] gh/xuanzhang816/18/base -> origin/gh/xuanzhang816/18/base 2025-08-14T21:24:12.8520748Z * [new branch] gh/xuanzhang816/18/head -> origin/gh/xuanzhang816/18/head 2025-08-14T21:24:12.8521388Z * [new branch] gh/xuanzhang816/18/orig -> origin/gh/xuanzhang816/18/orig 2025-08-14T21:24:12.8523014Z * [new branch] gh/xuanzhang816/19/base -> origin/gh/xuanzhang816/19/base 2025-08-14T21:24:12.8523407Z * [new branch] gh/xuanzhang816/19/head -> origin/gh/xuanzhang816/19/head 2025-08-14T21:24:12.8523804Z * [new branch] gh/xuanzhang816/19/orig -> origin/gh/xuanzhang816/19/orig 2025-08-14T21:24:12.8524978Z * [new branch] gh/xuanzhang816/20/base -> origin/gh/xuanzhang816/20/base 2025-08-14T21:24:12.8525743Z * [new branch] gh/xuanzhang816/20/head -> origin/gh/xuanzhang816/20/head 2025-08-14T21:24:12.8529732Z * [new branch] gh/xuanzhang816/20/orig -> origin/gh/xuanzhang816/20/orig 2025-08-14T21:24:12.8530126Z * [new branch] gh/xuanzhang816/21/base -> origin/gh/xuanzhang816/21/base 2025-08-14T21:24:12.8530521Z * [new branch] gh/xuanzhang816/21/head -> origin/gh/xuanzhang816/21/head 2025-08-14T21:24:12.8530884Z * [new branch] gh/xuanzhang816/21/orig -> origin/gh/xuanzhang816/21/orig 2025-08-14T21:24:12.8531264Z * [new branch] gh/xuanzhang816/22/base -> origin/gh/xuanzhang816/22/base 2025-08-14T21:24:12.8531884Z * [new branch] gh/xuanzhang816/22/head -> origin/gh/xuanzhang816/22/head 2025-08-14T21:24:12.8535549Z * [new branch] gh/xuanzhang816/22/orig -> origin/gh/xuanzhang816/22/orig 2025-08-14T21:24:12.8536034Z * [new branch] gh/xuanzhang816/23/base -> origin/gh/xuanzhang816/23/base 2025-08-14T21:24:12.8542060Z * [new branch] gh/xuanzhang816/23/head -> origin/gh/xuanzhang816/23/head 2025-08-14T21:24:12.8542511Z * [new branch] gh/xuanzhang816/23/orig -> origin/gh/xuanzhang816/23/orig 2025-08-14T21:24:12.8542903Z * [new branch] gh/xuanzhang816/24/base -> origin/gh/xuanzhang816/24/base 2025-08-14T21:24:12.8543293Z * [new branch] gh/xuanzhang816/24/head -> origin/gh/xuanzhang816/24/head 2025-08-14T21:24:12.8543665Z * [new branch] gh/xuanzhang816/24/orig -> origin/gh/xuanzhang816/24/orig 2025-08-14T21:24:12.8544062Z * [new branch] gh/yanbing-j/11/base -> origin/gh/yanbing-j/11/base 2025-08-14T21:24:12.8544484Z * [new branch] gh/yanbing-j/11/head -> origin/gh/yanbing-j/11/head 2025-08-14T21:24:12.8544841Z * [new branch] gh/yanbing-j/11/orig -> origin/gh/yanbing-j/11/orig 2025-08-14T21:24:12.8545201Z * [new branch] gh/yanbing-j/12/base -> origin/gh/yanbing-j/12/base 2025-08-14T21:24:12.8545549Z * [new branch] gh/yanbing-j/12/head -> origin/gh/yanbing-j/12/head 2025-08-14T21:24:12.8545905Z * [new branch] gh/yanbing-j/12/orig -> origin/gh/yanbing-j/12/orig 2025-08-14T21:24:12.8546427Z * [new branch] gh/yanbing-j/13/base -> origin/gh/yanbing-j/13/base 2025-08-14T21:24:12.8546906Z * [new branch] gh/yanbing-j/13/head -> origin/gh/yanbing-j/13/head 2025-08-14T21:24:12.8547257Z * [new branch] gh/yanbing-j/13/orig -> origin/gh/yanbing-j/13/orig 2025-08-14T21:24:12.8548092Z * [new branch] gh/yanbing-j/14/base -> origin/gh/yanbing-j/14/base 2025-08-14T21:24:12.8548841Z * [new branch] gh/yanbing-j/14/head -> origin/gh/yanbing-j/14/head 2025-08-14T21:24:12.8549400Z * [new branch] gh/yanbing-j/14/orig -> origin/gh/yanbing-j/14/orig 2025-08-14T21:24:12.8551240Z * [new branch] gh/yanbing-j/15/base -> origin/gh/yanbing-j/15/base 2025-08-14T21:24:12.8551654Z * [new branch] gh/yanbing-j/15/head -> origin/gh/yanbing-j/15/head 2025-08-14T21:24:12.8552020Z * [new branch] gh/yanbing-j/15/orig -> origin/gh/yanbing-j/15/orig 2025-08-14T21:24:12.8552696Z * [new branch] gh/yanbing-j/18/base -> origin/gh/yanbing-j/18/base 2025-08-14T21:24:12.8553269Z * [new branch] gh/yanbing-j/18/head -> origin/gh/yanbing-j/18/head 2025-08-14T21:24:12.8553952Z * [new branch] gh/yanbing-j/18/orig -> origin/gh/yanbing-j/18/orig 2025-08-14T21:24:12.8555733Z * [new branch] gh/yanbing-j/19/base -> origin/gh/yanbing-j/19/base 2025-08-14T21:24:12.8556323Z * [new branch] gh/yanbing-j/19/head -> origin/gh/yanbing-j/19/head 2025-08-14T21:24:12.8556824Z * [new branch] gh/yanbing-j/19/orig -> origin/gh/yanbing-j/19/orig 2025-08-14T21:24:12.8557379Z * [new branch] gh/yanbing-j/20/base -> origin/gh/yanbing-j/20/base 2025-08-14T21:24:12.8557949Z * [new branch] gh/yanbing-j/20/head -> origin/gh/yanbing-j/20/head 2025-08-14T21:24:12.8558752Z * [new branch] gh/yanbing-j/20/orig -> origin/gh/yanbing-j/20/orig 2025-08-14T21:24:12.8559853Z * [new branch] gh/yanbing-j/21/base -> origin/gh/yanbing-j/21/base 2025-08-14T21:24:12.8560215Z * [new branch] gh/yanbing-j/21/head -> origin/gh/yanbing-j/21/head 2025-08-14T21:24:12.8564633Z * [new branch] gh/yanbing-j/22/base -> origin/gh/yanbing-j/22/base 2025-08-14T21:24:12.8565298Z * [new branch] gh/yanbing-j/22/head -> origin/gh/yanbing-j/22/head 2025-08-14T21:24:12.8565917Z * [new branch] gh/yanbing-j/22/orig -> origin/gh/yanbing-j/22/orig 2025-08-14T21:24:12.8566292Z * [new branch] gh/yanbing-j/23/base -> origin/gh/yanbing-j/23/base 2025-08-14T21:24:12.8566652Z * [new branch] gh/yanbing-j/23/head -> origin/gh/yanbing-j/23/head 2025-08-14T21:24:12.8566998Z * [new branch] gh/yanbing-j/23/orig -> origin/gh/yanbing-j/23/orig 2025-08-14T21:24:12.8567350Z * [new branch] gh/yanbing-j/24/base -> origin/gh/yanbing-j/24/base 2025-08-14T21:24:12.8567860Z * [new branch] gh/yanbing-j/24/head -> origin/gh/yanbing-j/24/head 2025-08-14T21:24:12.8568230Z * [new branch] gh/yanbing-j/24/orig -> origin/gh/yanbing-j/24/orig 2025-08-14T21:24:12.8568710Z * [new branch] gh/yanbing-j/25/base -> origin/gh/yanbing-j/25/base 2025-08-14T21:24:12.8569166Z * [new branch] gh/yanbing-j/25/head -> origin/gh/yanbing-j/25/head 2025-08-14T21:24:12.8569635Z * [new branch] gh/yanbing-j/25/orig -> origin/gh/yanbing-j/25/orig 2025-08-14T21:24:12.8570438Z * [new branch] gh/yanbing-j/26/base -> origin/gh/yanbing-j/26/base 2025-08-14T21:24:12.8571054Z * [new branch] gh/yanbing-j/26/head -> origin/gh/yanbing-j/26/head 2025-08-14T21:24:12.8571667Z * [new branch] gh/yanbing-j/26/orig -> origin/gh/yanbing-j/26/orig 2025-08-14T21:24:12.8575316Z * [new branch] gh/yanbing-j/36/base -> origin/gh/yanbing-j/36/base 2025-08-14T21:24:12.8575912Z * [new branch] gh/yanbing-j/36/head -> origin/gh/yanbing-j/36/head 2025-08-14T21:24:12.8576402Z * [new branch] gh/yanbing-j/36/orig -> origin/gh/yanbing-j/36/orig 2025-08-14T21:24:12.8577280Z * [new branch] gh/yanbing-j/37/base -> origin/gh/yanbing-j/37/base 2025-08-14T21:24:12.8577957Z * [new branch] gh/yanbing-j/37/head -> origin/gh/yanbing-j/37/head 2025-08-14T21:24:12.8578319Z * [new branch] gh/yanbing-j/37/orig -> origin/gh/yanbing-j/37/orig 2025-08-14T21:24:12.8578667Z * [new branch] gh/yanbing-j/39/base -> origin/gh/yanbing-j/39/base 2025-08-14T21:24:12.8579016Z * [new branch] gh/yanbing-j/39/head -> origin/gh/yanbing-j/39/head 2025-08-14T21:24:12.8579363Z * [new branch] gh/yanbing-j/39/orig -> origin/gh/yanbing-j/39/orig 2025-08-14T21:24:12.8580200Z * [new branch] gh/yangw-dev/1/base -> origin/gh/yangw-dev/1/base 2025-08-14T21:24:12.8581204Z * [new branch] gh/yangw-dev/10/base -> origin/gh/yangw-dev/10/base 2025-08-14T21:24:12.8581736Z * [new branch] gh/yangw-dev/10/head -> origin/gh/yangw-dev/10/head 2025-08-14T21:24:12.8582425Z * [new branch] gh/yangw-dev/10/orig -> origin/gh/yangw-dev/10/orig 2025-08-14T21:24:12.8587674Z * [new branch] gh/yangw-dev/11/base -> origin/gh/yangw-dev/11/base 2025-08-14T21:24:12.8588098Z * [new branch] gh/yangw-dev/11/head -> origin/gh/yangw-dev/11/head 2025-08-14T21:24:12.8588447Z * [new branch] gh/yangw-dev/11/orig -> origin/gh/yangw-dev/11/orig 2025-08-14T21:24:12.8588791Z * [new branch] gh/yangw-dev/12/base -> origin/gh/yangw-dev/12/base 2025-08-14T21:24:12.8589129Z * [new branch] gh/yangw-dev/12/head -> origin/gh/yangw-dev/12/head 2025-08-14T21:24:12.8589462Z * [new branch] gh/yangw-dev/12/orig -> origin/gh/yangw-dev/12/orig 2025-08-14T21:24:12.8589800Z * [new branch] gh/yangw-dev/13/base -> origin/gh/yangw-dev/13/base 2025-08-14T21:24:12.8590131Z * [new branch] gh/yangw-dev/13/head -> origin/gh/yangw-dev/13/head 2025-08-14T21:24:12.8590611Z * [new branch] gh/yangw-dev/13/orig -> origin/gh/yangw-dev/13/orig 2025-08-14T21:24:12.8595395Z * [new branch] gh/yangw-dev/14/base -> origin/gh/yangw-dev/14/base 2025-08-14T21:24:12.8595831Z * [new branch] gh/yangw-dev/14/head -> origin/gh/yangw-dev/14/head 2025-08-14T21:24:12.8596164Z * [new branch] gh/yangw-dev/14/orig -> origin/gh/yangw-dev/14/orig 2025-08-14T21:24:12.8596491Z * [new branch] gh/yangw-dev/15/base -> origin/gh/yangw-dev/15/base 2025-08-14T21:24:12.8596825Z * [new branch] gh/yangw-dev/15/head -> origin/gh/yangw-dev/15/head 2025-08-14T21:24:12.8597167Z * [new branch] gh/yangw-dev/15/orig -> origin/gh/yangw-dev/15/orig 2025-08-14T21:24:12.8597481Z * [new branch] gh/yangw-dev/16/base -> origin/gh/yangw-dev/16/base 2025-08-14T21:24:12.8597801Z * [new branch] gh/yangw-dev/16/head -> origin/gh/yangw-dev/16/head 2025-08-14T21:24:12.8598132Z * [new branch] gh/yangw-dev/16/orig -> origin/gh/yangw-dev/16/orig 2025-08-14T21:24:12.8598447Z * [new branch] gh/yangw-dev/17/base -> origin/gh/yangw-dev/17/base 2025-08-14T21:24:12.8599038Z * [new branch] gh/yangw-dev/17/head -> origin/gh/yangw-dev/17/head 2025-08-14T21:24:12.8599402Z * [new branch] gh/yangw-dev/17/orig -> origin/gh/yangw-dev/17/orig 2025-08-14T21:24:12.8599974Z * [new branch] gh/yangw-dev/18/base -> origin/gh/yangw-dev/18/base 2025-08-14T21:24:12.8600321Z * [new branch] gh/yangw-dev/18/head -> origin/gh/yangw-dev/18/head 2025-08-14T21:24:12.8601005Z * [new branch] gh/yangw-dev/18/orig -> origin/gh/yangw-dev/18/orig 2025-08-14T21:24:12.8601927Z * [new branch] gh/yangw-dev/19/base -> origin/gh/yangw-dev/19/base 2025-08-14T21:24:12.8602472Z * [new branch] gh/yangw-dev/19/head -> origin/gh/yangw-dev/19/head 2025-08-14T21:24:12.8603248Z * [new branch] gh/yangw-dev/19/orig -> origin/gh/yangw-dev/19/orig 2025-08-14T21:24:12.8604318Z * [new branch] gh/yangw-dev/2/base -> origin/gh/yangw-dev/2/base 2025-08-14T21:24:12.8604780Z * [new branch] gh/yangw-dev/2/head -> origin/gh/yangw-dev/2/head 2025-08-14T21:24:12.8606452Z * [new branch] gh/yangw-dev/3/base -> origin/gh/yangw-dev/3/base 2025-08-14T21:24:12.8606956Z * [new branch] gh/yangw-dev/3/head -> origin/gh/yangw-dev/3/head 2025-08-14T21:24:12.8607453Z * [new branch] gh/yangw-dev/4/base -> origin/gh/yangw-dev/4/base 2025-08-14T21:24:12.8607944Z * [new branch] gh/yangw-dev/4/head -> origin/gh/yangw-dev/4/head 2025-08-14T21:24:12.8609151Z * [new branch] gh/yangw-dev/5/base -> origin/gh/yangw-dev/5/base 2025-08-14T21:24:12.8609574Z * [new branch] gh/yangw-dev/5/head -> origin/gh/yangw-dev/5/head 2025-08-14T21:24:12.8609973Z * [new branch] gh/yangw-dev/6/base -> origin/gh/yangw-dev/6/base 2025-08-14T21:24:12.8610552Z * [new branch] gh/yangw-dev/6/head -> origin/gh/yangw-dev/6/head 2025-08-14T21:24:12.8612197Z * [new branch] gh/yangw-dev/7/base -> origin/gh/yangw-dev/7/base 2025-08-14T21:24:12.8612603Z * [new branch] gh/yangw-dev/7/head -> origin/gh/yangw-dev/7/head 2025-08-14T21:24:12.8613379Z * [new branch] gh/yangw-dev/8/base -> origin/gh/yangw-dev/8/base 2025-08-14T21:24:12.8613816Z * [new branch] gh/yangw-dev/8/head -> origin/gh/yangw-dev/8/head 2025-08-14T21:24:12.8614509Z * [new branch] gh/yangw-dev/8/orig -> origin/gh/yangw-dev/8/orig 2025-08-14T21:24:12.8617576Z * [new branch] gh/yangw-dev/9/base -> origin/gh/yangw-dev/9/base 2025-08-14T21:24:12.8618132Z * [new branch] gh/yangw-dev/9/head -> origin/gh/yangw-dev/9/head 2025-08-14T21:24:12.8618509Z * [new branch] gh/yangw-dev/9/orig -> origin/gh/yangw-dev/9/orig 2025-08-14T21:24:12.8618845Z * [new branch] gh/ydwu4/233/base -> origin/gh/ydwu4/233/base 2025-08-14T21:24:12.8619170Z * [new branch] gh/ydwu4/233/head -> origin/gh/ydwu4/233/head 2025-08-14T21:24:12.8619476Z * [new branch] gh/ydwu4/233/orig -> origin/gh/ydwu4/233/orig 2025-08-14T21:24:12.8620577Z * [new branch] gh/ydwu4/246/base -> origin/gh/ydwu4/246/base 2025-08-14T21:24:12.8620894Z * [new branch] gh/ydwu4/246/head -> origin/gh/ydwu4/246/head 2025-08-14T21:24:12.8621924Z * [new branch] gh/ydwu4/246/orig -> origin/gh/ydwu4/246/orig 2025-08-14T21:24:12.8622896Z * [new branch] gh/ydwu4/253/base -> origin/gh/ydwu4/253/base 2025-08-14T21:24:12.8623495Z * [new branch] gh/ydwu4/253/head -> origin/gh/ydwu4/253/head 2025-08-14T21:24:12.8624099Z * [new branch] gh/ydwu4/253/orig -> origin/gh/ydwu4/253/orig 2025-08-14T21:24:12.8625228Z * [new branch] gh/ydwu4/255/base -> origin/gh/ydwu4/255/base 2025-08-14T21:24:12.8625650Z * [new branch] gh/ydwu4/255/head -> origin/gh/ydwu4/255/head 2025-08-14T21:24:12.8626185Z * [new branch] gh/ydwu4/255/orig -> origin/gh/ydwu4/255/orig 2025-08-14T21:24:12.8630037Z * [new branch] gh/ydwu4/259/base -> origin/gh/ydwu4/259/base 2025-08-14T21:24:12.8630414Z * [new branch] gh/ydwu4/259/head -> origin/gh/ydwu4/259/head 2025-08-14T21:24:12.8630744Z * [new branch] gh/ydwu4/259/orig -> origin/gh/ydwu4/259/orig 2025-08-14T21:24:12.8631061Z * [new branch] gh/ydwu4/262/base -> origin/gh/ydwu4/262/base 2025-08-14T21:24:12.8631381Z * [new branch] gh/ydwu4/262/head -> origin/gh/ydwu4/262/head 2025-08-14T21:24:12.8631849Z * [new branch] gh/ydwu4/262/orig -> origin/gh/ydwu4/262/orig 2025-08-14T21:24:12.8632691Z * [new branch] gh/ydwu4/263/base -> origin/gh/ydwu4/263/base 2025-08-14T21:24:12.8633119Z * [new branch] gh/ydwu4/263/head -> origin/gh/ydwu4/263/head 2025-08-14T21:24:12.8633492Z * [new branch] gh/ydwu4/263/orig -> origin/gh/ydwu4/263/orig 2025-08-14T21:24:12.8634586Z * [new branch] gh/ydwu4/269/base -> origin/gh/ydwu4/269/base 2025-08-14T21:24:12.8634965Z * [new branch] gh/ydwu4/269/head -> origin/gh/ydwu4/269/head 2025-08-14T21:24:12.8635578Z * [new branch] gh/ydwu4/269/orig -> origin/gh/ydwu4/269/orig 2025-08-14T21:24:12.8637855Z * [new branch] gh/ydwu4/270/base -> origin/gh/ydwu4/270/base 2025-08-14T21:24:12.8638267Z * [new branch] gh/ydwu4/270/head -> origin/gh/ydwu4/270/head 2025-08-14T21:24:12.8638588Z * [new branch] gh/ydwu4/270/orig -> origin/gh/ydwu4/270/orig 2025-08-14T21:24:12.8638983Z * [new branch] gh/ydwu4/272/base -> origin/gh/ydwu4/272/base 2025-08-14T21:24:12.8639758Z * [new branch] gh/ydwu4/272/head -> origin/gh/ydwu4/272/head 2025-08-14T21:24:12.8640421Z * [new branch] gh/ydwu4/272/orig -> origin/gh/ydwu4/272/orig 2025-08-14T21:24:12.8641245Z * [new branch] gh/ydwu4/275/base -> origin/gh/ydwu4/275/base 2025-08-14T21:24:12.8641886Z * [new branch] gh/ydwu4/275/head -> origin/gh/ydwu4/275/head 2025-08-14T21:24:12.8642579Z * [new branch] gh/ydwu4/275/orig -> origin/gh/ydwu4/275/orig 2025-08-14T21:24:12.8643559Z * [new branch] gh/ydwu4/276/base -> origin/gh/ydwu4/276/base 2025-08-14T21:24:12.8644194Z * [new branch] gh/ydwu4/276/head -> origin/gh/ydwu4/276/head 2025-08-14T21:24:12.8644702Z * [new branch] gh/ydwu4/276/orig -> origin/gh/ydwu4/276/orig 2025-08-14T21:24:12.8645978Z * [new branch] gh/ydwu4/277/base -> origin/gh/ydwu4/277/base 2025-08-14T21:24:12.8646361Z * [new branch] gh/ydwu4/277/head -> origin/gh/ydwu4/277/head 2025-08-14T21:24:12.8647064Z * [new branch] gh/ydwu4/277/orig -> origin/gh/ydwu4/277/orig 2025-08-14T21:24:12.8650890Z * [new branch] gh/ydwu4/278/base -> origin/gh/ydwu4/278/base 2025-08-14T21:24:12.8651265Z * [new branch] gh/ydwu4/278/head -> origin/gh/ydwu4/278/head 2025-08-14T21:24:12.8651574Z * [new branch] gh/ydwu4/278/orig -> origin/gh/ydwu4/278/orig 2025-08-14T21:24:12.8651885Z * [new branch] gh/ydwu4/279/base -> origin/gh/ydwu4/279/base 2025-08-14T21:24:12.8652385Z * [new branch] gh/ydwu4/279/head -> origin/gh/ydwu4/279/head 2025-08-14T21:24:12.8653318Z * [new branch] gh/ydwu4/279/orig -> origin/gh/ydwu4/279/orig 2025-08-14T21:24:12.8653678Z * [new branch] gh/ydwu4/280/base -> origin/gh/ydwu4/280/base 2025-08-14T21:24:12.8654156Z * [new branch] gh/ydwu4/280/head -> origin/gh/ydwu4/280/head 2025-08-14T21:24:12.8654593Z * [new branch] gh/ydwu4/280/orig -> origin/gh/ydwu4/280/orig 2025-08-14T21:24:12.8656668Z * [new branch] gh/ydwu4/281/base -> origin/gh/ydwu4/281/base 2025-08-14T21:24:12.8657183Z * [new branch] gh/ydwu4/281/head -> origin/gh/ydwu4/281/head 2025-08-14T21:24:12.8657623Z * [new branch] gh/ydwu4/281/orig -> origin/gh/ydwu4/281/orig 2025-08-14T21:24:12.8658112Z * [new branch] gh/ydwu4/282/base -> origin/gh/ydwu4/282/base 2025-08-14T21:24:12.8659550Z * [new branch] gh/ydwu4/282/head -> origin/gh/ydwu4/282/head 2025-08-14T21:24:12.8660283Z * [new branch] gh/ydwu4/282/orig -> origin/gh/ydwu4/282/orig 2025-08-14T21:24:12.8660721Z * [new branch] gh/ydwu4/283/base -> origin/gh/ydwu4/283/base 2025-08-14T21:24:12.8661161Z * [new branch] gh/ydwu4/283/head -> origin/gh/ydwu4/283/head 2025-08-14T21:24:12.8661560Z * [new branch] gh/ydwu4/283/orig -> origin/gh/ydwu4/283/orig 2025-08-14T21:24:12.8663753Z * [new branch] gh/ydwu4/284/base -> origin/gh/ydwu4/284/base 2025-08-14T21:24:12.8664283Z * [new branch] gh/ydwu4/284/head -> origin/gh/ydwu4/284/head 2025-08-14T21:24:12.8664720Z * [new branch] gh/ydwu4/284/orig -> origin/gh/ydwu4/284/orig 2025-08-14T21:24:12.8665167Z * [new branch] gh/ydwu4/285/base -> origin/gh/ydwu4/285/base 2025-08-14T21:24:12.8665555Z * [new branch] gh/ydwu4/285/head -> origin/gh/ydwu4/285/head 2025-08-14T21:24:12.8665991Z * [new branch] gh/ydwu4/285/orig -> origin/gh/ydwu4/285/orig 2025-08-14T21:24:12.8667442Z * [new branch] gh/ydwu4/286/base -> origin/gh/ydwu4/286/base 2025-08-14T21:24:12.8667932Z * [new branch] gh/ydwu4/286/head -> origin/gh/ydwu4/286/head 2025-08-14T21:24:12.8668366Z * [new branch] gh/ydwu4/286/orig -> origin/gh/ydwu4/286/orig 2025-08-14T21:24:12.8668811Z * [new branch] gh/ydwu4/287/base -> origin/gh/ydwu4/287/base 2025-08-14T21:24:12.8669439Z * [new branch] gh/ydwu4/287/head -> origin/gh/ydwu4/287/head 2025-08-14T21:24:12.8670257Z * [new branch] gh/ydwu4/287/orig -> origin/gh/ydwu4/287/orig 2025-08-14T21:24:12.8671657Z * [new branch] gh/ydwu4/288/base -> origin/gh/ydwu4/288/base 2025-08-14T21:24:12.8672233Z * [new branch] gh/ydwu4/288/head -> origin/gh/ydwu4/288/head 2025-08-14T21:24:12.8672637Z * [new branch] gh/ydwu4/288/orig -> origin/gh/ydwu4/288/orig 2025-08-14T21:24:12.8674894Z * [new branch] gh/ydwu4/289/base -> origin/gh/ydwu4/289/base 2025-08-14T21:24:12.8675401Z * [new branch] gh/ydwu4/289/head -> origin/gh/ydwu4/289/head 2025-08-14T21:24:12.8675821Z * [new branch] gh/ydwu4/289/orig -> origin/gh/ydwu4/289/orig 2025-08-14T21:24:12.8676246Z * [new branch] gh/ydwu4/290/base -> origin/gh/ydwu4/290/base 2025-08-14T21:24:12.8676542Z * [new branch] gh/ydwu4/290/head -> origin/gh/ydwu4/290/head 2025-08-14T21:24:12.8676995Z * [new branch] gh/ydwu4/290/orig -> origin/gh/ydwu4/290/orig 2025-08-14T21:24:12.8678569Z * [new branch] gh/ydwu4/291/base -> origin/gh/ydwu4/291/base 2025-08-14T21:24:12.8678947Z * [new branch] gh/ydwu4/291/head -> origin/gh/ydwu4/291/head 2025-08-14T21:24:12.8679284Z * [new branch] gh/ydwu4/291/orig -> origin/gh/ydwu4/291/orig 2025-08-14T21:24:12.8680918Z * [new branch] gh/ydwu4/292/base -> origin/gh/ydwu4/292/base 2025-08-14T21:24:12.8681280Z * [new branch] gh/ydwu4/292/head -> origin/gh/ydwu4/292/head 2025-08-14T21:24:12.8681589Z * [new branch] gh/ydwu4/292/orig -> origin/gh/ydwu4/292/orig 2025-08-14T21:24:12.8682219Z * [new branch] gh/ydwu4/293/base -> origin/gh/ydwu4/293/base 2025-08-14T21:24:12.8682790Z * [new branch] gh/ydwu4/293/head -> origin/gh/ydwu4/293/head 2025-08-14T21:24:12.8683410Z * [new branch] gh/ydwu4/293/orig -> origin/gh/ydwu4/293/orig 2025-08-14T21:24:12.8684689Z * [new branch] gh/ydwu4/294/base -> origin/gh/ydwu4/294/base 2025-08-14T21:24:12.8685022Z * [new branch] gh/ydwu4/294/head -> origin/gh/ydwu4/294/head 2025-08-14T21:24:12.8685792Z * [new branch] gh/ydwu4/294/orig -> origin/gh/ydwu4/294/orig 2025-08-14T21:24:12.8687313Z * [new branch] gh/ydwu4/295/base -> origin/gh/ydwu4/295/base 2025-08-14T21:24:12.8687792Z * [new branch] gh/ydwu4/295/head -> origin/gh/ydwu4/295/head 2025-08-14T21:24:12.8688233Z * [new branch] gh/ydwu4/295/orig -> origin/gh/ydwu4/295/orig 2025-08-14T21:24:12.8688852Z * [new branch] gh/ydwu4/296/base -> origin/gh/ydwu4/296/base 2025-08-14T21:24:12.8689430Z * [new branch] gh/ydwu4/296/head -> origin/gh/ydwu4/296/head 2025-08-14T21:24:12.8690105Z * [new branch] gh/ydwu4/296/orig -> origin/gh/ydwu4/296/orig 2025-08-14T21:24:12.8695497Z * [new branch] gh/ydwu4/297/base -> origin/gh/ydwu4/297/base 2025-08-14T21:24:12.8696043Z * [new branch] gh/ydwu4/297/head -> origin/gh/ydwu4/297/head 2025-08-14T21:24:12.8696508Z * [new branch] gh/ydwu4/297/orig -> origin/gh/ydwu4/297/orig 2025-08-14T21:24:12.8697266Z * [new branch] gh/ydwu4/298/base -> origin/gh/ydwu4/298/base 2025-08-14T21:24:12.8697659Z * [new branch] gh/ydwu4/298/head -> origin/gh/ydwu4/298/head 2025-08-14T21:24:12.8697988Z * [new branch] gh/ydwu4/298/orig -> origin/gh/ydwu4/298/orig 2025-08-14T21:24:12.8698311Z * [new branch] gh/ydwu4/299/base -> origin/gh/ydwu4/299/base 2025-08-14T21:24:12.8698639Z * [new branch] gh/ydwu4/299/head -> origin/gh/ydwu4/299/head 2025-08-14T21:24:12.8698971Z * [new branch] gh/ydwu4/299/orig -> origin/gh/ydwu4/299/orig 2025-08-14T21:24:12.8699293Z * [new branch] gh/ydwu4/300/base -> origin/gh/ydwu4/300/base 2025-08-14T21:24:12.8704834Z * [new branch] gh/ydwu4/300/head -> origin/gh/ydwu4/300/head 2025-08-14T21:24:12.8705269Z * [new branch] gh/ydwu4/300/orig -> origin/gh/ydwu4/300/orig 2025-08-14T21:24:12.8705620Z * [new branch] gh/ydwu4/301/base -> origin/gh/ydwu4/301/base 2025-08-14T21:24:12.8705957Z * [new branch] gh/ydwu4/301/head -> origin/gh/ydwu4/301/head 2025-08-14T21:24:12.8706284Z * [new branch] gh/ydwu4/301/orig -> origin/gh/ydwu4/301/orig 2025-08-14T21:24:12.8706620Z * [new branch] gh/ydwu4/302/base -> origin/gh/ydwu4/302/base 2025-08-14T21:24:12.8706934Z * [new branch] gh/ydwu4/302/head -> origin/gh/ydwu4/302/head 2025-08-14T21:24:12.8707260Z * [new branch] gh/ydwu4/302/orig -> origin/gh/ydwu4/302/orig 2025-08-14T21:24:12.8707574Z * [new branch] gh/ydwu4/303/base -> origin/gh/ydwu4/303/base 2025-08-14T21:24:12.8707902Z * [new branch] gh/ydwu4/303/head -> origin/gh/ydwu4/303/head 2025-08-14T21:24:12.8708234Z * [new branch] gh/ydwu4/303/orig -> origin/gh/ydwu4/303/orig 2025-08-14T21:24:12.8713767Z * [new branch] gh/ydwu4/304/base -> origin/gh/ydwu4/304/base 2025-08-14T21:24:12.8714322Z * [new branch] gh/ydwu4/304/head -> origin/gh/ydwu4/304/head 2025-08-14T21:24:12.8714793Z * [new branch] gh/ydwu4/304/orig -> origin/gh/ydwu4/304/orig 2025-08-14T21:24:12.8715568Z * [new branch] gh/ydwu4/305/base -> origin/gh/ydwu4/305/base 2025-08-14T21:24:12.8715956Z * [new branch] gh/ydwu4/305/head -> origin/gh/ydwu4/305/head 2025-08-14T21:24:12.8716283Z * [new branch] gh/ydwu4/305/orig -> origin/gh/ydwu4/305/orig 2025-08-14T21:24:12.8716595Z * [new branch] gh/ydwu4/306/base -> origin/gh/ydwu4/306/base 2025-08-14T21:24:12.8716948Z * [new branch] gh/ydwu4/306/head -> origin/gh/ydwu4/306/head 2025-08-14T21:24:12.8717474Z * [new branch] gh/ydwu4/306/orig -> origin/gh/ydwu4/306/orig 2025-08-14T21:24:12.8717795Z * [new branch] gh/ydwu4/307/base -> origin/gh/ydwu4/307/base 2025-08-14T21:24:12.8718104Z * [new branch] gh/ydwu4/307/head -> origin/gh/ydwu4/307/head 2025-08-14T21:24:12.8718451Z * [new branch] gh/ydwu4/307/orig -> origin/gh/ydwu4/307/orig 2025-08-14T21:24:12.8718760Z * [new branch] gh/ydwu4/308/base -> origin/gh/ydwu4/308/base 2025-08-14T21:24:12.8719068Z * [new branch] gh/ydwu4/308/head -> origin/gh/ydwu4/308/head 2025-08-14T21:24:12.8719370Z * [new branch] gh/ydwu4/308/orig -> origin/gh/ydwu4/308/orig 2025-08-14T21:24:12.8719850Z * [new branch] gh/ydwu4/309/base -> origin/gh/ydwu4/309/base 2025-08-14T21:24:12.8720624Z * [new branch] gh/ydwu4/309/head -> origin/gh/ydwu4/309/head 2025-08-14T21:24:12.8721029Z * [new branch] gh/ydwu4/309/orig -> origin/gh/ydwu4/309/orig 2025-08-14T21:24:12.8721362Z * [new branch] gh/ydwu4/310/base -> origin/gh/ydwu4/310/base 2025-08-14T21:24:12.8721692Z * [new branch] gh/ydwu4/310/head -> origin/gh/ydwu4/310/head 2025-08-14T21:24:12.8722016Z * [new branch] gh/ydwu4/310/orig -> origin/gh/ydwu4/310/orig 2025-08-14T21:24:12.8722332Z * [new branch] gh/ydwu4/311/base -> origin/gh/ydwu4/311/base 2025-08-14T21:24:12.8722657Z * [new branch] gh/ydwu4/311/head -> origin/gh/ydwu4/311/head 2025-08-14T21:24:12.8722986Z * [new branch] gh/ydwu4/311/orig -> origin/gh/ydwu4/311/orig 2025-08-14T21:24:12.8723307Z * [new branch] gh/yf225/133/base -> origin/gh/yf225/133/base 2025-08-14T21:24:12.8723792Z * [new branch] gh/yf225/133/head -> origin/gh/yf225/133/head 2025-08-14T21:24:12.8724619Z * [new branch] gh/yf225/171/base -> origin/gh/yf225/171/base 2025-08-14T21:24:12.8725337Z * [new branch] gh/yf225/171/head -> origin/gh/yf225/171/head 2025-08-14T21:24:12.8726049Z * [new branch] gh/yf225/171/orig -> origin/gh/yf225/171/orig 2025-08-14T21:24:12.8730737Z * [new branch] gh/yf225/172/base -> origin/gh/yf225/172/base 2025-08-14T21:24:12.8735710Z * [new branch] gh/yf225/172/head -> origin/gh/yf225/172/head 2025-08-14T21:24:12.8738154Z * [new branch] gh/yf225/172/orig -> origin/gh/yf225/172/orig 2025-08-14T21:24:12.8738534Z * [new branch] gh/yf225/93/base -> origin/gh/yf225/93/base 2025-08-14T21:24:12.8738876Z * [new branch] gh/yf225/93/head -> origin/gh/yf225/93/head 2025-08-14T21:24:12.8739245Z * [new branch] gh/yifuwang/152/base -> origin/gh/yifuwang/152/base 2025-08-14T21:24:12.8739729Z * [new branch] gh/yifuwang/152/head -> origin/gh/yifuwang/152/head 2025-08-14T21:24:12.8742705Z * [new branch] gh/yifuwang/152/orig -> origin/gh/yifuwang/152/orig 2025-08-14T21:24:12.8743186Z * [new branch] gh/yifuwang/195/base -> origin/gh/yifuwang/195/base 2025-08-14T21:24:12.8746658Z * [new branch] gh/yifuwang/195/head -> origin/gh/yifuwang/195/head 2025-08-14T21:24:12.8746994Z * [new branch] gh/yifuwang/195/orig -> origin/gh/yifuwang/195/orig 2025-08-14T21:24:12.8752032Z * [new branch] gh/yiming0416/1/base -> origin/gh/yiming0416/1/base 2025-08-14T21:24:12.8755337Z * [new branch] gh/yiming0416/1/head -> origin/gh/yiming0416/1/head 2025-08-14T21:24:12.8755743Z * [new branch] gh/yiming0416/2/base -> origin/gh/yiming0416/2/base 2025-08-14T21:24:12.8756335Z * [new branch] gh/yiming0416/2/head -> origin/gh/yiming0416/2/head 2025-08-14T21:24:12.8756685Z * [new branch] gh/ysiraichi/79/base -> origin/gh/ysiraichi/79/base 2025-08-14T21:24:12.8757056Z * [new branch] gh/ysiraichi/79/head -> origin/gh/ysiraichi/79/head 2025-08-14T21:24:12.8757421Z * [new branch] gh/ysiraichi/79/orig -> origin/gh/ysiraichi/79/orig 2025-08-14T21:24:12.8757775Z * [new branch] gh/ysiraichi/81/base -> origin/gh/ysiraichi/81/base 2025-08-14T21:24:12.8758125Z * [new branch] gh/ysiraichi/81/head -> origin/gh/ysiraichi/81/head 2025-08-14T21:24:12.8758542Z * [new branch] gh/ysiraichi/81/orig -> origin/gh/ysiraichi/81/orig 2025-08-14T21:24:12.8758890Z * [new branch] gh/ysiraichi/84/base -> origin/gh/ysiraichi/84/base 2025-08-14T21:24:12.8759256Z * [new branch] gh/ysiraichi/84/head -> origin/gh/ysiraichi/84/head 2025-08-14T21:24:12.8759624Z * [new branch] gh/ysiraichi/84/orig -> origin/gh/ysiraichi/84/orig 2025-08-14T21:24:12.8759988Z * [new branch] gh/ysiraichi/85/base -> origin/gh/ysiraichi/85/base 2025-08-14T21:24:12.8760349Z * [new branch] gh/ysiraichi/85/head -> origin/gh/ysiraichi/85/head 2025-08-14T21:24:12.8760691Z * [new branch] gh/ysiraichi/85/orig -> origin/gh/ysiraichi/85/orig 2025-08-14T21:24:12.8761047Z * [new branch] gh/ysiraichi/86/base -> origin/gh/ysiraichi/86/base 2025-08-14T21:24:12.8761459Z * [new branch] gh/ysiraichi/86/head -> origin/gh/ysiraichi/86/head 2025-08-14T21:24:12.8761820Z * [new branch] gh/ysiraichi/86/orig -> origin/gh/ysiraichi/86/orig 2025-08-14T21:24:12.8762170Z * [new branch] gh/ysiraichi/87/base -> origin/gh/ysiraichi/87/base 2025-08-14T21:24:12.8762595Z * [new branch] gh/ysiraichi/87/head -> origin/gh/ysiraichi/87/head 2025-08-14T21:24:12.8762956Z * [new branch] gh/ysiraichi/87/orig -> origin/gh/ysiraichi/87/orig 2025-08-14T21:24:12.8763305Z * [new branch] gh/ysiraichi/88/base -> origin/gh/ysiraichi/88/base 2025-08-14T21:24:12.8763667Z * [new branch] gh/ysiraichi/88/head -> origin/gh/ysiraichi/88/head 2025-08-14T21:24:12.8764020Z * [new branch] gh/ysiraichi/88/orig -> origin/gh/ysiraichi/88/orig 2025-08-14T21:24:12.8764375Z * [new branch] gh/yuguo68/1/base -> origin/gh/yuguo68/1/base 2025-08-14T21:24:12.8764711Z * [new branch] gh/yuguo68/1/head -> origin/gh/yuguo68/1/head 2025-08-14T21:24:12.8765031Z * [new branch] gh/yuguo68/1/orig -> origin/gh/yuguo68/1/orig 2025-08-14T21:24:12.8765365Z * [new branch] gh/yuguo68/2/base -> origin/gh/yuguo68/2/base 2025-08-14T21:24:12.8765972Z * [new branch] gh/yuguo68/2/head -> origin/gh/yuguo68/2/head 2025-08-14T21:24:12.8766317Z * [new branch] gh/yuguo68/2/orig -> origin/gh/yuguo68/2/orig 2025-08-14T21:24:12.8766658Z * [new branch] gh/zhxchen17/25/base -> origin/gh/zhxchen17/25/base 2025-08-14T21:24:12.8767022Z * [new branch] gh/zhxchen17/25/head -> origin/gh/zhxchen17/25/head 2025-08-14T21:24:12.8767382Z * [new branch] gh/zhxchen17/25/orig -> origin/gh/zhxchen17/25/orig 2025-08-14T21:24:12.8767723Z * [new branch] gh/zhxchen17/31/base -> origin/gh/zhxchen17/31/base 2025-08-14T21:24:12.8768068Z * [new branch] gh/zhxchen17/31/head -> origin/gh/zhxchen17/31/head 2025-08-14T21:24:12.8768417Z * [new branch] gh/zhxchen17/31/orig -> origin/gh/zhxchen17/31/orig 2025-08-14T21:24:12.8768895Z * [new branch] gh/zhxchen17/33/base -> origin/gh/zhxchen17/33/base 2025-08-14T21:24:12.8769424Z * [new branch] gh/zhxchen17/33/head -> origin/gh/zhxchen17/33/head 2025-08-14T21:24:12.8769865Z * [new branch] gh/zhxchen17/33/orig -> origin/gh/zhxchen17/33/orig 2025-08-14T21:24:12.8770656Z * [new branch] gh/zhxchen17/34/base -> origin/gh/zhxchen17/34/base 2025-08-14T21:24:12.8771041Z * [new branch] gh/zhxchen17/34/head -> origin/gh/zhxchen17/34/head 2025-08-14T21:24:12.8771660Z * [new branch] gh/zhxchen17/35/base -> origin/gh/zhxchen17/35/base 2025-08-14T21:24:12.8772324Z * [new branch] gh/zhxchen17/35/head -> origin/gh/zhxchen17/35/head 2025-08-14T21:24:12.8776453Z * [new branch] gh/zhxchen17/36/base -> origin/gh/zhxchen17/36/base 2025-08-14T21:24:12.8776835Z * [new branch] gh/zhxchen17/36/head -> origin/gh/zhxchen17/36/head 2025-08-14T21:24:12.8777178Z * [new branch] gh/zhxchen17/36/orig -> origin/gh/zhxchen17/36/orig 2025-08-14T21:24:12.8777509Z * [new branch] gh/zklaus/1/base -> origin/gh/zklaus/1/base 2025-08-14T21:24:12.8777816Z * [new branch] gh/zklaus/1/head -> origin/gh/zklaus/1/head 2025-08-14T21:24:12.8778110Z * [new branch] gh/zklaus/1/orig -> origin/gh/zklaus/1/orig 2025-08-14T21:24:12.8778416Z * [new branch] gh/zklaus/10/base -> origin/gh/zklaus/10/base 2025-08-14T21:24:12.8778753Z * [new branch] gh/zklaus/10/head -> origin/gh/zklaus/10/head 2025-08-14T21:24:12.8779712Z * [new branch] gh/zklaus/10/orig -> origin/gh/zklaus/10/orig 2025-08-14T21:24:12.8780272Z * [new branch] gh/zklaus/11/base -> origin/gh/zklaus/11/base 2025-08-14T21:24:12.8781129Z * [new branch] gh/zklaus/11/head -> origin/gh/zklaus/11/head 2025-08-14T21:24:12.8781631Z * [new branch] gh/zklaus/11/orig -> origin/gh/zklaus/11/orig 2025-08-14T21:24:12.8783006Z * [new branch] gh/zklaus/12/base -> origin/gh/zklaus/12/base 2025-08-14T21:24:12.8783533Z * [new branch] gh/zklaus/12/head -> origin/gh/zklaus/12/head 2025-08-14T21:24:12.8783940Z * [new branch] gh/zklaus/12/orig -> origin/gh/zklaus/12/orig 2025-08-14T21:24:12.8786297Z * [new branch] gh/zklaus/14/base -> origin/gh/zklaus/14/base 2025-08-14T21:24:12.8786657Z * [new branch] gh/zklaus/14/head -> origin/gh/zklaus/14/head 2025-08-14T21:24:12.8786970Z * [new branch] gh/zklaus/14/orig -> origin/gh/zklaus/14/orig 2025-08-14T21:24:12.8787287Z * [new branch] gh/zklaus/15/base -> origin/gh/zklaus/15/base 2025-08-14T21:24:12.8787625Z * [new branch] gh/zklaus/15/head -> origin/gh/zklaus/15/head 2025-08-14T21:24:12.8788306Z * [new branch] gh/zklaus/15/orig -> origin/gh/zklaus/15/orig 2025-08-14T21:24:12.8789244Z * [new branch] gh/zklaus/16/base -> origin/gh/zklaus/16/base 2025-08-14T21:24:12.8789658Z * [new branch] gh/zklaus/16/head -> origin/gh/zklaus/16/head 2025-08-14T21:24:12.8790577Z * [new branch] gh/zklaus/16/orig -> origin/gh/zklaus/16/orig 2025-08-14T21:24:12.8791135Z * [new branch] gh/zklaus/17/base -> origin/gh/zklaus/17/base 2025-08-14T21:24:12.8791728Z * [new branch] gh/zklaus/17/head -> origin/gh/zklaus/17/head 2025-08-14T21:24:12.8793446Z * [new branch] gh/zklaus/17/orig -> origin/gh/zklaus/17/orig 2025-08-14T21:24:12.8793819Z * [new branch] gh/zklaus/18/base -> origin/gh/zklaus/18/base 2025-08-14T21:24:12.8794136Z * [new branch] gh/zklaus/18/head -> origin/gh/zklaus/18/head 2025-08-14T21:24:12.8794681Z * [new branch] gh/zklaus/18/orig -> origin/gh/zklaus/18/orig 2025-08-14T21:24:12.8795445Z * [new branch] gh/zklaus/19/base -> origin/gh/zklaus/19/base 2025-08-14T21:24:12.8796024Z * [new branch] gh/zklaus/19/head -> origin/gh/zklaus/19/head 2025-08-14T21:24:12.8796640Z * [new branch] gh/zklaus/19/orig -> origin/gh/zklaus/19/orig 2025-08-14T21:24:12.8797481Z * [new branch] gh/zklaus/7/base -> origin/gh/zklaus/7/base 2025-08-14T21:24:12.8798017Z * [new branch] gh/zklaus/7/head -> origin/gh/zklaus/7/head 2025-08-14T21:24:12.8798695Z * [new branch] gh/zklaus/7/orig -> origin/gh/zklaus/7/orig 2025-08-14T21:24:12.8799687Z * [new branch] gh/zklaus/9/base -> origin/gh/zklaus/9/base 2025-08-14T21:24:12.8800163Z * [new branch] gh/zklaus/9/head -> origin/gh/zklaus/9/head 2025-08-14T21:24:12.8800785Z * [new branch] gh/zklaus/9/orig -> origin/gh/zklaus/9/orig 2025-08-14T21:24:12.8802104Z * [new branch] gh/zou3519/1175/base -> origin/gh/zou3519/1175/base 2025-08-14T21:24:12.8803137Z * [new branch] gh/zou3519/1175/head -> origin/gh/zou3519/1175/head 2025-08-14T21:24:12.8803446Z * [new branch] gh/zou3519/1175/orig -> origin/gh/zou3519/1175/orig 2025-08-14T21:24:12.8805830Z * [new branch] gh/zou3519/1177/base -> origin/gh/zou3519/1177/base 2025-08-14T21:24:12.8806254Z * [new branch] gh/zou3519/1177/head -> origin/gh/zou3519/1177/head 2025-08-14T21:24:12.8806729Z * [new branch] gh/zou3519/1177/orig -> origin/gh/zou3519/1177/orig 2025-08-14T21:24:12.8813136Z * [new branch] gh/zou3519/1187/base -> origin/gh/zou3519/1187/base 2025-08-14T21:24:12.8818246Z * [new branch] gh/zou3519/1187/head -> origin/gh/zou3519/1187/head 2025-08-14T21:24:12.8823792Z * [new branch] gh/zou3519/1187/orig -> origin/gh/zou3519/1187/orig 2025-08-14T21:24:12.8828417Z * [new branch] gh/zou3519/1188/base -> origin/gh/zou3519/1188/base 2025-08-14T21:24:12.8833996Z * [new branch] gh/zou3519/1188/head -> origin/gh/zou3519/1188/head 2025-08-14T21:24:12.8836521Z * [new branch] gh/zou3519/1188/orig -> origin/gh/zou3519/1188/orig 2025-08-14T21:24:12.8836974Z * [new branch] gh/zou3519/1189/base -> origin/gh/zou3519/1189/base 2025-08-14T21:24:12.8840198Z * [new branch] gh/zou3519/1189/head -> origin/gh/zou3519/1189/head 2025-08-14T21:24:12.8840567Z * [new branch] gh/zou3519/1189/orig -> origin/gh/zou3519/1189/orig 2025-08-14T21:24:12.8840905Z * [new branch] gh/zou3519/1190/base -> origin/gh/zou3519/1190/base 2025-08-14T21:24:12.8841249Z * [new branch] gh/zou3519/1190/head -> origin/gh/zou3519/1190/head 2025-08-14T21:24:12.8841595Z * [new branch] gh/zou3519/1190/orig -> origin/gh/zou3519/1190/orig 2025-08-14T21:24:12.8841937Z * [new branch] gh/zou3519/1191/base -> origin/gh/zou3519/1191/base 2025-08-14T21:24:12.8842268Z * [new branch] gh/zou3519/1191/head -> origin/gh/zou3519/1191/head 2025-08-14T21:24:12.8842598Z * [new branch] gh/zou3519/1191/orig -> origin/gh/zou3519/1191/orig 2025-08-14T21:24:12.8842932Z * [new branch] gh/zpcore/1/base -> origin/gh/zpcore/1/base 2025-08-14T21:24:12.8843253Z * [new branch] gh/zpcore/1/head -> origin/gh/zpcore/1/head 2025-08-14T21:24:12.8843590Z * [new branch] gh/zpcore/10/base -> origin/gh/zpcore/10/base 2025-08-14T21:24:12.8843908Z * [new branch] gh/zpcore/10/head -> origin/gh/zpcore/10/head 2025-08-14T21:24:12.8844225Z * [new branch] gh/zpcore/10/orig -> origin/gh/zpcore/10/orig 2025-08-14T21:24:12.8844565Z * [new branch] gh/zpcore/11/base -> origin/gh/zpcore/11/base 2025-08-14T21:24:12.8845105Z * [new branch] gh/zpcore/11/head -> origin/gh/zpcore/11/head 2025-08-14T21:24:12.8845438Z * [new branch] gh/zpcore/11/orig -> origin/gh/zpcore/11/orig 2025-08-14T21:24:12.8845987Z * [new branch] gh/zpcore/12/base -> origin/gh/zpcore/12/base 2025-08-14T21:24:12.8846328Z * [new branch] gh/zpcore/12/head -> origin/gh/zpcore/12/head 2025-08-14T21:24:12.8846664Z * [new branch] gh/zpcore/12/orig -> origin/gh/zpcore/12/orig 2025-08-14T21:24:12.8847000Z * [new branch] gh/zpcore/2/base -> origin/gh/zpcore/2/base 2025-08-14T21:24:12.8847330Z * [new branch] gh/zpcore/2/head -> origin/gh/zpcore/2/head 2025-08-14T21:24:12.8847658Z * [new branch] gh/zpcore/3/base -> origin/gh/zpcore/3/base 2025-08-14T21:24:12.8847997Z * [new branch] gh/zpcore/3/head -> origin/gh/zpcore/3/head 2025-08-14T21:24:12.8848314Z * [new branch] gh/zpcore/4/base -> origin/gh/zpcore/4/base 2025-08-14T21:24:12.8848642Z * [new branch] gh/zpcore/4/head -> origin/gh/zpcore/4/head 2025-08-14T21:24:12.8848946Z * [new branch] gh/zpcore/5/base -> origin/gh/zpcore/5/base 2025-08-14T21:24:12.8849249Z * [new branch] gh/zpcore/5/head -> origin/gh/zpcore/5/head 2025-08-14T21:24:12.8849546Z * [new branch] gh/zpcore/6/base -> origin/gh/zpcore/6/base 2025-08-14T21:24:12.8849849Z * [new branch] gh/zpcore/6/head -> origin/gh/zpcore/6/head 2025-08-14T21:24:12.8850149Z * [new branch] gh/zpcore/7/base -> origin/gh/zpcore/7/base 2025-08-14T21:24:12.8850452Z * [new branch] gh/zpcore/7/head -> origin/gh/zpcore/7/head 2025-08-14T21:24:12.8850813Z * [new branch] gh/zpcore/8/base -> origin/gh/zpcore/8/base 2025-08-14T21:24:12.8851121Z * [new branch] gh/zpcore/8/head -> origin/gh/zpcore/8/head 2025-08-14T21:24:12.8851423Z * [new branch] gh/zpcore/9/head -> origin/gh/zpcore/9/head 2025-08-14T21:24:12.8851712Z * [new branch] gh/zpcore/9/orig -> origin/gh/zpcore/9/orig 2025-08-14T21:24:12.8852017Z * [new branch] google-main -> origin/google-main 2025-08-14T21:24:12.8852358Z * [new branch] guangyey/external_stream -> origin/guangyey/external_stream 2025-08-14T21:24:12.8852707Z * [new branch] guangyey/host_alloc -> origin/guangyey/host_alloc 2025-08-14T21:24:12.8853026Z * [new branch] guangyey/test_2025 -> origin/guangyey/test_2025 2025-08-14T21:24:12.8853462Z * [new branch] guilhermeleobas/cherry-pick-55d87d9dfd9 -> origin/guilhermeleobas/cherry-pick-55d87d9dfd9 2025-08-14T21:24:12.8853904Z * [new branch] haozhe/bf16-dynamic-shape -> origin/haozhe/bf16-dynamic-shape 2025-08-14T21:24:12.8854233Z * [new branch] hc_baseline -> origin/hc_baseline 2025-08-14T21:24:12.8854546Z * [new branch] headeronlyScalarType -> origin/headeronlyScalarType 2025-08-14T21:24:12.8854858Z * [new branch] hf_update -> origin/hf_update 2025-08-14T21:24:12.8855154Z * [new branch] hhh_decomp_mul -> origin/hhh_decomp_mul 2025-08-14T21:24:12.8855441Z * [new branch] hhh_rand -> origin/hhh_rand 2025-08-14T21:24:12.8855719Z * [new branch] hoy/mmsplitk -> origin/hoy/mmsplitk 2025-08-14T21:24:12.8856025Z * [new branch] hoy/triton-PR3973 -> origin/hoy/triton-PR3973 2025-08-14T21:24:12.8856396Z * [new branch] hoy/triton-coalescing-baseline -> origin/hoy/triton-coalescing-baseline 2025-08-14T21:24:12.8856836Z * [new branch] hoy/triton-coalescing-min -> origin/hoy/triton-coalescing-min 2025-08-14T21:24:12.8857210Z * [new branch] hoy/triton-coalescing-new -> origin/hoy/triton-coalescing-new 2025-08-14T21:24:12.8857570Z * [new branch] hoy/triton-coalescing-vec -> origin/hoy/triton-coalescing-vec 2025-08-14T21:24:12.8857912Z * [new branch] inductordecompfix -> origin/inductordecompfix 2025-08-14T21:24:12.8858375Z * [new branch] inline -> origin/inline 2025-08-14T21:24:12.8863399Z * [new branch] inlining -> origin/inlining 2025-08-14T21:24:12.8863579Z * [new branch] inlining-ezyang -> origin/inlining-ezyang 2025-08-14T21:24:12.8863751Z * [new branch] int8_sdpa -> origin/int8_sdpa 2025-08-14T21:24:12.8863915Z * [new branch] invoke-subgraph -> origin/invoke-subgraph 2025-08-14T21:24:12.8864221Z * [new branch] issue#58739 -> origin/issue#58739 2025-08-14T21:24:12.8864347Z * [new branch] issue-154849 -> origin/issue-154849 2025-08-14T21:24:12.8864553Z * [new branch] ivanov/cherry-pick-ckpt-fixes -> origin/ivanov/cherry-pick-ckpt-fixes 2025-08-14T21:24:12.8864818Z * [new branch] jcaip/test-cusparselt-version-0.6.2 -> origin/jcaip/test-cusparselt-version-0.6.2 2025-08-14T21:24:12.8865048Z * [new branch] jcaip/update-cusparselt-0.6.2 -> origin/jcaip/update-cusparselt-0.6.2 2025-08-14T21:24:12.8865242Z * [new branch] jithunnair-amd-patch-1 -> origin/jithunnair-amd-patch-1 2025-08-14T21:24:12.8865436Z * [new branch] justinchu/attention-tests -> origin/justinchu/attention-tests 2025-08-14T21:24:12.8870054Z * [new branch] justinchu/native-qdq -> origin/justinchu/native-qdq 2025-08-14T21:24:12.8875031Z * [new branch] justinchuby/JitScalarType -> origin/justinchuby/JitScalarType 2025-08-14T21:24:12.8876665Z * [new branch] justinchuby/dynamo-true -> origin/justinchuby/dynamo-true 2025-08-14T21:24:12.8876849Z * [new branch] justinchuby/opset-20 -> origin/justinchuby/opset-20 2025-08-14T21:24:12.8877003Z * [new branch] kainan666/xlf_debug -> origin/kainan666/xlf_debug 2025-08-14T21:24:12.8877128Z * [new branch] kainan_test -> origin/kainan_test 2025-08-14T21:24:12.8877321Z * [new branch] leslie/enable_poc_reduction_fusion -> origin/leslie/enable_poc_reduction_fusion 2025-08-14T21:24:12.8877520Z * [new branch] leslie/test_group_gemm_epilogues -> origin/leslie/test_group_gemm_epilogues 2025-08-14T21:24:12.8877739Z * [new branch] lessw2020/fix_cutlass_cache_error -> origin/lessw2020/fix_cutlass_cache_error 2025-08-14T21:24:12.8877882Z * [new branch] liaoxuan/shm_all_reduce -> origin/liaoxuan/shm_all_reduce 2025-08-14T21:24:12.8878020Z * [new branch] liaoxuan/tags_issue -> origin/liaoxuan/tags_issue 2025-08-14T21:24:12.8878182Z * [new branch] liaoxuan/test_fa_disable_softmax -> origin/liaoxuan/test_fa_disable_softmax 2025-08-14T21:24:12.8878323Z * [new branch] liaoxuan/test_int8_sdpa -> origin/liaoxuan/test_int8_sdpa 2025-08-14T21:24:12.8878450Z * [new branch] lintbuilddocker -> origin/lintbuilddocker 2025-08-14T21:24:12.8878570Z * [new branch] llama4-stable -> origin/llama4-stable 2025-08-14T21:24:12.8878691Z * [new branch] logdetfix -> origin/logdetfix 2025-08-14T21:24:12.8878809Z * [new branch] lts/release/1.8 -> origin/lts/release/1.8 2025-08-14T21:24:12.8881419Z * [new branch] lucaskabela/#94773 -> origin/lucaskabela/#94773 2025-08-14T21:24:12.8882180Z * [new branch] lucaskabela/fix_157452 -> origin/lucaskabela/fix_157452 2025-08-14T21:24:12.8882643Z * [new branch] lucaskabela/fix_circular_import_158120 -> origin/lucaskabela/fix_circular_import_158120 2025-08-14T21:24:12.8882859Z * [new branch] lucaskabela/func_under_decomp -> origin/lucaskabela/func_under_decomp 2025-08-14T21:24:12.8883057Z * [new branch] lucaskabela/functional_in_dynamo -> origin/lucaskabela/functional_in_dynamo 2025-08-14T21:24:12.8883292Z * [new branch] lucaskabela/install_params_as_graph_attr -> origin/lucaskabela/install_params_as_graph_attr 2025-08-14T21:24:12.8883453Z * [new branch] lucaskabela/issue_120648 -> origin/lucaskabela/issue_120648 2025-08-14T21:24:12.8883662Z * [new branch] lucaskabela/parameters_as_graph_attr -> origin/lucaskabela/parameters_as_graph_attr 2025-08-14T21:24:12.8883832Z * [new branch] lucaskabela/registry_fix -> origin/lucaskabela/registry_fix 2025-08-14T21:24:12.8884080Z * [new branch] lucaskabela/remove_aot_dispatcher_metadata -> origin/lucaskabela/remove_aot_dispatcher_metadata 2025-08-14T21:24:12.8884250Z * [new branch] lucaskabela/type_guards -> origin/lucaskabela/type_guards 2025-08-14T21:24:12.8884417Z * [new branch] lucaskabela/typing-misc -> origin/lucaskabela/typing-misc 2025-08-14T21:24:12.8884586Z * [new branch] lucaskabela/typing_backends -> origin/lucaskabela/typing_backends 2025-08-14T21:24:12.8884861Z * [new branch] lucaskabela/typing_bytecode_analysis_transform -> origin/lucaskabela/typing_bytecode_analysis_transform 2025-08-14T21:24:12.8885070Z * [new branch] lucaskabela/typing_cache_files -> origin/lucaskabela/typing_cache_files 2025-08-14T21:24:12.8885279Z * [new branch] lucaskabela/typing_compile_autograd -> origin/lucaskabela/typing_compile_autograd 2025-08-14T21:24:12.8885963Z * [new branch] lucaskabela/typing_debug_utils.py -> origin/lucaskabela/typing_debug_utils.py 2025-08-14T21:24:12.8892021Z * [new branch] lucaskabela/typing_decorators -> origin/lucaskabela/typing_decorators 2025-08-14T21:24:12.8894285Z * [new branch] lucaskabela/typing_eval_frame -> origin/lucaskabela/typing_eval_frame 2025-08-14T21:24:12.8894587Z * [new branch] lucaskabela/typing_for_codegen -> origin/lucaskabela/typing_for_codegen 2025-08-14T21:24:12.8900075Z * [new branch] lucaskabela/typing_output_graph -> origin/lucaskabela/typing_output_graph 2025-08-14T21:24:12.8900412Z * [new branch] lucaskabela/typing_side_effects -> origin/lucaskabela/typing_side_effects 2025-08-14T21:24:12.8900674Z * [new branch] lucaskabela/typing_source_guard -> origin/lucaskabela/typing_source_guard 2025-08-14T21:24:12.8900861Z * [new branch] lucaskabela/typing_trace_rules -> origin/lucaskabela/typing_trace_rules 2025-08-14T21:24:12.8901049Z * [new branch] lucaskabela/typing_utils.py -> origin/lucaskabela/typing_utils.py 2025-08-14T21:24:12.8901407Z * [new branch] lucaskabela/typing_utils_improvements -> origin/lucaskabela/typing_utils_improvements 2025-08-14T21:24:12.8901556Z * [new branch] main -> origin/main 2025-08-14T21:24:12.8901769Z * [new branch] main-enable-b200-distributed-tests -> origin/main-enable-b200-distributed-tests 2025-08-14T21:24:12.8901911Z * [new branch] malfet-patch-1 -> origin/malfet-patch-1 2025-08-14T21:24:12.8902059Z * [new branch] malfet-patch-10 -> origin/malfet-patch-10 2025-08-14T21:24:12.8902184Z * [new branch] malfet-patch-11 -> origin/malfet-patch-11 2025-08-14T21:24:12.8902303Z * [new branch] malfet-patch-13 -> origin/malfet-patch-13 2025-08-14T21:24:12.8902431Z * [new branch] malfet-patch-14 -> origin/malfet-patch-14 2025-08-14T21:24:12.8902560Z * [new branch] malfet-patch-2 -> origin/malfet-patch-2 2025-08-14T21:24:12.8902830Z * [new branch] malfet-patch-3 -> origin/malfet-patch-3 2025-08-14T21:24:12.8902947Z * [new branch] malfet-patch-4 -> origin/malfet-patch-4 2025-08-14T21:24:12.8903064Z * [new branch] malfet-patch-5 -> origin/malfet-patch-5 2025-08-14T21:24:12.8903188Z * [new branch] malfet-patch-6 -> origin/malfet-patch-6 2025-08-14T21:24:12.8903316Z * [new branch] malfet-patch-7 -> origin/malfet-patch-7 2025-08-14T21:24:12.8903455Z * [new branch] malfet-patch-8 -> origin/malfet-patch-8 2025-08-14T21:24:12.8903781Z * [new branch] malfet-patch-9 -> origin/malfet-patch-9 2025-08-14T21:24:12.8905173Z * [new branch] malfet/delete-upsteam-cuda -> origin/malfet/delete-upsteam-cuda 2025-08-14T21:24:12.8905450Z * [new branch] malfet/mps-implement-col2im -> origin/malfet/mps-implement-col2im 2025-08-14T21:24:12.8907699Z * [new branch] manuel/fix_multidim_boolean_indexing -> origin/manuel/fix_multidim_boolean_indexing 2025-08-14T21:24:12.8908025Z * [new branch] manuel/np_empty_ellipsis -> origin/manuel/np_empty_ellipsis 2025-08-14T21:24:12.8908263Z * [new branch] manuel/test-ops-common-allow-mps -> origin/manuel/test-ops-common-allow-mps 2025-08-14T21:24:12.8908491Z * [new branch] metascroy-patch-1 -> origin/metascroy-patch-1 2025-08-14T21:24:12.8909420Z * [new branch] mlazos/S429861-debug -> origin/mlazos/S429861-debug 2025-08-14T21:24:12.8909668Z * [new branch] mlazos/aa -> origin/mlazos/aa 2025-08-14T21:24:12.8912425Z * [new branch] mlazos/arg-renames -> origin/mlazos/arg-renames 2025-08-14T21:24:12.8912944Z * [new branch] mlazos/backup-test-branch -> origin/mlazos/backup-test-branch 2025-08-14T21:24:12.8913245Z * [new branch] mlazos/bad-cudagraphs -> origin/mlazos/bad-cudagraphs 2025-08-14T21:24:12.8913434Z * [new branch] mlazos/baseline -> origin/mlazos/baseline 2025-08-14T21:24:12.8913616Z * [new branch] mlazos/baseline-graph-breaks -> origin/mlazos/baseline-graph-breaks 2025-08-14T21:24:12.8913865Z * [new branch] mlazos/beta-tensor -> origin/mlazos/beta-tensor 2025-08-14T21:24:12.8914136Z * [new branch] mlazos/buffers -> origin/mlazos/buffers 2025-08-14T21:24:12.8915387Z * [new branch] mlazos/buffers2 -> origin/mlazos/buffers2 2025-08-14T21:24:12.8915526Z * [new branch] mlazos/buffers3 -> origin/mlazos/buffers3 2025-08-14T21:24:12.8919013Z * [new branch] mlazos/ck2 -> origin/mlazos/ck2 2025-08-14T21:24:12.8921677Z * [new branch] mlazos/combokernels -> origin/mlazos/combokernels 2025-08-14T21:24:12.8921956Z * [new branch] mlazos/ctx-cleanup -> origin/mlazos/ctx-cleanup 2025-08-14T21:24:12.8922188Z * [new branch] mlazos/cudagraph-tests -> origin/mlazos/cudagraph-tests 2025-08-14T21:24:12.8922459Z * [new branch] mlazos/cudagraphs-measurement -> origin/mlazos/cudagraphs-measurement 2025-08-14T21:24:12.8922626Z * [new branch] mlazos/cutlass-test -> origin/mlazos/cutlass-test 2025-08-14T21:24:12.8922817Z * [new branch] mlazos/cutlass-topo-bug -> origin/mlazos/cutlass-topo-bug 2025-08-14T21:24:12.8922970Z * [new branch] mlazos/data-gather -> origin/mlazos/data-gather 2025-08-14T21:24:12.8923118Z * [new branch] mlazos/data-ptrs2 -> origin/mlazos/data-ptrs2 2025-08-14T21:24:12.8923329Z * [new branch] mlazos/data-ptrs3 -> origin/mlazos/data-ptrs3 2025-08-14T21:24:12.8923835Z * [new branch] mlazos/dataclass-proxy -> origin/mlazos/dataclass-proxy 2025-08-14T21:24:12.8924176Z * [new branch] mlazos/dc-attrs -> origin/mlazos/dc-attrs 2025-08-14T21:24:12.8924574Z * [new branch] mlazos/dc-helion -> origin/mlazos/dc-helion 2025-08-14T21:24:12.8925507Z * [new branch] mlazos/dict-fix -> origin/mlazos/dict-fix 2025-08-14T21:24:12.8926066Z * [new branch] mlazos/disable-closures -> origin/mlazos/disable-closures 2025-08-14T21:24:12.8926797Z * [new branch] mlazos/disable-tf -> origin/mlazos/disable-tf 2025-08-14T21:24:12.8927205Z * [new branch] mlazos/dupe-fix -> origin/mlazos/dupe-fix 2025-08-14T21:24:12.8929650Z * [new branch] mlazos/dyn-batch -> origin/mlazos/dyn-batch 2025-08-14T21:24:12.8929803Z * [new branch] mlazos/evt -> origin/mlazos/evt 2025-08-14T21:24:12.8929968Z * [new branch] mlazos/exp_disable -> origin/mlazos/exp_disable 2025-08-14T21:24:12.8930366Z * [new branch] mlazos/extract-examples -> origin/mlazos/extract-examples 2025-08-14T21:24:12.8932118Z * [new branch] mlazos/foreach-op -> origin/mlazos/foreach-op 2025-08-14T21:24:12.8932424Z * [new branch] mlazos/fp8 -> origin/mlazos/fp8 2025-08-14T21:24:12.8932598Z * [new branch] mlazos/fp8-bias -> origin/mlazos/fp8-bias 2025-08-14T21:24:12.8933064Z * [new branch] mlazos/fp8-bias-fusion -> origin/mlazos/fp8-bias-fusion 2025-08-14T21:24:12.8935017Z * [new branch] mlazos/freezing -> origin/mlazos/freezing 2025-08-14T21:24:12.8935310Z * [new branch] mlazos/h-comp -> origin/mlazos/h-comp 2025-08-14T21:24:12.8935469Z * [new branch] mlazos/h-comp2 -> origin/mlazos/h-comp2 2025-08-14T21:24:12.8935792Z * [new branch] mlazos/hash-hop -> origin/mlazos/hash-hop 2025-08-14T21:24:12.8936216Z * [new branch] mlazos/hc -> origin/mlazos/hc 2025-08-14T21:24:12.8937999Z * [new branch] mlazos/hc-cycles -> origin/mlazos/hc-cycles 2025-08-14T21:24:12.8938279Z * [new branch] mlazos/hc-fixes -> origin/mlazos/hc-fixes 2025-08-14T21:24:12.8945580Z * [new branch] mlazos/hc-fixes3 -> origin/mlazos/hc-fixes3 2025-08-14T21:24:12.8945880Z * [new branch] mlazos/hc-fixes4 -> origin/mlazos/hc-fixes4 2025-08-14T21:24:12.8946025Z * [new branch] mlazos/hc-hf -> origin/mlazos/hc-hf 2025-08-14T21:24:12.8946239Z * [new branch] mlazos/hc-mut -> origin/mlazos/hc-mut 2025-08-14T21:24:12.8946539Z * [new branch] mlazos/hc10 -> origin/mlazos/hc10 2025-08-14T21:24:12.8947832Z * [new branch] mlazos/hc11 -> origin/mlazos/hc11 2025-08-14T21:24:12.8948051Z * [new branch] mlazos/hc12 -> origin/mlazos/hc12 2025-08-14T21:24:12.8948430Z * [new branch] mlazos/hc13 -> origin/mlazos/hc13 2025-08-14T21:24:12.8950276Z * [new branch] mlazos/hc14 -> origin/mlazos/hc14 2025-08-14T21:24:12.8950568Z * [new branch] mlazos/hc15 -> origin/mlazos/hc15 2025-08-14T21:24:12.8950697Z * [new branch] mlazos/hc2 -> origin/mlazos/hc2 2025-08-14T21:24:12.8951043Z * [new branch] mlazos/hc4 -> origin/mlazos/hc4 2025-08-14T21:24:12.8953346Z * [new branch] mlazos/hc5 -> origin/mlazos/hc5 2025-08-14T21:24:12.8953630Z * [new branch] mlazos/hc6 -> origin/mlazos/hc6 2025-08-14T21:24:12.8953755Z * [new branch] mlazos/hc7 -> origin/mlazos/hc7 2025-08-14T21:24:12.8953898Z * [new branch] mlazos/hc8 -> origin/mlazos/hc8 2025-08-14T21:24:12.8955516Z * [new branch] mlazos/hc9 -> origin/mlazos/hc9 2025-08-14T21:24:12.8955857Z * [new branch] mlazos/hc_baseline2 -> origin/mlazos/hc_baseline2 2025-08-14T21:24:12.8956100Z * [new branch] mlazos/hop-modes -> origin/mlazos/hop-modes 2025-08-14T21:24:12.8958711Z * [new branch] mlazos/init-per-param -> origin/mlazos/init-per-param 2025-08-14T21:24:12.8959049Z * [new branch] mlazos/init_per_param -> origin/mlazos/init_per_param 2025-08-14T21:24:12.8959279Z * [new branch] mlazos/less-guards -> origin/mlazos/less-guards 2025-08-14T21:24:12.8961111Z * [new branch] mlazos/lr-composibility -> origin/mlazos/lr-composibility 2025-08-14T21:24:12.8961262Z * [new branch] mlazos/main -> origin/mlazos/main 2025-08-14T21:24:12.8961446Z * [new branch] mlazos/main-test-enablement -> origin/mlazos/main-test-enablement 2025-08-14T21:24:12.8961588Z * [new branch] mlazos/main2 -> origin/mlazos/main2 2025-08-14T21:24:12.8962124Z * [new branch] mlazos/mcg -> origin/mlazos/mcg 2025-08-14T21:24:12.8962281Z * [new branch] mlazos/mcg2 -> origin/mlazos/mcg2 2025-08-14T21:24:12.8962442Z * [new branch] mlazos/meta-guards -> origin/mlazos/meta-guards 2025-08-14T21:24:12.8963790Z * [new branch] mlazos/mlazos/ck2 -> origin/mlazos/mlazos/ck2 2025-08-14T21:24:12.8964058Z * [new branch] mlazos/mlazos/foreach-map-adam -> origin/mlazos/mlazos/foreach-map-adam 2025-08-14T21:24:12.8964754Z * [new branch] mlazos/mlazos/tf-mode-backup -> origin/mlazos/mlazos/tf-mode-backup 2025-08-14T21:24:12.8965434Z * [new branch] mlazos/mod-fix -> origin/mlazos/mod-fix 2025-08-14T21:24:12.8970031Z * [new branch] mlazos/mode-fix -> origin/mlazos/mode-fix 2025-08-14T21:24:12.8970383Z * [new branch] mlazos/more-tests -> origin/mlazos/more-tests 2025-08-14T21:24:12.8970631Z * [new branch] mlazos/nested-dc -> origin/mlazos/nested-dc 2025-08-14T21:24:12.8970786Z * [new branch] mlazos/no-cpp -> origin/mlazos/no-cpp 2025-08-14T21:24:12.8970970Z * [new branch] mlazos/no-init-group-handling -> origin/mlazos/no-init-group-handling 2025-08-14T21:24:12.8971105Z * [new branch] mlazos/offsets -> origin/mlazos/offsets 2025-08-14T21:24:12.8971378Z * [new branch] mlazos/opt-bench-exp2 -> origin/mlazos/opt-bench-exp2 2025-08-14T21:24:12.8971522Z * [new branch] mlazos/opt-incr -> origin/mlazos/opt-incr 2025-08-14T21:24:12.8971754Z * [new branch] mlazos/proxy-ctors -> origin/mlazos/proxy-ctors 2025-08-14T21:24:12.8972495Z * [new branch] mlazos/proxy-opt -> origin/mlazos/proxy-opt 2025-08-14T21:24:12.8977706Z * [new branch] mlazos/quant-fix -> origin/mlazos/quant-fix 2025-08-14T21:24:12.8978025Z * [new branch] mlazos/rm-buf-names -> origin/mlazos/rm-buf-names 2025-08-14T21:24:12.8978190Z * [new branch] mlazos/rm-spam -> origin/mlazos/rm-spam 2025-08-14T21:24:12.8978411Z * [new branch] mlazos/rtp -> origin/mlazos/rtp 2025-08-14T21:24:12.8978592Z * [new branch] mlazos/static-idx-dbg -> origin/mlazos/static-idx-dbg 2025-08-14T21:24:12.8978883Z * [new branch] mlazos/static-inputs-log -> origin/mlazos/static-inputs-log 2025-08-14T21:24:12.8979061Z * [new branch] mlazos/sub-param-fix -> origin/mlazos/sub-param-fix 2025-08-14T21:24:12.8979620Z * [new branch] mlazos/td-fix2 -> origin/mlazos/td-fix2 2025-08-14T21:24:12.8979989Z * [new branch] mlazos/tensor-hasattr2 -> origin/mlazos/tensor-hasattr2 2025-08-14T21:24:12.8980122Z * [new branch] mlazos/test -> origin/mlazos/test 2025-08-14T21:24:12.8980371Z * [new branch] mlazos/tf-mode -> origin/mlazos/tf-mode 2025-08-14T21:24:12.8980533Z * [new branch] mlazos/tf-mode-backup2 -> origin/mlazos/tf-mode-backup2 2025-08-14T21:24:12.8982096Z * [new branch] mlazos/tf-mode-reland -> origin/mlazos/tf-mode-reland 2025-08-14T21:24:12.8982423Z * [new branch] mlazos/tf-mode-reland2 -> origin/mlazos/tf-mode-reland2 2025-08-14T21:24:12.8982646Z * [new branch] mlazos/tf-mode-reland3 -> origin/mlazos/tf-mode-reland3 2025-08-14T21:24:12.8983003Z * [new branch] mlazos/topo-fix -> origin/mlazos/topo-fix 2025-08-14T21:24:12.8988447Z * [new branch] mlazos/triton-no-epi -> origin/mlazos/triton-no-epi 2025-08-14T21:24:12.8993650Z * [new branch] mlazos/tune-proto -> origin/mlazos/tune-proto 2025-08-14T21:24:12.8993829Z * [new branch] mlazos/tuple-fixes -> origin/mlazos/tuple-fixes 2025-08-14T21:24:12.8994305Z * [new branch] mlazos/tuple-fixes2 -> origin/mlazos/tuple-fixes2 2025-08-14T21:24:12.8994488Z * [new branch] mlazos/tuple-handling -> origin/mlazos/tuple-handling 2025-08-14T21:24:12.8994650Z * [new branch] mlazos/user-streams -> origin/mlazos/user-streams 2025-08-14T21:24:12.8994784Z * [new branch] mlazos/vary-beta -> origin/mlazos/vary-beta 2025-08-14T21:24:12.8994925Z * [new branch] mlazos/vary-beta2 -> origin/mlazos/vary-beta2 2025-08-14T21:24:12.8995062Z * [new branch] mlazos/weird-perf1 -> origin/mlazos/weird-perf1 2025-08-14T21:24:12.8995354Z * [new branch] mm_out_dtype_compile -> origin/mm_out_dtype_compile 2025-08-14T21:24:12.8995504Z * [new branch] modify-setupvllm -> origin/modify-setupvllm 2025-08-14T21:24:12.8995681Z * [new branch] move-theme-out-docker -> origin/move-theme-out-docker 2025-08-14T21:24:12.8995818Z * [new branch] mps-linear-1d -> origin/mps-linear-1d 2025-08-14T21:24:12.8995940Z * [new branch] msaroufim/be1 -> origin/msaroufim/be1 2025-08-14T21:24:12.8996075Z * [new branch] msaroufim/cn_path -> origin/msaroufim/cn_path 2025-08-14T21:24:12.8996244Z * [new branch] msaroufim/dtensorfusedadam -> origin/msaroufim/dtensorfusedadam 2025-08-14T21:24:12.8996379Z * [new branch] msaroufim/reduce -> origin/msaroufim/reduce 2025-08-14T21:24:12.8996516Z * [new branch] mtia/basic-cmake -> origin/mtia/basic-cmake 2025-08-14T21:24:12.8996631Z * [new branch] muon_dev -> origin/muon_dev 2025-08-14T21:24:12.8998288Z * [new branch] new-modifiy-setupvllm -> origin/new-modifiy-setupvllm 2025-08-14T21:24:12.8998454Z * [new branch] new-setupvllm -> origin/new-setupvllm 2025-08-14T21:24:12.8998591Z * [new branch] newtest-base -> origin/newtest-base 2025-08-14T21:24:12.9000615Z * [new branch] ngimel/cat_perf -> origin/ngimel/cat_perf 2025-08-14T21:24:12.9002369Z * [new branch] ngimel/cudamoduleload -> origin/ngimel/cudamoduleload 2025-08-14T21:24:12.9002747Z * [new branch] ngimel/fabric_driver_version -> origin/ngimel/fabric_driver_version 2025-08-14T21:24:12.9002905Z * [new branch] ngimel/fabric_symm -> origin/ngimel/fabric_symm 2025-08-14T21:24:12.9003042Z * [new branch] ngimel/gg_new -> origin/ngimel/gg_new 2025-08-14T21:24:12.9003225Z * [new branch] ngimel/grouped_mm_checks -> origin/ngimel/grouped_mm_checks 2025-08-14T21:24:12.9003520Z * [new branch] ngimel/guardfabric -> origin/ngimel/guardfabric 2025-08-14T21:24:12.9003928Z * [new branch] ngimel/index_None -> origin/ngimel/index_None 2025-08-14T21:24:12.9004875Z * [new branch] ngimel/modeguard -> origin/ngimel/modeguard 2025-08-14T21:24:12.9009376Z * [new branch] ngimel/multicast_fix -> origin/ngimel/multicast_fix 2025-08-14T21:24:12.9009578Z * [new branch] ngimel/unbind_multimem -> origin/ngimel/unbind_multimem 2025-08-14T21:24:12.9016093Z * [new branch] nightly -> origin/nightly 2025-08-14T21:24:12.9018360Z * [new branch] nmacchioni-patch-10 -> origin/nmacchioni-patch-10 2025-08-14T21:24:12.9018665Z * [new branch] nmacchioni-patch-7 -> origin/nmacchioni-patch-7 2025-08-14T21:24:12.9024735Z * [new branch] nmacchioni-patch-8 -> origin/nmacchioni-patch-8 2025-08-14T21:24:12.9026869Z * [new branch] nmacchioni-patch-9 -> origin/nmacchioni-patch-9 2025-08-14T21:24:12.9027169Z * [new branch] nullplay_fuse_matmul -> origin/nullplay_fuse_matmul 2025-08-14T21:24:12.9027609Z * [new branch] nweidia/enable-B200-inductor-nightly-ci -> origin/nweidia/enable-B200-inductor-nightly-ci 2025-08-14T21:24:12.9027730Z * [new branch] one-off -> origin/one-off 2025-08-14T21:24:12.9027950Z * [new branch] orig/release/1.10 -> origin/orig/release/1.10 2025-08-14T21:24:12.9033368Z * [new branch] orig/release/1.11 -> origin/orig/release/1.11 2025-08-14T21:24:12.9035486Z * [new branch] orig/release/1.12 -> origin/orig/release/1.12 2025-08-14T21:24:12.9035762Z * [new branch] orig/release/1.13 -> origin/orig/release/1.13 2025-08-14T21:24:12.9041417Z * [new branch] orig/release/1.6 -> origin/orig/release/1.6 2025-08-14T21:24:12.9041763Z * [new branch] orig/release/1.7 -> origin/orig/release/1.7 2025-08-14T21:24:12.9041911Z * [new branch] orig/release/1.8 -> origin/orig/release/1.8 2025-08-14T21:24:12.9042070Z * [new branch] orig/release/1.9 -> origin/orig/release/1.9 2025-08-14T21:24:12.9042216Z * [new branch] orig/release/2.0 -> origin/orig/release/2.0 2025-08-14T21:24:12.9042335Z * [new branch] orig/release/2.1 -> origin/orig/release/2.1 2025-08-14T21:24:12.9042566Z * [new branch] orig/release/2.2 -> origin/orig/release/2.2 2025-08-14T21:24:12.9043050Z * [new branch] orig/release/2.3 -> origin/orig/release/2.3 2025-08-14T21:24:12.9043225Z * [new branch] orig/release/2.4 -> origin/orig/release/2.4 2025-08-14T21:24:12.9043380Z * [new branch] orig/release/2.5 -> origin/orig/release/2.5 2025-08-14T21:24:12.9043521Z * [new branch] orig/release/2.6 -> origin/orig/release/2.6 2025-08-14T21:24:12.9043659Z * [new branch] orig/release/2.7 -> origin/orig/release/2.7 2025-08-14T21:24:12.9043786Z * [new branch] orig/release/2.8 -> origin/orig/release/2.8 2025-08-14T21:24:12.9043931Z * [new branch] oulgen/fx_graph -> origin/oulgen/fx_graph 2025-08-14T21:24:12.9044091Z * [new branch] padded-tensor -> origin/padded-tensor 2025-08-14T21:24:12.9044223Z * [new branch] parallel_cat -> origin/parallel_cat 2025-08-14T21:24:12.9044351Z * [new branch] pca2 -> origin/pca2 2025-08-14T21:24:12.9044500Z * [new branch] pianpwk-patch-1 -> origin/pianpwk-patch-1 2025-08-14T21:24:12.9044733Z * [new branch] pianpwk/backed_size_oblivious_export -> origin/pianpwk/backed_size_oblivious_export 2025-08-14T21:24:12.9045194Z * [new branch] pianpwk/dde_repeat_cat -> origin/pianpwk/dde_repeat_cat 2025-08-14T21:24:12.9045387Z * [new branch] pianpwk/draft_export_normalize -> origin/pianpwk/draft_export_normalize 2025-08-14T21:24:12.9045860Z * [new branch] pianpwk/dynamic_source_dim -> origin/pianpwk/dynamic_source_dim 2025-08-14T21:24:12.9046051Z * [new branch] pianpwk/invalidate_fake_memo -> origin/pianpwk/invalidate_fake_memo 2025-08-14T21:24:12.9046233Z * [new branch] pianpwk/lru_cache_bound_sympy -> origin/pianpwk/lru_cache_bound_sympy 2025-08-14T21:24:12.9046383Z * [new branch] pianpwk/max_1_strides -> origin/pianpwk/max_1_strides 2025-08-14T21:24:12.9046531Z * [new branch] pianpwk/nonzero_memo -> origin/pianpwk/nonzero_memo 2025-08-14T21:24:12.9046769Z * [new branch] pianpwk/oblivious_reshape_view_better -> origin/pianpwk/oblivious_reshape_view_better 2025-08-14T21:24:12.9046947Z * [new branch] pianpwk/oblivious_should_swap -> origin/pianpwk/oblivious_should_swap 2025-08-14T21:24:12.9047131Z * [new branch] pianpwk/oblivious_slice_forward -> origin/pianpwk/oblivious_slice_forward 2025-08-14T21:24:12.9047294Z * [new branch] pianpwk/oblivious_where -> origin/pianpwk/oblivious_where 2025-08-14T21:24:12.9047453Z * [new branch] pianpwk/param_static_pgo -> origin/pianpwk/param_static_pgo 2025-08-14T21:24:12.9047613Z * [new branch] pianpwk/pre_forward_hook -> origin/pianpwk/pre_forward_hook 2025-08-14T21:24:12.9047790Z * [new branch] pianpwk/remove_guard_fail_break -> origin/pianpwk/remove_guard_fail_break 2025-08-14T21:24:12.9047950Z * [new branch] pianpwk/slice_fresh_symbols -> origin/pianpwk/slice_fresh_symbols 2025-08-14T21:24:12.9048142Z * [new branch] pianpwk/sym_sym -> origin/pianpwk/sym_sym 2025-08-14T21:24:12.9048311Z * [new branch] pianpwk/test_slice_fake_impl -> origin/pianpwk/test_slice_fake_impl 2025-08-14T21:24:12.9048516Z * [new branch] pianpwk/unbacked_channels_last -> origin/pianpwk/unbacked_channels_last 2025-08-14T21:24:12.9048680Z * [new branch] pianpwk/unbacked_safe_conv1d -> origin/pianpwk/unbacked_safe_conv1d 2025-08-14T21:24:12.9048842Z * [new branch] pianpwk/unbacked_sdpa_flash -> origin/pianpwk/unbacked_sdpa_flash 2025-08-14T21:24:12.9049016Z * [new branch] pianpwk/unbacked_should_swap -> origin/pianpwk/unbacked_should_swap 2025-08-14T21:24:12.9049191Z * [new branch] pianpwk/unbacked_should_swap_2 -> origin/pianpwk/unbacked_should_swap_2 2025-08-14T21:24:12.9049370Z * [new branch] pianpwk/unbacked_slice_binding -> origin/pianpwk/unbacked_slice_binding 2025-08-14T21:24:12.9049552Z * [new branch] pianpwk/unbacked_slice_forward -> origin/pianpwk/unbacked_slice_forward 2025-08-14T21:24:12.9049723Z * [new branch] pianpwk/verbose_tensor_guards -> origin/pianpwk/verbose_tensor_guards 2025-08-14T21:24:12.9049880Z * [new branch] pianpwk/wan21_reshape -> origin/pianpwk/wan21_reshape 2025-08-14T21:24:12.9050047Z * [new branch] pianpwk/whitelist_optimizer -> origin/pianpwk/whitelist_optimizer 2025-08-14T21:24:12.9050182Z * [new branch] pin-torchao -> origin/pin-torchao 2025-08-14T21:24:12.9050679Z * [new branch] piz/fall_back_missing_0705 -> origin/piz/fall_back_missing_0705 2025-08-14T21:24:12.9050826Z * [new branch] piz/fall_back_missing_0716 -> origin/piz/fall_back_missing_0716 2025-08-14T21:24:12.9050980Z * [new branch] piz/fill_dist_cost_0702-3 -> origin/piz/fill_dist_cost_0702-3 2025-08-14T21:24:12.9051330Z * [new branch] piz/fill_dist_cost_0702-4 -> origin/piz/fill_dist_cost_0702-4 2025-08-14T21:24:12.9051835Z * [new branch] piz/fill_dist_cost_0702-5 -> origin/piz/fill_dist_cost_0702-5 2025-08-14T21:24:12.9057012Z * [new branch] piz/fix_sort_ -> origin/piz/fix_sort_ 2025-08-14T21:24:12.9059172Z * [new branch] piz/improve_scatter_0808 -> origin/piz/improve_scatter_0808 2025-08-14T21:24:12.9059440Z * [new branch] pool-separate -> origin/pool-separate 2025-08-14T21:24:12.9062626Z * [new branch] pr-156087 -> origin/pr-156087 2025-08-14T21:24:12.9062876Z * [new branch] pr/131860 -> origin/pr/131860 2025-08-14T21:24:12.9068198Z * [new branch] predispatch_to -> origin/predispatch_to 2025-08-14T21:24:12.9073093Z * [new branch] pt-opt-cuda3 -> origin/pt-opt-cuda3 2025-08-14T21:24:12.9073291Z * [new branch] pt2e-cache-model-device -> origin/pt2e-cache-model-device 2025-08-14T21:24:12.9073454Z * [new branch] pull-latest-theme -> origin/pull-latest-theme 2025-08-14T21:24:12.9073586Z * [new branch] pyobjectslot -> origin/pyobjectslot 2025-08-14T21:24:12.9073746Z * [new branch] python_compiled_autograd -> origin/python_compiled_autograd 2025-08-14T21:24:12.9073893Z * [new branch] qchip/export-D54134695 -> origin/qchip/export-D54134695 2025-08-14T21:24:12.9074013Z * [new branch] quint-bits -> origin/quint-bits 2025-08-14T21:24:12.9074152Z * [new branch] release/1.10 -> origin/release/1.10 2025-08-14T21:24:12.9074265Z * [new branch] release/1.11 -> origin/release/1.11 2025-08-14T21:24:12.9074386Z * [new branch] release/1.12 -> origin/release/1.12 2025-08-14T21:24:12.9074497Z * [new branch] release/1.13 -> origin/release/1.13 2025-08-14T21:24:12.9074768Z * [new branch] release/1.4 -> origin/release/1.4 2025-08-14T21:24:12.9074906Z * [new branch] release/1.4.1 -> origin/release/1.4.1 2025-08-14T21:24:12.9075023Z * [new branch] release/1.5 -> origin/release/1.5 2025-08-14T21:24:12.9075137Z * [new branch] release/1.6 -> origin/release/1.6 2025-08-14T21:24:12.9075260Z * [new branch] release/1.7 -> origin/release/1.7 2025-08-14T21:24:12.9075372Z * [new branch] release/1.8 -> origin/release/1.8 2025-08-14T21:24:12.9075488Z * [new branch] release/1.9 -> origin/release/1.9 2025-08-14T21:24:12.9075598Z * [new branch] release/2.0 -> origin/release/2.0 2025-08-14T21:24:12.9075709Z * [new branch] release/2.1 -> origin/release/2.1 2025-08-14T21:24:12.9075829Z * [new branch] release/2.2 -> origin/release/2.2 2025-08-14T21:24:12.9075943Z * [new branch] release/2.3 -> origin/release/2.3 2025-08-14T21:24:12.9076065Z * [new branch] release/2.4 -> origin/release/2.4 2025-08-14T21:24:12.9076183Z * [new branch] release/2.5 -> origin/release/2.5 2025-08-14T21:24:12.9076294Z * [new branch] release/2.6 -> origin/release/2.6 2025-08-14T21:24:12.9076409Z * [new branch] release/2.7 -> origin/release/2.7 2025-08-14T21:24:12.9076766Z * [new branch] release/2.8 -> origin/release/2.8 2025-08-14T21:24:12.9080076Z * [new branch] release_notes -> origin/release_notes 2025-08-14T21:24:12.9080415Z * [new branch] remove-actionable-label -> origin/remove-actionable-label 2025-08-14T21:24:12.9080571Z * [new branch] remove-ao -> origin/remove-ao 2025-08-14T21:24:12.9080805Z * [new branch] replace-pytorch-labs-20250812-195836 -> origin/replace-pytorch-labs-20250812-195836 2025-08-14T21:24:12.9081322Z * [new branch] replace-pytorch-labs-20250812-200248 -> origin/replace-pytorch-labs-20250812-200248 2025-08-14T21:24:12.9081990Z * [new branch] replace-pytorch-labs-20250812-200324 -> origin/replace-pytorch-labs-20250812-200324 2025-08-14T21:24:12.9082227Z * [new branch] replace-pytorch-labs-20250812-204020 -> origin/replace-pytorch-labs-20250812-204020 2025-08-14T21:24:12.9082635Z * [new branch] replace-pytorch-labs-20250812-204125 -> origin/replace-pytorch-labs-20250812-204125 2025-08-14T21:24:12.9084005Z * [new branch] replace-pytorch-labs-20250812-205624 -> origin/replace-pytorch-labs-20250812-205624 2025-08-14T21:24:12.9086779Z * [new branch] revert-131069-gh/krzysztofjordan/1/head -> origin/revert-131069-gh/krzysztofjordan/1/head 2025-08-14T21:24:12.9087197Z * [new branch] revert-131469-gh/andrewor14/51/head -> origin/revert-131469-gh/andrewor14/51/head 2025-08-14T21:24:12.9090212Z * [new branch] revert-156870-gh/skarjala/3/head -> origin/revert-156870-gh/skarjala/3/head 2025-08-14T21:24:12.9095026Z * [new branch] revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ -> origin/revert-157914-cherry-pick-157503-by-pytorch_bot_bot_ 2025-08-14T21:24:12.9099086Z * [new branch] revert-direct-updates -> origin/revert-direct-updates 2025-08-14T21:24:12.9099258Z * [new branch] rocm-monitoring -> origin/rocm-monitoring 2025-08-14T21:24:12.9099507Z * [new branch] ryanguo99/cleanup-dynamo-expected-failures -> origin/ryanguo99/cleanup-dynamo-expected-failures 2025-08-14T21:24:12.9099686Z * [new branch] ryanguo99/fix-closure-var -> origin/ryanguo99/fix-closure-var 2025-08-14T21:24:12.9099827Z * [new branch] rzou/faketensor_bench -> origin/rzou/faketensor_bench 2025-08-14T21:24:12.9100087Z * [new branch] rzou/njt -> origin/rzou/njt 2025-08-14T21:24:12.9100234Z * [new branch] rzou/operator -> origin/rzou/operator 2025-08-14T21:24:12.9100343Z * [new branch] rzou/pca -> origin/rzou/pca 2025-08-14T21:24:12.9100466Z * [new branch] rzou/pipe_split -> origin/rzou/pipe_split 2025-08-14T21:24:12.9100587Z * [new branch] rzou/realprop -> origin/rzou/realprop 2025-08-14T21:24:12.9100715Z * [new branch] rzou/setup_context -> origin/rzou/setup_context 2025-08-14T21:24:12.9100930Z * [new branch] sanchitintel/refactor_aten_int8_woq_gemm -> origin/sanchitintel/refactor_aten_int8_woq_gemm 2025-08-14T21:24:12.9101204Z * [new branch] sanchitintel/weird_thing_with_test_cpu_select_algorithm -> origin/sanchitintel/weird_thing_with_test_cpu_select_algorithm 2025-08-14T21:24:12.9101366Z * [new branch] sapling-pr-archive-SS-JIA -> origin/sapling-pr-archive-SS-JIA 2025-08-14T21:24:12.9101484Z * [new branch] save -> origin/save 2025-08-14T21:24:12.9101595Z * [new branch] sdym/2.5.1 -> origin/sdym/2.5.1 2025-08-14T21:24:12.9102510Z * [new branch] seemethere-patch-1 -> origin/seemethere-patch-1 2025-08-14T21:24:12.9102681Z * [new branch] setup-torchci -> origin/setup-torchci 2025-08-14T21:24:12.9102810Z * [new branch] setupvllm -> origin/setupvllm 2025-08-14T21:24:12.9102940Z * [new branch] share_and_pin_fork -> origin/share_and_pin_fork 2025-08-14T21:24:12.9103171Z * [new branch] shengf/fx-xform-perf -> origin/shengf/fx-xform-perf 2025-08-14T21:24:12.9103378Z * [new branch] shikaili_fp8_allgather -> origin/shikaili_fp8_allgather 2025-08-14T21:24:12.9110483Z * [new branch] shoumikhin-patch-12 -> origin/shoumikhin-patch-12 2025-08-14T21:24:12.9116019Z * [new branch] simplify-fq-per-channel -> origin/simplify-fq-per-channel 2025-08-14T21:24:12.9120628Z * [new branch] solve-accuracy-fix -> origin/solve-accuracy-fix 2025-08-14T21:24:12.9120810Z * [new branch] sqzhang/flight4 -> origin/sqzhang/flight4 2025-08-14T21:24:12.9120959Z * [new branch] sqzhang/flight4plus -> origin/sqzhang/flight4plus 2025-08-14T21:24:12.9121128Z * [new branch] sraikund/record_funct_test -> origin/sraikund/record_funct_test 2025-08-14T21:24:12.9121255Z * [new branch] sraikund16/test -> origin/sraikund16/test 2025-08-14T21:24:12.9121429Z * [new branch] stablize-compilation-time -> origin/stablize-compilation-time 2025-08-14T21:24:12.9121576Z * [new branch] standalone-templates -> origin/standalone-templates 2025-08-14T21:24:12.9121739Z * [new branch] standalone_package_weights -> origin/standalone_package_weights 2025-08-14T21:24:12.9121890Z * [new branch] starterTaskUpdate -> origin/starterTaskUpdate 2025-08-14T21:24:12.9122016Z * [new branch] step2vllmsetup -> origin/step2vllmsetup 2025-08-14T21:24:12.9122140Z * [new branch] subgraph_fuse -> origin/subgraph_fuse 2025-08-14T21:24:12.9122295Z * [new branch] support-uv-in-collect_env -> origin/support-uv-in-collect_env 2025-08-14T21:24:12.9122439Z * [new branch] suryasub/fix-nccl-hang -> origin/suryasub/fix-nccl-hang 2025-08-14T21:24:12.9122565Z * [new branch] sve-poc -> origin/sve-poc 2025-08-14T21:24:12.9122698Z * [new branch] svekars-patch-1 -> origin/svekars-patch-1 2025-08-14T21:24:12.9122830Z * [new branch] svekars-patch-2 -> origin/svekars-patch-2 2025-08-14T21:24:12.9123090Z * [new branch] switch-bn -> origin/switch-bn 2025-08-14T21:24:12.9123256Z * [new branch] sympy-bottleneck-repro -> origin/sympy-bottleneck-repro 2025-08-14T21:24:12.9123434Z * [new branch] tenpercent/ck_inductor_gfx950 -> origin/tenpercent/ck_inductor_gfx950 2025-08-14T21:24:12.9123576Z * [new branch] tensordict_integration -> origin/tensordict_integration 2025-08-14T21:24:12.9123762Z * [new branch] test-half-migration-internally -> origin/test-half-migration-internally 2025-08-14T21:24:12.9123904Z * [new branch] test-internal-et -> origin/test-internal-et 2025-08-14T21:24:12.9124050Z * [new branch] test-move-conda-builds -> origin/test-move-conda-builds 2025-08-14T21:24:12.9124242Z * [new branch] test-myst-markdown-docstring -> origin/test-myst-markdown-docstring 2025-08-14T21:24:12.9124361Z * [new branch] test-old -> origin/test-old 2025-08-14T21:24:12.9124536Z * [new branch] test-vec-migration-internally -> origin/test-vec-migration-internally 2025-08-14T21:24:12.9124657Z * [new branch] test/bmm_heur -> origin/test/bmm_heur 2025-08-14T21:24:12.9124776Z * [new branch] test/inductor -> origin/test/inductor 2025-08-14T21:24:12.9124919Z * [new branch] tidy_performance_cyy -> origin/tidy_performance_cyy 2025-08-14T21:24:12.9125096Z * [new branch] torchtitan_ep -> origin/torchtitan_ep 2025-08-14T21:24:12.9127821Z * [new branch] trace_fsdp_torchtune_lora -> origin/trace_fsdp_torchtune_lora 2025-08-14T21:24:12.9132817Z * [new branch] traceable_fsdp_unit_tests -> origin/traceable_fsdp_unit_tests 2025-08-14T21:24:12.9133016Z * [new branch] trackMonitor -> origin/trackMonitor 2025-08-14T21:24:12.9133167Z * [new branch] tree_loop_vec_base -> origin/tree_loop_vec_base 2025-08-14T21:24:12.9133437Z * [new branch] tree_vec_base -> origin/tree_vec_base 2025-08-14T21:24:12.9133712Z * [new branch] triton-update -> origin/triton-update 2025-08-14T21:24:12.9133934Z * [new branch] triton_kernel -> origin/triton_kernel 2025-08-14T21:24:12.9134063Z * [new branch] triton_kernel_perf -> origin/triton_kernel_perf 2025-08-14T21:24:12.9134311Z * [new branch] try-runllm -> origin/try-runllm 2025-08-14T21:24:12.9134502Z * [new branch] type_dec -> origin/type_dec 2025-08-14T21:24:12.9134667Z * [new branch] udate-sphinx-dependancies -> origin/udate-sphinx-dependancies 2025-08-14T21:24:12.9134910Z * [new branch] update-audio-commit-hash/16307312222-1661-1 -> origin/update-audio-commit-hash/16307312222-1661-1 2025-08-14T21:24:12.9135138Z * [new branch] update-audio-commit-hash/16431348808-1673-1 -> origin/update-audio-commit-hash/16431348808-1673-1 2025-08-14T21:24:12.9135360Z * [new branch] update-audio-commit-hash/16510774365-1683-1 -> origin/update-audio-commit-hash/16510774365-1683-1 2025-08-14T21:24:12.9137154Z * [new branch] update-audio-commit-hash/16583472358-1693-1 -> origin/update-audio-commit-hash/16583472358-1693-1 2025-08-14T21:24:12.9137559Z * [new branch] update-audio-commit-hash/16663082088-1700-1 -> origin/update-audio-commit-hash/16663082088-1700-1 2025-08-14T21:24:12.9137961Z * [new branch] update-audio-commit-hash/16737365217-1704-1 -> origin/update-audio-commit-hash/16737365217-1704-1 2025-08-14T21:24:12.9138556Z * [new branch] update-audio-commit-hash/16791960928-1711-1 -> origin/update-audio-commit-hash/16791960928-1711-1 2025-08-14T21:24:12.9144533Z * [new branch] update-audio-commit-hash/16818882925-1712-1 -> origin/update-audio-commit-hash/16818882925-1712-1 2025-08-14T21:24:12.9145188Z * [new branch] update-audio-commit-hash/16895560422-1720-1 -> origin/update-audio-commit-hash/16895560422-1720-1 2025-08-14T21:24:12.9145572Z * [new branch] update-audio-commit-hash/16924174496-1738-1 -> origin/update-audio-commit-hash/16924174496-1738-1 2025-08-14T21:24:12.9145762Z * [new branch] update-dynamic-shapes-doc -> origin/update-dynamic-shapes-doc 2025-08-14T21:24:12.9146067Z * [new branch] update-executorch-commit-hash/15694981040-1626-1 -> origin/update-executorch-commit-hash/15694981040-1626-1 2025-08-14T21:24:12.9146290Z * [new branch] update-triton-commit-hash/13663274526-1487-2 -> origin/update-triton-commit-hash/13663274526-1487-2 2025-08-14T21:24:12.9146523Z * [new branch] update-vision-commit-hash/15336342773-1607-1 -> origin/update-vision-commit-hash/15336342773-1607-1 2025-08-14T21:24:12.9146735Z * [new branch] update-vllm-commit-hash/16431348808-1673-1 -> origin/update-vllm-commit-hash/16431348808-1673-1 2025-08-14T21:24:12.9146950Z * [new branch] update-vllm-commit-hash/16484773233-1682-1 -> origin/update-vllm-commit-hash/16484773233-1682-1 2025-08-14T21:24:12.9147154Z * [new branch] update-vllm-commit-hash/16510774365-1683-1 -> origin/update-vllm-commit-hash/16510774365-1683-1 2025-08-14T21:24:12.9147357Z * [new branch] update-vllm-commit-hash/16534031105-1684-1 -> origin/update-vllm-commit-hash/16534031105-1684-1 2025-08-14T21:24:12.9149523Z * [new branch] update-vllm-commit-hash/16545403308-1687-1 -> origin/update-vllm-commit-hash/16545403308-1687-1 2025-08-14T21:24:12.9149809Z * [new branch] update-vllm-commit-hash/16557202787-1688-1 -> origin/update-vllm-commit-hash/16557202787-1688-1 2025-08-14T21:24:12.9150049Z * [new branch] update-vllm-commit-hash/16583472358-1693-1 -> origin/update-vllm-commit-hash/16583472358-1693-1 2025-08-14T21:24:12.9150261Z * [new branch] update-vllm-commit-hash/16663082088-1700-1 -> origin/update-vllm-commit-hash/16663082088-1700-1 2025-08-14T21:24:12.9150544Z * [new branch] update-vllm-commit-hash/16737365217-1704-1 -> origin/update-vllm-commit-hash/16737365217-1704-1 2025-08-14T21:24:12.9150868Z * [new branch] update-vllm-commit-hash/16843157111-1713-1 -> origin/update-vllm-commit-hash/16843157111-1713-1 2025-08-14T21:24:12.9151426Z * [new branch] update-vllm-commit-hash/16855312394-1714-1 -> origin/update-vllm-commit-hash/16855312394-1714-1 2025-08-14T21:24:12.9151674Z * [new branch] update-vllm-commit-hash/16924174496-1738-1 -> origin/update-vllm-commit-hash/16924174496-1738-1 2025-08-14T21:24:12.9152063Z * [new branch] update-vllm-commit-hash/16952608705-1745-1 -> origin/update-vllm-commit-hash/16952608705-1745-1 2025-08-14T21:24:12.9152544Z * [new branch] update-xla-commit-hash/16260974441-194-1 -> origin/update-xla-commit-hash/16260974441-194-1 2025-08-14T21:24:12.9153425Z * [new branch] update-xla-commit-hash/16717126778-197-1 -> origin/update-xla-commit-hash/16717126778-197-1 2025-08-14T21:24:12.9153828Z * [new branch] update-xla-commit-hash/16873912760-198-1 -> origin/update-xla-commit-hash/16873912760-198-1 2025-08-14T21:24:12.9157385Z * [new branch] update_docs_torch_multinomial_issue#125388 -> origin/update_docs_torch_multinomial_issue#125388 2025-08-14T21:24:12.9157729Z * [new branch] update_executorch_pin -> origin/update_executorch_pin 2025-08-14T21:24:12.9157989Z * [new branch] update_slow_tests_1722488736 -> origin/update_slow_tests_1722488736 2025-08-14T21:24:12.9158168Z * [new branch] update_slow_tests_1722879173 -> origin/update_slow_tests_1722879173 2025-08-14T21:24:12.9158322Z * [new branch] update_slow_tests_1752478971 -> origin/update_slow_tests_1752478971 2025-08-14T21:24:12.9158784Z * [new branch] update_submodule_FBGEMM -> origin/update_submodule_FBGEMM 2025-08-14T21:24:12.9158960Z * [new branch] update_submodule_kineto -> origin/update_submodule_kineto 2025-08-14T21:24:12.9161109Z * [new branch] update_submodule_tensorpipe -> origin/update_submodule_tensorpipe 2025-08-14T21:24:12.9161310Z * [new branch] v0.1.2 -> origin/v0.1.2 2025-08-14T21:24:12.9161481Z * [new branch] v1.0.1 -> origin/v1.0.1 2025-08-14T21:24:12.9161604Z * [new branch] v1.0.3 -> origin/v1.0.3 2025-08-14T21:24:12.9163475Z * [new branch] v1.1.0 -> origin/v1.1.0 2025-08-14T21:24:12.9163617Z * [new branch] v1.2.0 -> origin/v1.2.0 2025-08-14T21:24:12.9164160Z * [new branch] v1.3.0 -> origin/v1.3.0 2025-08-14T21:24:12.9165736Z * [new branch] v1.3.1 -> origin/v1.3.1 2025-08-14T21:24:12.9166071Z * [new branch] validate_fn -> origin/validate_fn 2025-08-14T21:24:12.9166593Z * [new branch] validations_2.6 -> origin/validations_2.6 2025-08-14T21:24:12.9169651Z * [new branch] validations_2.8 -> origin/validations_2.8 2025-08-14T21:24:12.9169824Z * [new branch] viable/strict -> origin/viable/strict 2025-08-14T21:24:12.9169950Z * [new branch] vllmbuildci -> origin/vllmbuildci 2025-08-14T21:24:12.9170078Z * [new branch] vllmpin -> origin/vllmpin 2025-08-14T21:24:12.9171974Z * [new branch] vllmpintest -> origin/vllmpintest 2025-08-14T21:24:12.9172285Z * [new branch] wdvr-patch-1 -> origin/wdvr-patch-1 2025-08-14T21:24:12.9172429Z * [new branch] wdvr-patch-2 -> origin/wdvr-patch-2 2025-08-14T21:24:12.9174445Z * [new branch] wdvr/conda_devcontainer -> origin/wdvr/conda_devcontainer 2025-08-14T21:24:12.9175027Z * [new branch] wdvr/fix_logging_test -> origin/wdvr/fix_logging_test 2025-08-14T21:24:12.9175185Z * [new branch] wdvr/iss_145259 -> origin/wdvr/iss_145259 2025-08-14T21:24:12.9175463Z * [new branch] weight_sharing_cpp -> origin/weight_sharing_cpp 2025-08-14T21:24:12.9177279Z * [new branch] whc/flight -> origin/whc/flight 2025-08-14T21:24:12.9177569Z * [new branch] whc/flight4 -> origin/whc/flight4 2025-08-14T21:24:12.9177747Z * [new branch] whc/flight51 -> origin/whc/flight51 2025-08-14T21:24:12.9179688Z * [new branch] whc/flight53 -> origin/whc/flight53 2025-08-14T21:24:12.9179855Z * [new branch] whc/p2phang -> origin/whc/p2phang 2025-08-14T21:24:12.9179993Z * [new branch] whc/stage2 -> origin/whc/stage2 2025-08-14T21:24:12.9180479Z * [new branch] whc/uneven -> origin/whc/uneven 2025-08-14T21:24:12.9184689Z * [new branch] whc/uneven-merge -> origin/whc/uneven-merge 2025-08-14T21:24:12.9184843Z * [new branch] win_warnings -> origin/win_warnings 2025-08-14T21:24:12.9184984Z * [new branch] workonoldcommit -> origin/workonoldcommit 2025-08-14T21:24:12.9185175Z * [new branch] wwen/programming-model-2.8 -> origin/wwen/programming-model-2.8 2025-08-14T21:24:12.9185301Z * [new branch] xmfan/ca_0516 -> origin/xmfan/ca_0516 2025-08-14T21:24:12.9185439Z * [new branch] xmfan/ca_1051b93192 -> origin/xmfan/ca_1051b93192 2025-08-14T21:24:12.9185740Z * [new branch] xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 -> origin/xmfan/ca_1a722f62c248391fc4a542e8851a5559aa356ae8 2025-08-14T21:24:12.9186365Z * [new branch] xmfan/ca_5a2be192d1 -> origin/xmfan/ca_5a2be192d1 2025-08-14T21:24:12.9186750Z * [new branch] xmfan/ca_9d59b516e9 -> origin/xmfan/ca_9d59b516e9 2025-08-14T21:24:12.9187591Z * [new branch] xmfan/ca_api -> origin/xmfan/ca_api 2025-08-14T21:24:12.9187827Z * [new branch] xmfan/ca_apr8 -> origin/xmfan/ca_apr8 2025-08-14T21:24:12.9193072Z * [new branch] xmfan/ca_base -> origin/xmfan/ca_base 2025-08-14T21:24:12.9193247Z * [new branch] xmfan/ca_cudagraphs -> origin/xmfan/ca_cudagraphs 2025-08-14T21:24:12.9193387Z * [new branch] xmfan/ca_dynamic -> origin/xmfan/ca_dynamic 2025-08-14T21:24:12.9193505Z * [new branch] xmfan/ca_fix_dyn -> origin/xmfan/ca_fix_dyn 2025-08-14T21:24:12.9193652Z * [new branch] xmfan/ca_fix_lowering -> origin/xmfan/ca_fix_lowering 2025-08-14T21:24:12.9198117Z * [new branch] xmfan/ca_fix_polyfills -> origin/xmfan/ca_fix_polyfills 2025-08-14T21:24:12.9198276Z * [new branch] xmfan/ca_jan3 -> origin/xmfan/ca_jan3 2025-08-14T21:24:12.9198412Z * [new branch] xmfan/ca_jun18 -> origin/xmfan/ca_jun18 2025-08-14T21:24:12.9198529Z * [new branch] xmfan/ca_jun24 -> origin/xmfan/ca_jun24 2025-08-14T21:24:12.9198664Z * [new branch] xmfan/ca_mem_base -> origin/xmfan/ca_mem_base 2025-08-14T21:24:12.9198792Z * [new branch] xmfan/ca_mem_fix -> origin/xmfan/ca_mem_fix 2025-08-14T21:24:12.9198921Z * [new branch] xmfan/ca_memory_fix -> origin/xmfan/ca_memory_fix 2025-08-14T21:24:12.9199081Z * [new branch] xmfan/ca_memory_fix_rebased -> origin/xmfan/ca_memory_fix_rebased 2025-08-14T21:24:12.9199233Z * [new branch] xmfan/ca_memory_fix_rebased2 -> origin/xmfan/ca_memory_fix_rebased2 2025-08-14T21:24:12.9199858Z * [new branch] xmfan/ca_move_to_cuda -> origin/xmfan/ca_move_to_cuda 2025-08-14T21:24:12.9200169Z * [new branch] xmfan/ca_nested -> origin/xmfan/ca_nested 2025-08-14T21:24:12.9200309Z * [new branch] xmfan/ca_overhead -> origin/xmfan/ca_overhead 2025-08-14T21:24:12.9200506Z * [new branch] xmfan/ca_overhead_0eba7e5451 -> origin/xmfan/ca_overhead_0eba7e5451 2025-08-14T21:24:12.9200645Z * [new branch] xmfan/ca_scalar -> origin/xmfan/ca_scalar 2025-08-14T21:24:12.9200813Z * [new branch] xmfan/ca_subclass_mem_fix -> origin/xmfan/ca_subclass_mem_fix 2025-08-14T21:24:12.9201068Z * [new branch] xmfan/ca_warm_mem -> origin/xmfan/ca_warm_mem 2025-08-14T21:24:12.9201490Z * [new branch] xmfan/ca_warm_mem_base -> origin/xmfan/ca_warm_mem_base 2025-08-14T21:24:12.9202380Z * [new branch] xmfan/cacu_jun18 -> origin/xmfan/cacu_jun18 2025-08-14T21:24:12.9202772Z * [new branch] xmfan/cacu_jun19 -> origin/xmfan/cacu_jun19 2025-08-14T21:24:12.9205422Z * [new branch] xmfan/cacu_jun4 -> origin/xmfan/cacu_jun4 2025-08-14T21:24:12.9205839Z * [new branch] xmfan/cacu_may27 -> origin/xmfan/cacu_may27 2025-08-14T21:24:12.9206017Z * [new branch] xmfan/circular_dep -> origin/xmfan/circular_dep 2025-08-14T21:24:12.9206211Z * [new branch] xmfan/compiled_autograd_feb_29 -> origin/xmfan/compiled_autograd_feb_29 2025-08-14T21:24:12.9208976Z * [new branch] xmfan/compiled_autograd_graph_breaks -> origin/xmfan/compiled_autograd_graph_breaks 2025-08-14T21:24:12.9209140Z * [new branch] xmfan/disable_duck_shape -> origin/xmfan/disable_duck_shape 2025-08-14T21:24:12.9209341Z * [new branch] xmfan/fca_cpp_node_passthrough -> origin/xmfan/fca_cpp_node_passthrough 2025-08-14T21:24:12.9209645Z * [new branch] xmfan/issue_123374 -> origin/xmfan/issue_123374 2025-08-14T21:24:12.9209968Z * [new branch] xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/post_3945954741e2d37023c5d6954f9483008e0892f9 2025-08-14T21:24:12.9214384Z * [new branch] xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 -> origin/xmfan/pre_3945954741e2d37023c5d6954f9483008e0892f9 2025-08-14T21:24:12.9214659Z * [new branch] xmfan/segfault_test -> origin/xmfan/segfault_test 2025-08-14T21:24:12.9218615Z * [new branch] xmfan/single_step -> origin/xmfan/single_step 2025-08-14T21:24:12.9218865Z * [new branch] xmfan/sth_0829 -> origin/xmfan/sth_0829 2025-08-14T21:24:12.9224383Z * [new branch] xmfan/test -> origin/xmfan/test 2025-08-14T21:24:12.9228588Z * [new branch] y-do-we-have-7-build-systems -> origin/y-do-we-have-7-build-systems 2025-08-14T21:24:12.9232725Z * [new branch] yguo/debug-0226-constexpr -> origin/yguo/debug-0226-constexpr 2025-08-14T21:24:12.9232909Z * [new branch] yguo/new_latest_changes -> origin/yguo/new_latest_changes 2025-08-14T21:24:12.9233104Z * [new branch] yguo/patch_constexpr_changes -> origin/yguo/patch_constexpr_changes 2025-08-14T21:24:12.9233246Z * [new branch] yihan_quantization -> origin/yihan_quantization 2025-08-14T21:24:12.9233406Z * [new branch] yiming/add_nativert_benchmark -> origin/yiming/add_nativert_benchmark 2025-08-14T21:24:12.9233530Z * [new branch] yiming/bootcamp -> origin/yiming/bootcamp 2025-08-14T21:24:12.9233670Z * [new branch] zainr/canary-test -> origin/zainr/canary-test 2025-08-14T21:24:12.9233832Z * [new branch] zainr/cleanup-gh-runners -> origin/zainr/cleanup-gh-runners 2025-08-14T21:24:12.9233955Z * [new branch] zainr/fixlint -> origin/zainr/fixlint 2025-08-14T21:24:12.9234235Z * [new branch] zainr/git-push-v2 -> origin/zainr/git-push-v2 2025-08-14T21:24:12.9234364Z * [new branch] zainr/lint-py3.9 -> origin/zainr/lint-py3.9 2025-08-14T21:24:12.9234511Z * [new branch] zainr/mypy15-claude -> origin/zainr/mypy15-claude 2025-08-14T21:24:12.9234658Z * [new branch] zainr/pre-push-hooks -> origin/zainr/pre-push-hooks 2025-08-14T21:24:12.9234814Z * [new branch] zainr/pull-migration-c -> origin/zainr/pull-migration-c 2025-08-14T21:24:12.9234939Z * [new branch] zainr/test2 -> origin/zainr/test2 2025-08-14T21:24:12.9235058Z * [new branch] zainr/unstable -> origin/zainr/unstable 2025-08-14T21:24:12.9235196Z * [new branch] zainr/unstable-xla -> origin/zainr/unstable-xla 2025-08-14T21:24:12.9235320Z * [new branch] zainr/uv-pip-fix -> origin/zainr/uv-pip-fix 2025-08-14T21:24:12.9235452Z * [new branch] zainr/vs-aarch64 -> origin/zainr/vs-aarch64 2025-08-14T21:24:12.9235588Z * [new branch] zasdfgbnm-patch-3 -> origin/zasdfgbnm-patch-3 2025-08-14T21:24:12.9235692Z * [new branch] zb2p -> origin/zb2p 2025-08-14T21:24:12.9235824Z * [new branch] zdevito-patch-1 -> origin/zdevito-patch-1 2025-08-14T21:24:12.9235971Z * [new branch] zeros-and-scatter-part2 -> origin/zeros-and-scatter-part2 2025-08-14T21:24:12.9236107Z * [new branch] zhxchen17/nativert/0 -> origin/zhxchen17/nativert/0 2025-08-14T21:24:12.9236243Z * [new branch] zhxchen17/scratch/0 -> origin/zhxchen17/scratch/0 2025-08-14T21:24:12.9236376Z * [new branch] zhxhcen17/moodycamel -> origin/zhxhcen17/moodycamel 2025-08-14T21:24:12.9236503Z * [new branch] zxiiro/bazel -> origin/zxiiro/bazel 2025-08-14T21:24:12.9237366Z * [new branch] zxiiro/get-hardware -> origin/zxiiro/get-hardware 2025-08-14T21:24:12.9237488Z * [new branch] zxiiro/main -> origin/zxiiro/main 2025-08-14T21:24:12.9237731Z * [new branch] zxiiro/test -> origin/zxiiro/test 2025-08-14T21:24:12.9238026Z * [new tag] bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug -> bc2caa7fdf006894eff7af936babde69ab5a40f8-huydhn-debug 2025-08-14T21:24:12.9238150Z * [new tag] ci/binaries/77164 -> ci/binaries/77164 2025-08-14T21:24:12.9238269Z * [new tag] ciflow/binaries/138996 -> ciflow/binaries/138996 2025-08-14T21:24:12.9238752Z * [new tag] ciflow/binaries/143959 -> ciflow/binaries/143959 2025-08-14T21:24:12.9239047Z * [new tag] ciflow/binaries/154595 -> ciflow/binaries/154595 2025-08-14T21:24:12.9239556Z * [new tag] ciflow/binaries/156049 -> ciflow/binaries/156049 2025-08-14T21:24:12.9239917Z * [new tag] ciflow/binaries/156712 -> ciflow/binaries/156712 2025-08-14T21:24:12.9240380Z * [new tag] ciflow/binaries/157432 -> ciflow/binaries/157432 2025-08-14T21:24:12.9240743Z * [new tag] ciflow/binaries/157685 -> ciflow/binaries/157685 2025-08-14T21:24:12.9241271Z * [new tag] ciflow/binaries/157689 -> ciflow/binaries/157689 2025-08-14T21:24:12.9241980Z * [new tag] ciflow/binaries/158104 -> ciflow/binaries/158104 2025-08-14T21:24:12.9242434Z * [new tag] ciflow/binaries/158623 -> ciflow/binaries/158623 2025-08-14T21:24:12.9242828Z * [new tag] ciflow/binaries/159827 -> ciflow/binaries/159827 2025-08-14T21:24:12.9243750Z * [new tag] ciflow/binaries/159869 -> ciflow/binaries/159869 2025-08-14T21:24:12.9246474Z * [new tag] ciflow/binaries/160593 -> ciflow/binaries/160593 2025-08-14T21:24:12.9246963Z * [new tag] ciflow/binaries_libtorch/143959 -> ciflow/binaries_libtorch/143959 2025-08-14T21:24:12.9247157Z * [new tag] ciflow/binaries_libtorch/156049 -> ciflow/binaries_libtorch/156049 2025-08-14T21:24:12.9247387Z * [new tag] ciflow/binaries_libtorch/157432 -> ciflow/binaries_libtorch/157432 2025-08-14T21:24:12.9247542Z * [new tag] ciflow/binaries_wheel/143959 -> ciflow/binaries_wheel/143959 2025-08-14T21:24:12.9247677Z * [new tag] ciflow/binaries_wheel/156049 -> ciflow/binaries_wheel/156049 2025-08-14T21:24:12.9247805Z * [new tag] ciflow/binaries_wheel/157432 -> ciflow/binaries_wheel/157432 2025-08-14T21:24:12.9248067Z * [new tag] ciflow/binaries_wheel/158733 -> ciflow/binaries_wheel/158733 2025-08-14T21:24:12.9248210Z * [new tag] ciflow/binaries_wheel/160301 -> ciflow/binaries_wheel/160301 2025-08-14T21:24:12.9248345Z * [new tag] ciflow/binaries_wheel/160496 -> ciflow/binaries_wheel/160496 2025-08-14T21:24:12.9249878Z * [new tag] ciflow/h100-distributed/156703 -> ciflow/h100-distributed/156703 2025-08-14T21:24:12.9250079Z * [new tag] ciflow/h100-symm-mem/151845 -> ciflow/h100-symm-mem/151845 2025-08-14T21:24:12.9250354Z * [new tag] ciflow/h100-symm-mem/155923 -> ciflow/h100-symm-mem/155923 2025-08-14T21:24:12.9250798Z * [new tag] ciflow/h100-symm-mem/157635 -> ciflow/h100-symm-mem/157635 2025-08-14T21:24:12.9251388Z * [new tag] ciflow/h100-symm-mem/159118 -> ciflow/h100-symm-mem/159118 2025-08-14T21:24:12.9252000Z * [new tag] ciflow/h100-symm-mem/159562 -> ciflow/h100-symm-mem/159562 2025-08-14T21:24:12.9252316Z * [new tag] ciflow/h100-symm-mem/159889 -> ciflow/h100-symm-mem/159889 2025-08-14T21:24:12.9255060Z * [new tag] ciflow/h100/159158 -> ciflow/h100/159158 2025-08-14T21:24:12.9255401Z * [new tag] ciflow/h100/160450 -> ciflow/h100/160450 2025-08-14T21:24:12.9255536Z * [new tag] ciflow/h100/160480 -> ciflow/h100/160480 2025-08-14T21:24:12.9255639Z * [new tag] ciflow/h100/160614 -> ciflow/h100/160614 2025-08-14T21:24:12.9255901Z * [new tag] ciflow/inductor-perf-test-nightly-rocm/151845 -> ciflow/inductor-perf-test-nightly-rocm/151845 2025-08-14T21:24:12.9256132Z * [new tag] ciflow/inductor-perf-test-nightly-rocm/160538 -> ciflow/inductor-perf-test-nightly-rocm/160538 2025-08-14T21:24:12.9256712Z * [new tag] ciflow/inductor-perf-test-nightly-x86-zen/156599 -> ciflow/inductor-perf-test-nightly-x86-zen/156599 2025-08-14T21:24:12.9257962Z * [new tag] ciflow/inductor-periodic/160406 -> ciflow/inductor-periodic/160406 2025-08-14T21:24:12.9258174Z * [new tag] ciflow/inductor-periodic/160538 -> ciflow/inductor-periodic/160538 2025-08-14T21:24:12.9258357Z * [new tag] ciflow/inductor-rocm/151845 -> ciflow/inductor-rocm/151845 2025-08-14T21:24:12.9258648Z * [new tag] ciflow/inductor-rocm/159158 -> ciflow/inductor-rocm/159158 2025-08-14T21:24:12.9259256Z * [new tag] ciflow/inductor-rocm/160073 -> ciflow/inductor-rocm/160073 2025-08-14T21:24:12.9259527Z * [new tag] ciflow/inductor-rocm/160538 -> ciflow/inductor-rocm/160538 2025-08-14T21:24:12.9262688Z * [new tag] ciflow/inductor/134881 -> ciflow/inductor/134881 2025-08-14T21:24:12.9262948Z * [new tag] ciflow/inductor/137400 -> ciflow/inductor/137400 2025-08-14T21:24:12.9263174Z * [new tag] ciflow/inductor/144516 -> ciflow/inductor/144516 2025-08-14T21:24:12.9263322Z * [new tag] ciflow/inductor/146506 -> ciflow/inductor/146506 2025-08-14T21:24:12.9263472Z * [new tag] ciflow/inductor/147360 -> ciflow/inductor/147360 2025-08-14T21:24:12.9263883Z * [new tag] ciflow/inductor/147990 -> ciflow/inductor/147990 2025-08-14T21:24:12.9264014Z * [new tag] ciflow/inductor/148180 -> ciflow/inductor/148180 2025-08-14T21:24:12.9264567Z * [new tag] ciflow/inductor/148328 -> ciflow/inductor/148328 2025-08-14T21:24:12.9264725Z * [new tag] ciflow/inductor/148484 -> ciflow/inductor/148484 2025-08-14T21:24:12.9264840Z * [new tag] ciflow/inductor/148492 -> ciflow/inductor/148492 2025-08-14T21:24:12.9264960Z * [new tag] ciflow/inductor/150302 -> ciflow/inductor/150302 2025-08-14T21:24:12.9265076Z * [new tag] ciflow/inductor/151845 -> ciflow/inductor/151845 2025-08-14T21:24:12.9265503Z * [new tag] ciflow/inductor/152198 -> ciflow/inductor/152198 2025-08-14T21:24:12.9265977Z * [new tag] ciflow/inductor/152624 -> ciflow/inductor/152624 2025-08-14T21:24:12.9266385Z * [new tag] ciflow/inductor/153966 -> ciflow/inductor/153966 2025-08-14T21:24:12.9266827Z * [new tag] ciflow/inductor/154193 -> ciflow/inductor/154193 2025-08-14T21:24:12.9269301Z * [new tag] ciflow/inductor/154650 -> ciflow/inductor/154650 2025-08-14T21:24:12.9269586Z * [new tag] ciflow/inductor/154694 -> ciflow/inductor/154694 2025-08-14T21:24:12.9269726Z * [new tag] ciflow/inductor/155072 -> ciflow/inductor/155072 2025-08-14T21:24:12.9269917Z * [new tag] ciflow/inductor/155152 -> ciflow/inductor/155152 2025-08-14T21:24:12.9270045Z * [new tag] ciflow/inductor/155153 -> ciflow/inductor/155153 2025-08-14T21:24:12.9270174Z * [new tag] ciflow/inductor/155154 -> ciflow/inductor/155154 2025-08-14T21:24:12.9270295Z * [new tag] ciflow/inductor/155501 -> ciflow/inductor/155501 2025-08-14T21:24:12.9270714Z * [new tag] ciflow/inductor/155502 -> ciflow/inductor/155502 2025-08-14T21:24:12.9270928Z * [new tag] ciflow/inductor/155503 -> ciflow/inductor/155503 2025-08-14T21:24:12.9271807Z * [new tag] ciflow/inductor/155504 -> ciflow/inductor/155504 2025-08-14T21:24:12.9271951Z * [new tag] ciflow/inductor/155557 -> ciflow/inductor/155557 2025-08-14T21:24:12.9275940Z * [new tag] ciflow/inductor/155608 -> ciflow/inductor/155608 2025-08-14T21:24:12.9280971Z * [new tag] ciflow/inductor/155923 -> ciflow/inductor/155923 2025-08-14T21:24:12.9281242Z * [new tag] ciflow/inductor/155928 -> ciflow/inductor/155928 2025-08-14T21:24:12.9281488Z * [new tag] ciflow/inductor/155958 -> ciflow/inductor/155958 2025-08-14T21:24:12.9281625Z * [new tag] ciflow/inductor/156049 -> ciflow/inductor/156049 2025-08-14T21:24:12.9281786Z * [new tag] ciflow/inductor/156851 -> ciflow/inductor/156851 2025-08-14T21:24:12.9281908Z * [new tag] ciflow/inductor/156967 -> ciflow/inductor/156967 2025-08-14T21:24:12.9282015Z * [new tag] ciflow/inductor/157148 -> ciflow/inductor/157148 2025-08-14T21:24:12.9282121Z * [new tag] ciflow/inductor/157149 -> ciflow/inductor/157149 2025-08-14T21:24:12.9282234Z * [new tag] ciflow/inductor/157152 -> ciflow/inductor/157152 2025-08-14T21:24:12.9282344Z * [new tag] ciflow/inductor/157542 -> ciflow/inductor/157542 2025-08-14T21:24:12.9282452Z * [new tag] ciflow/inductor/157572 -> ciflow/inductor/157572 2025-08-14T21:24:12.9282565Z * [new tag] ciflow/inductor/157635 -> ciflow/inductor/157635 2025-08-14T21:24:12.9282671Z * [new tag] ciflow/inductor/157685 -> ciflow/inductor/157685 2025-08-14T21:24:12.9282786Z * [new tag] ciflow/inductor/157686 -> ciflow/inductor/157686 2025-08-14T21:24:12.9283040Z * [new tag] ciflow/inductor/157689 -> ciflow/inductor/157689 2025-08-14T21:24:12.9283161Z * [new tag] ciflow/inductor/157699 -> ciflow/inductor/157699 2025-08-14T21:24:12.9283279Z * [new tag] ciflow/inductor/157743 -> ciflow/inductor/157743 2025-08-14T21:24:12.9283386Z * [new tag] ciflow/inductor/157944 -> ciflow/inductor/157944 2025-08-14T21:24:12.9283501Z * [new tag] ciflow/inductor/157971 -> ciflow/inductor/157971 2025-08-14T21:24:12.9283607Z * [new tag] ciflow/inductor/157994 -> ciflow/inductor/157994 2025-08-14T21:24:12.9283713Z * [new tag] ciflow/inductor/158061 -> ciflow/inductor/158061 2025-08-14T21:24:12.9283829Z * [new tag] ciflow/inductor/158091 -> ciflow/inductor/158091 2025-08-14T21:24:12.9283941Z * [new tag] ciflow/inductor/158097 -> ciflow/inductor/158097 2025-08-14T21:24:12.9284202Z * [new tag] ciflow/inductor/158098 -> ciflow/inductor/158098 2025-08-14T21:24:12.9284383Z * [new tag] ciflow/inductor/158104 -> ciflow/inductor/158104 2025-08-14T21:24:12.9284507Z * [new tag] ciflow/inductor/158168 -> ciflow/inductor/158168 2025-08-14T21:24:12.9284777Z * [new tag] ciflow/inductor/158250 -> ciflow/inductor/158250 2025-08-14T21:24:12.9285952Z * [new tag] ciflow/inductor/158321 -> ciflow/inductor/158321 2025-08-14T21:24:12.9286107Z * [new tag] ciflow/inductor/158609 -> ciflow/inductor/158609 2025-08-14T21:24:12.9286226Z * [new tag] ciflow/inductor/158647 -> ciflow/inductor/158647 2025-08-14T21:24:12.9286639Z * [new tag] ciflow/inductor/158914 -> ciflow/inductor/158914 2025-08-14T21:24:12.9290014Z * [new tag] ciflow/inductor/158932 -> ciflow/inductor/158932 2025-08-14T21:24:12.9290378Z * [new tag] ciflow/inductor/158987 -> ciflow/inductor/158987 2025-08-14T21:24:12.9290601Z * [new tag] ciflow/inductor/159009 -> ciflow/inductor/159009 2025-08-14T21:24:12.9290783Z * [new tag] ciflow/inductor/159010 -> ciflow/inductor/159010 2025-08-14T21:24:12.9290960Z * [new tag] ciflow/inductor/159093 -> ciflow/inductor/159093 2025-08-14T21:24:12.9291487Z * [new tag] ciflow/inductor/159158 -> ciflow/inductor/159158 2025-08-14T21:24:12.9291641Z * [new tag] ciflow/inductor/159197 -> ciflow/inductor/159197 2025-08-14T21:24:12.9291771Z * [new tag] ciflow/inductor/159274 -> ciflow/inductor/159274 2025-08-14T21:24:12.9291884Z * [new tag] ciflow/inductor/159281 -> ciflow/inductor/159281 2025-08-14T21:24:12.9292015Z * [new tag] ciflow/inductor/159329 -> ciflow/inductor/159329 2025-08-14T21:24:12.9292443Z * [new tag] ciflow/inductor/159361 -> ciflow/inductor/159361 2025-08-14T21:24:12.9292580Z * [new tag] ciflow/inductor/159365 -> ciflow/inductor/159365 2025-08-14T21:24:12.9292703Z * [new tag] ciflow/inductor/159366 -> ciflow/inductor/159366 2025-08-14T21:24:12.9293323Z * [new tag] ciflow/inductor/159367 -> ciflow/inductor/159367 2025-08-14T21:24:12.9293569Z * [new tag] ciflow/inductor/159368 -> ciflow/inductor/159368 2025-08-14T21:24:12.9293984Z * [new tag] ciflow/inductor/159473 -> ciflow/inductor/159473 2025-08-14T21:24:12.9294438Z * [new tag] ciflow/inductor/159483 -> ciflow/inductor/159483 2025-08-14T21:24:12.9294874Z * [new tag] ciflow/inductor/159508 -> ciflow/inductor/159508 2025-08-14T21:24:12.9295694Z * [new tag] ciflow/inductor/159523 -> ciflow/inductor/159523 2025-08-14T21:24:12.9295932Z * [new tag] ciflow/inductor/159678 -> ciflow/inductor/159678 2025-08-14T21:24:12.9296228Z * [new tag] ciflow/inductor/159691 -> ciflow/inductor/159691 2025-08-14T21:24:12.9297978Z * [new tag] ciflow/inductor/159778 -> ciflow/inductor/159778 2025-08-14T21:24:12.9298302Z * [new tag] ciflow/inductor/159786 -> ciflow/inductor/159786 2025-08-14T21:24:12.9298458Z * [new tag] ciflow/inductor/159817 -> ciflow/inductor/159817 2025-08-14T21:24:12.9298597Z * [new tag] ciflow/inductor/159842 -> ciflow/inductor/159842 2025-08-14T21:24:12.9298711Z * [new tag] ciflow/inductor/159864 -> ciflow/inductor/159864 2025-08-14T21:24:12.9299026Z * [new tag] ciflow/inductor/159865 -> ciflow/inductor/159865 2025-08-14T21:24:12.9300866Z * [new tag] ciflow/inductor/159869 -> ciflow/inductor/159869 2025-08-14T21:24:12.9301192Z * [new tag] ciflow/inductor/159875 -> ciflow/inductor/159875 2025-08-14T21:24:12.9301408Z * [new tag] ciflow/inductor/159889 -> ciflow/inductor/159889 2025-08-14T21:24:12.9301546Z * [new tag] ciflow/inductor/159902 -> ciflow/inductor/159902 2025-08-14T21:24:12.9301748Z * [new tag] ciflow/inductor/159923 -> ciflow/inductor/159923 2025-08-14T21:24:12.9301879Z * [new tag] ciflow/inductor/159944 -> ciflow/inductor/159944 2025-08-14T21:24:12.9302205Z * [new tag] ciflow/inductor/160004 -> ciflow/inductor/160004 2025-08-14T21:24:12.9304436Z * [new tag] ciflow/inductor/160080 -> ciflow/inductor/160080 2025-08-14T21:24:12.9304737Z * [new tag] ciflow/inductor/160108 -> ciflow/inductor/160108 2025-08-14T21:24:12.9304924Z * [new tag] ciflow/inductor/160109 -> ciflow/inductor/160109 2025-08-14T21:24:12.9305221Z * [new tag] ciflow/inductor/160111 -> ciflow/inductor/160111 2025-08-14T21:24:12.9305464Z * [new tag] ciflow/inductor/160113 -> ciflow/inductor/160113 2025-08-14T21:24:12.9305605Z * [new tag] ciflow/inductor/160127 -> ciflow/inductor/160127 2025-08-14T21:24:12.9305802Z * [new tag] ciflow/inductor/160131 -> ciflow/inductor/160131 2025-08-14T21:24:12.9306004Z * [new tag] ciflow/inductor/160132 -> ciflow/inductor/160132 2025-08-14T21:24:12.9306890Z * [new tag] ciflow/inductor/160136 -> ciflow/inductor/160136 2025-08-14T21:24:12.9307023Z * [new tag] ciflow/inductor/160138 -> ciflow/inductor/160138 2025-08-14T21:24:12.9310484Z * [new tag] ciflow/inductor/160151 -> ciflow/inductor/160151 2025-08-14T21:24:12.9310780Z * [new tag] ciflow/inductor/160152 -> ciflow/inductor/160152 2025-08-14T21:24:12.9310925Z * [new tag] ciflow/inductor/160154 -> ciflow/inductor/160154 2025-08-14T21:24:12.9311054Z * [new tag] ciflow/inductor/160156 -> ciflow/inductor/160156 2025-08-14T21:24:12.9311302Z * [new tag] ciflow/inductor/160161 -> ciflow/inductor/160161 2025-08-14T21:24:12.9311415Z * [new tag] ciflow/inductor/160166 -> ciflow/inductor/160166 2025-08-14T21:24:12.9311645Z * [new tag] ciflow/inductor/160168 -> ciflow/inductor/160168 2025-08-14T21:24:12.9312173Z * [new tag] ciflow/inductor/160174 -> ciflow/inductor/160174 2025-08-14T21:24:12.9312315Z * [new tag] ciflow/inductor/160181 -> ciflow/inductor/160181 2025-08-14T21:24:12.9312440Z * [new tag] ciflow/inductor/160183 -> ciflow/inductor/160183 2025-08-14T21:24:12.9312562Z * [new tag] ciflow/inductor/160190 -> ciflow/inductor/160190 2025-08-14T21:24:12.9312805Z * [new tag] ciflow/inductor/160198 -> ciflow/inductor/160198 2025-08-14T21:24:12.9313270Z * [new tag] ciflow/inductor/160201 -> ciflow/inductor/160201 2025-08-14T21:24:12.9313948Z * [new tag] ciflow/inductor/160209 -> ciflow/inductor/160209 2025-08-14T21:24:12.9314416Z * [new tag] ciflow/inductor/160218 -> ciflow/inductor/160218 2025-08-14T21:24:12.9320718Z * [new tag] ciflow/inductor/160239 -> ciflow/inductor/160239 2025-08-14T21:24:12.9320909Z * [new tag] ciflow/inductor/160250 -> ciflow/inductor/160250 2025-08-14T21:24:12.9321063Z * [new tag] ciflow/inductor/160253 -> ciflow/inductor/160253 2025-08-14T21:24:12.9321202Z * [new tag] ciflow/inductor/160266 -> ciflow/inductor/160266 2025-08-14T21:24:12.9321354Z * [new tag] ciflow/inductor/160282 -> ciflow/inductor/160282 2025-08-14T21:24:12.9321578Z * [new tag] ciflow/inductor/160298 -> ciflow/inductor/160298 2025-08-14T21:24:12.9321804Z * [new tag] ciflow/inductor/160301 -> ciflow/inductor/160301 2025-08-14T21:24:12.9322000Z * [new tag] ciflow/inductor/160310 -> ciflow/inductor/160310 2025-08-14T21:24:12.9322138Z * [new tag] ciflow/inductor/160323 -> ciflow/inductor/160323 2025-08-14T21:24:12.9322697Z * [new tag] ciflow/inductor/160324 -> ciflow/inductor/160324 2025-08-14T21:24:12.9322858Z * [new tag] ciflow/inductor/160325 -> ciflow/inductor/160325 2025-08-14T21:24:12.9322977Z * [new tag] ciflow/inductor/160326 -> ciflow/inductor/160326 2025-08-14T21:24:12.9323098Z * [new tag] ciflow/inductor/160327 -> ciflow/inductor/160327 2025-08-14T21:24:12.9323212Z * [new tag] ciflow/inductor/160328 -> ciflow/inductor/160328 2025-08-14T21:24:12.9323479Z * [new tag] ciflow/inductor/160329 -> ciflow/inductor/160329 2025-08-14T21:24:12.9323622Z * [new tag] ciflow/inductor/160351 -> ciflow/inductor/160351 2025-08-14T21:24:12.9323905Z * [new tag] ciflow/inductor/160353 -> ciflow/inductor/160353 2025-08-14T21:24:12.9324104Z * [new tag] ciflow/inductor/160362 -> ciflow/inductor/160362 2025-08-14T21:24:12.9324345Z * [new tag] ciflow/inductor/160363 -> ciflow/inductor/160363 2025-08-14T21:24:12.9324795Z * [new tag] ciflow/inductor/160364 -> ciflow/inductor/160364 2025-08-14T21:24:12.9325247Z * [new tag] ciflow/inductor/160365 -> ciflow/inductor/160365 2025-08-14T21:24:12.9325959Z * [new tag] ciflow/inductor/160366 -> ciflow/inductor/160366 2025-08-14T21:24:12.9326249Z * [new tag] ciflow/inductor/160367 -> ciflow/inductor/160367 2025-08-14T21:24:12.9326956Z * [new tag] ciflow/inductor/160368 -> ciflow/inductor/160368 2025-08-14T21:24:12.9327216Z * [new tag] ciflow/inductor/160369 -> ciflow/inductor/160369 2025-08-14T21:24:12.9331177Z * [new tag] ciflow/inductor/160371 -> ciflow/inductor/160371 2025-08-14T21:24:12.9331355Z * [new tag] ciflow/inductor/160374 -> ciflow/inductor/160374 2025-08-14T21:24:12.9331481Z * [new tag] ciflow/inductor/160375 -> ciflow/inductor/160375 2025-08-14T21:24:12.9331608Z * [new tag] ciflow/inductor/160377 -> ciflow/inductor/160377 2025-08-14T21:24:12.9331734Z * [new tag] ciflow/inductor/160380 -> ciflow/inductor/160380 2025-08-14T21:24:12.9331868Z * [new tag] ciflow/inductor/160381 -> ciflow/inductor/160381 2025-08-14T21:24:12.9331992Z * [new tag] ciflow/inductor/160383 -> ciflow/inductor/160383 2025-08-14T21:24:12.9332113Z * [new tag] ciflow/inductor/160394 -> ciflow/inductor/160394 2025-08-14T21:24:12.9332433Z * [new tag] ciflow/inductor/160401 -> ciflow/inductor/160401 2025-08-14T21:24:12.9332811Z * [new tag] ciflow/inductor/160402 -> ciflow/inductor/160402 2025-08-14T21:24:12.9332967Z * [new tag] ciflow/inductor/160403 -> ciflow/inductor/160403 2025-08-14T21:24:12.9333511Z * [new tag] ciflow/inductor/160424 -> ciflow/inductor/160424 2025-08-14T21:24:12.9333960Z * [new tag] ciflow/inductor/160426 -> ciflow/inductor/160426 2025-08-14T21:24:12.9334461Z * [new tag] ciflow/inductor/160431 -> ciflow/inductor/160431 2025-08-14T21:24:12.9334880Z * [new tag] ciflow/inductor/160448 -> ciflow/inductor/160448 2025-08-14T21:24:12.9338469Z * [new tag] ciflow/inductor/160450 -> ciflow/inductor/160450 2025-08-14T21:24:12.9338752Z * [new tag] ciflow/inductor/160455 -> ciflow/inductor/160455 2025-08-14T21:24:12.9339023Z * [new tag] ciflow/inductor/160456 -> ciflow/inductor/160456 2025-08-14T21:24:12.9339147Z * [new tag] ciflow/inductor/160461 -> ciflow/inductor/160461 2025-08-14T21:24:12.9339265Z * [new tag] ciflow/inductor/160462 -> ciflow/inductor/160462 2025-08-14T21:24:12.9339372Z * [new tag] ciflow/inductor/160467 -> ciflow/inductor/160467 2025-08-14T21:24:12.9347731Z * [new tag] ciflow/inductor/160470 -> ciflow/inductor/160470 2025-08-14T21:24:12.9348043Z * [new tag] ciflow/inductor/160473 -> ciflow/inductor/160473 2025-08-14T21:24:12.9348186Z * [new tag] ciflow/inductor/160476 -> ciflow/inductor/160476 2025-08-14T21:24:12.9348306Z * [new tag] ciflow/inductor/160480 -> ciflow/inductor/160480 2025-08-14T21:24:12.9348416Z * [new tag] ciflow/inductor/160481 -> ciflow/inductor/160481 2025-08-14T21:24:12.9348866Z * [new tag] ciflow/inductor/160482 -> ciflow/inductor/160482 2025-08-14T21:24:12.9349138Z * [new tag] ciflow/inductor/160483 -> ciflow/inductor/160483 2025-08-14T21:24:12.9349745Z * [new tag] ciflow/inductor/160485 -> ciflow/inductor/160485 2025-08-14T21:24:12.9350053Z * [new tag] ciflow/inductor/160486 -> ciflow/inductor/160486 2025-08-14T21:24:12.9350190Z * [new tag] ciflow/inductor/160503 -> ciflow/inductor/160503 2025-08-14T21:24:12.9350395Z * [new tag] ciflow/inductor/160510 -> ciflow/inductor/160510 2025-08-14T21:24:12.9350527Z * [new tag] ciflow/inductor/160527 -> ciflow/inductor/160527 2025-08-14T21:24:12.9351604Z * [new tag] ciflow/inductor/160530 -> ciflow/inductor/160530 2025-08-14T21:24:12.9351763Z * [new tag] ciflow/inductor/160531 -> ciflow/inductor/160531 2025-08-14T21:24:12.9352046Z * [new tag] ciflow/inductor/160538 -> ciflow/inductor/160538 2025-08-14T21:24:12.9354535Z * [new tag] ciflow/inductor/160539 -> ciflow/inductor/160539 2025-08-14T21:24:12.9354816Z * [new tag] ciflow/inductor/160540 -> ciflow/inductor/160540 2025-08-14T21:24:12.9355049Z * [new tag] ciflow/inductor/160548 -> ciflow/inductor/160548 2025-08-14T21:24:12.9355165Z * [new tag] ciflow/inductor/160561 -> ciflow/inductor/160561 2025-08-14T21:24:12.9355412Z * [new tag] ciflow/inductor/160576 -> ciflow/inductor/160576 2025-08-14T21:24:12.9355594Z * [new tag] ciflow/inductor/160578 -> ciflow/inductor/160578 2025-08-14T21:24:12.9357176Z * [new tag] ciflow/inductor/160580 -> ciflow/inductor/160580 2025-08-14T21:24:12.9357453Z * [new tag] ciflow/inductor/160583 -> ciflow/inductor/160583 2025-08-14T21:24:12.9357753Z * [new tag] ciflow/inductor/160589 -> ciflow/inductor/160589 2025-08-14T21:24:12.9358086Z * [new tag] ciflow/inductor/160590 -> ciflow/inductor/160590 2025-08-14T21:24:12.9358298Z * [new tag] ciflow/inductor/160592 -> ciflow/inductor/160592 2025-08-14T21:24:12.9358630Z * [new tag] ciflow/inductor/160596 -> ciflow/inductor/160596 2025-08-14T21:24:12.9360088Z * [new tag] ciflow/inductor/160601 -> ciflow/inductor/160601 2025-08-14T21:24:12.9360434Z * [new tag] ciflow/inductor/160607 -> ciflow/inductor/160607 2025-08-14T21:24:12.9360596Z * [new tag] ciflow/inductor/160608 -> ciflow/inductor/160608 2025-08-14T21:24:12.9360715Z * [new tag] ciflow/inductor/160611 -> ciflow/inductor/160611 2025-08-14T21:24:12.9361009Z * [new tag] ciflow/inductor/160614 -> ciflow/inductor/160614 2025-08-14T21:24:12.9361855Z * [new tag] ciflow/inductor/160616 -> ciflow/inductor/160616 2025-08-14T21:24:12.9362148Z * [new tag] ciflow/inductor/160619 -> ciflow/inductor/160619 2025-08-14T21:24:12.9362734Z * [new tag] ciflow/inductor/160625 -> ciflow/inductor/160625 2025-08-14T21:24:12.9363159Z * [new tag] ciflow/inductor/160635 -> ciflow/inductor/160635 2025-08-14T21:24:12.9363703Z * [new tag] ciflow/inductor/160649 -> ciflow/inductor/160649 2025-08-14T21:24:12.9364126Z * [new tag] ciflow/inductor/160658 -> ciflow/inductor/160658 2025-08-14T21:24:12.9364607Z * [new tag] ciflow/inductor/160662 -> ciflow/inductor/160662 2025-08-14T21:24:12.9364962Z * [new tag] ciflow/inductor/160668 -> ciflow/inductor/160668 2025-08-14T21:24:12.9365666Z * [new tag] ciflow/inductor/160669 -> ciflow/inductor/160669 2025-08-14T21:24:12.9366067Z * [new tag] ciflow/inductor/160670 -> ciflow/inductor/160670 2025-08-14T21:24:12.9371504Z * [new tag] ciflow/inductor/160671 -> ciflow/inductor/160671 2025-08-14T21:24:12.9371830Z * [new tag] ciflow/inductor/160677 -> ciflow/inductor/160677 2025-08-14T21:24:12.9371992Z * [new tag] ciflow/inductor/160679 -> ciflow/inductor/160679 2025-08-14T21:24:12.9372119Z * [new tag] ciflow/inductor/3b9a386 -> ciflow/inductor/3b9a386 2025-08-14T21:24:12.9372242Z * [new tag] ciflow/inductor/3d4b92b -> ciflow/inductor/3d4b92b 2025-08-14T21:24:12.9372370Z * [new tag] ciflow/inductor/d224ac7 -> ciflow/inductor/d224ac7 2025-08-14T21:24:12.9372511Z * [new tag] ciflow/linux-aarch64/147855 -> ciflow/linux-aarch64/147855 2025-08-14T21:24:12.9372650Z * [new tag] ciflow/linux-aarch64/157994 -> ciflow/linux-aarch64/157994 2025-08-14T21:24:12.9372775Z * [new tag] ciflow/linux-aarch64/159737 -> ciflow/linux-aarch64/159737 2025-08-14T21:24:12.9372907Z * [new tag] ciflow/linux-aarch64/160078 -> ciflow/linux-aarch64/160078 2025-08-14T21:24:12.9373049Z * [new tag] ciflow/linux-aarch64/160299 -> ciflow/linux-aarch64/160299 2025-08-14T21:24:12.9373525Z * [new tag] ciflow/linux-aarch64/160301 -> ciflow/linux-aarch64/160301 2025-08-14T21:24:12.9373666Z * [new tag] ciflow/mps/155923 -> ciflow/mps/155923 2025-08-14T21:24:12.9373772Z * [new tag] ciflow/mps/157553 -> ciflow/mps/157553 2025-08-14T21:24:12.9373907Z * [new tag] ciflow/mps/157635 -> ciflow/mps/157635 2025-08-14T21:24:12.9374272Z * [new tag] ciflow/mps/160541 -> ciflow/mps/160541 2025-08-14T21:24:12.9374964Z * [new tag] ciflow/nightly/156049 -> ciflow/nightly/156049 2025-08-14T21:24:12.9375235Z * [new tag] ciflow/nightly/158104 -> ciflow/nightly/158104 2025-08-14T21:24:12.9378056Z * [new tag] ciflow/op-benchmark/157994 -> ciflow/op-benchmark/157994 2025-08-14T21:24:12.9378585Z * [new tag] ciflow/periodic-rocm-mi300/139971 -> ciflow/periodic-rocm-mi300/139971 2025-08-14T21:24:12.9378765Z * [new tag] ciflow/periodic-rocm-mi300/160073 -> ciflow/periodic-rocm-mi300/160073 2025-08-14T21:24:12.9378959Z * [new tag] ciflow/periodic-rocm-mi300/160538 -> ciflow/periodic-rocm-mi300/160538 2025-08-14T21:24:12.9379100Z * [new tag] ciflow/periodic/054a2fd -> ciflow/periodic/054a2fd 2025-08-14T21:24:12.9379706Z * [new tag] ciflow/periodic/131296 -> ciflow/periodic/131296 2025-08-14T21:24:12.9379874Z * [new tag] ciflow/periodic/139971 -> ciflow/periodic/139971 2025-08-14T21:24:12.9380014Z * [new tag] ciflow/periodic/143959 -> ciflow/periodic/143959 2025-08-14T21:24:12.9380150Z * [new tag] ciflow/periodic/154595 -> ciflow/periodic/154595 2025-08-14T21:24:12.9380287Z * [new tag] ciflow/periodic/156703 -> ciflow/periodic/156703 2025-08-14T21:24:12.9380684Z * [new tag] ciflow/periodic/160201 -> ciflow/periodic/160201 2025-08-14T21:24:12.9381185Z * [new tag] ciflow/periodic/160424 -> ciflow/periodic/160424 2025-08-14T21:24:12.9381593Z * [new tag] ciflow/periodic/160538 -> ciflow/periodic/160538 2025-08-14T21:24:12.9387035Z * [new tag] ciflow/periodic/1febab2a89302464f6c7d69cfbef7a24c421ea65 -> ciflow/periodic/1febab2a89302464f6c7d69cfbef7a24c421ea65 2025-08-14T21:24:12.9387330Z * [new tag] ciflow/periodic/2a6d37d -> ciflow/periodic/2a6d37d 2025-08-14T21:24:12.9387606Z * [new tag] ciflow/periodic/2ee22e435131369a7e4f8cc4732579acc29a941b -> ciflow/periodic/2ee22e435131369a7e4f8cc4732579acc29a941b 2025-08-14T21:24:12.9387878Z * [new tag] ciflow/periodic/317eeb8 -> ciflow/periodic/317eeb8 2025-08-14T21:24:12.9388013Z * [new tag] ciflow/periodic/3c32 -> ciflow/periodic/3c32 2025-08-14T21:24:12.9388134Z * [new tag] ciflow/periodic/3e98831 -> ciflow/periodic/3e98831 2025-08-14T21:24:12.9388416Z * [new tag] ciflow/periodic/4a773e1e867f28a8ff0b15203e5cd9548f74fcee -> ciflow/periodic/4a773e1e867f28a8ff0b15203e5cd9548f74fcee 2025-08-14T21:24:12.9388683Z * [new tag] ciflow/periodic/5f5f508aa836a46dfe88857fb223049616b94e93 -> ciflow/periodic/5f5f508aa836a46dfe88857fb223049616b94e93 2025-08-14T21:24:12.9388830Z * [new tag] ciflow/periodic/94512-point -> ciflow/periodic/94512-point 2025-08-14T21:24:12.9394611Z * [new tag] ciflow/periodic/csl/test87519 -> ciflow/periodic/csl/test87519 2025-08-14T21:24:12.9398770Z * [new tag] ciflow/periodic/csltest88275 -> ciflow/periodic/csltest88275 2025-08-14T21:24:12.9399073Z * [new tag] ciflow/periodic/csltest88761 -> ciflow/periodic/csltest88761 2025-08-14T21:24:12.9399383Z * [new tag] ciflow/periodic/d7114f05b10de8e6de81ffc567d63944c3117d51 -> ciflow/periodic/d7114f05b10de8e6de81ffc567d63944c3117d51 2025-08-14T21:24:12.9399645Z * [new tag] ciflow/periodic/release_1.12 -> ciflow/periodic/release_1.12 2025-08-14T21:24:12.9399905Z * [new tag] ciflow/periodic/release_1.12.0 -> ciflow/periodic/release_1.12.0 2025-08-14T21:24:12.9400435Z * [new tag] ciflow/periodic/sha-ec5b83 -> ciflow/periodic/sha-ec5b83 2025-08-14T21:24:12.9400988Z * [new tag] ciflow/rocm-mi300/151360 -> ciflow/rocm-mi300/151360 2025-08-14T21:24:12.9401154Z * [new tag] ciflow/rocm-mi300/159158 -> ciflow/rocm-mi300/159158 2025-08-14T21:24:12.9401275Z * [new tag] ciflow/rocm-mi300/160073 -> ciflow/rocm-mi300/160073 2025-08-14T21:24:12.9401391Z * [new tag] ciflow/rocm-mi300/160468 -> ciflow/rocm-mi300/160468 2025-08-14T21:24:12.9401715Z * [new tag] ciflow/rocm-mi300/160538 -> ciflow/rocm-mi300/160538 2025-08-14T21:24:12.9401834Z * [new tag] ciflow/rocm-mi355/160215 -> ciflow/rocm-mi355/160215 2025-08-14T21:24:12.9401956Z * [new tag] ciflow/rocm/148492 -> ciflow/rocm/148492 2025-08-14T21:24:12.9402071Z * [new tag] ciflow/rocm/151360 -> ciflow/rocm/151360 2025-08-14T21:24:12.9402186Z * [new tag] ciflow/rocm/151845 -> ciflow/rocm/151845 2025-08-14T21:24:12.9402311Z * [new tag] ciflow/rocm/154864 -> ciflow/rocm/154864 2025-08-14T21:24:12.9402420Z * [new tag] ciflow/rocm/156491 -> ciflow/rocm/156491 2025-08-14T21:24:12.9402536Z * [new tag] ciflow/rocm/158219 -> ciflow/rocm/158219 2025-08-14T21:24:12.9402655Z * [new tag] ciflow/rocm/158220 -> ciflow/rocm/158220 2025-08-14T21:24:12.9402770Z * [new tag] ciflow/rocm/158224 -> ciflow/rocm/158224 2025-08-14T21:24:12.9402893Z * [new tag] ciflow/rocm/159158 -> ciflow/rocm/159158 2025-08-14T21:24:12.9403003Z * [new tag] ciflow/rocm/160215 -> ciflow/rocm/160215 2025-08-14T21:24:12.9403120Z * [new tag] ciflow/rocm/160468 -> ciflow/rocm/160468 2025-08-14T21:24:12.9403232Z * [new tag] ciflow/rocm/160538 -> ciflow/rocm/160538 2025-08-14T21:24:12.9403340Z * [new tag] ciflow/s390/143959 -> ciflow/s390/143959 2025-08-14T21:24:12.9403469Z * [new tag] ciflow/slow/01c7106 -> ciflow/slow/01c7106 2025-08-14T21:24:12.9403583Z * [new tag] ciflow/slow/0577043 -> ciflow/slow/0577043 2025-08-14T21:24:12.9403911Z * [new tag] ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym -> ciflow/slow/0d5b74da0cab798fbfdb9caa53fad816999c8386-sdym 2025-08-14T21:24:12.9404088Z * [new tag] ciflow/slow/0e81104 -> ciflow/slow/0e81104 2025-08-14T21:24:12.9404206Z * [new tag] ciflow/slow/154595 -> ciflow/slow/154595 2025-08-14T21:24:12.9404326Z * [new tag] ciflow/slow/1732077 -> ciflow/slow/1732077 2025-08-14T21:24:12.9405441Z * [new tag] ciflow/slow/187eb7c -> ciflow/slow/187eb7c 2025-08-14T21:24:12.9405795Z * [new tag] ciflow/slow/1faef89 -> ciflow/slow/1faef89 2025-08-14T21:24:12.9406653Z * [new tag] ciflow/slow/3920ec1 -> ciflow/slow/3920ec1 2025-08-14T21:24:12.9407145Z * [new tag] ciflow/slow/3b7c6b2 -> ciflow/slow/3b7c6b2 2025-08-14T21:24:12.9409711Z * [new tag] ciflow/slow/59a3759 -> ciflow/slow/59a3759 2025-08-14T21:24:12.9409878Z * [new tag] ciflow/slow/70ef0bb -> ciflow/slow/70ef0bb 2025-08-14T21:24:12.9410020Z * [new tag] ciflow/slow/788ff06 -> ciflow/slow/788ff06 2025-08-14T21:24:12.9410352Z * [new tag] ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym -> ciflow/slow/8751002215790a3a88750faa8f4366933e296693-sdym 2025-08-14T21:24:12.9410640Z * [new tag] ciflow/slow/9d85864 -> ciflow/slow/9d85864 2025-08-14T21:24:12.9410824Z * [new tag] ciflow/slow/9ffad5b -> ciflow/slow/9ffad5b 2025-08-14T21:24:12.9415561Z * [new tag] ciflow/slow/a206e8b -> ciflow/slow/a206e8b 2025-08-14T21:24:12.9417651Z * [new tag] ciflow/slow/a837609 -> ciflow/slow/a837609 2025-08-14T21:24:12.9417957Z * [new tag] ciflow/slow/af841f3 -> ciflow/slow/af841f3 2025-08-14T21:24:12.9418577Z * [new tag] ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym -> ciflow/slow/da3aba1e46157c4df504b067477cdf2b3c96b194-sdym 2025-08-14T21:24:12.9418802Z * [new tag] ciflow/trunk/131296 -> ciflow/trunk/131296 2025-08-14T21:24:12.9423841Z * [new tag] ciflow/trunk/137400 -> ciflow/trunk/137400 2025-08-14T21:24:12.9424093Z * [new tag] ciflow/trunk/138996 -> ciflow/trunk/138996 2025-08-14T21:24:12.9424218Z * [new tag] ciflow/trunk/139971 -> ciflow/trunk/139971 2025-08-14T21:24:12.9424339Z * [new tag] ciflow/trunk/147360 -> ciflow/trunk/147360 2025-08-14T21:24:12.9424468Z * [new tag] ciflow/trunk/147855 -> ciflow/trunk/147855 2025-08-14T21:24:12.9424656Z * [new tag] ciflow/trunk/148180 -> ciflow/trunk/148180 2025-08-14T21:24:12.9424789Z * [new tag] ciflow/trunk/148328 -> ciflow/trunk/148328 2025-08-14T21:24:12.9424907Z * [new tag] ciflow/trunk/148492 -> ciflow/trunk/148492 2025-08-14T21:24:12.9425019Z * [new tag] ciflow/trunk/150282 -> ciflow/trunk/150282 2025-08-14T21:24:12.9425257Z * [new tag] ciflow/trunk/150302 -> ciflow/trunk/150302 2025-08-14T21:24:12.9425589Z * [new tag] ciflow/trunk/151845 -> ciflow/trunk/151845 2025-08-14T21:24:12.9425702Z * [new tag] ciflow/trunk/152624 -> ciflow/trunk/152624 2025-08-14T21:24:12.9425812Z * [new tag] ciflow/trunk/154193 -> ciflow/trunk/154193 2025-08-14T21:24:12.9426104Z * [new tag] ciflow/trunk/154595 -> ciflow/trunk/154595 2025-08-14T21:24:12.9426209Z * [new tag] ciflow/trunk/154650 -> ciflow/trunk/154650 2025-08-14T21:24:12.9426334Z * [new tag] ciflow/trunk/154694 -> ciflow/trunk/154694 2025-08-14T21:24:12.9426446Z * [new tag] ciflow/trunk/155958 -> ciflow/trunk/155958 2025-08-14T21:24:12.9426551Z * [new tag] ciflow/trunk/156049 -> ciflow/trunk/156049 2025-08-14T21:24:12.9426799Z * [new tag] ciflow/trunk/156703 -> ciflow/trunk/156703 2025-08-14T21:24:12.9426912Z * [new tag] ciflow/trunk/156851 -> ciflow/trunk/156851 2025-08-14T21:24:12.9427040Z * [new tag] ciflow/trunk/157148 -> ciflow/trunk/157148 2025-08-14T21:24:12.9427158Z * [new tag] ciflow/trunk/157152 -> ciflow/trunk/157152 2025-08-14T21:24:12.9427261Z * [new tag] ciflow/trunk/157432 -> ciflow/trunk/157432 2025-08-14T21:24:12.9427362Z * [new tag] ciflow/trunk/157685 -> ciflow/trunk/157685 2025-08-14T21:24:12.9427475Z * [new tag] ciflow/trunk/157689 -> ciflow/trunk/157689 2025-08-14T21:24:12.9427577Z * [new tag] ciflow/trunk/157699 -> ciflow/trunk/157699 2025-08-14T21:24:12.9427688Z * [new tag] ciflow/trunk/157813 -> ciflow/trunk/157813 2025-08-14T21:24:12.9427792Z * [new tag] ciflow/trunk/157994 -> ciflow/trunk/157994 2025-08-14T21:24:12.9433163Z * [new tag] ciflow/trunk/158091 -> ciflow/trunk/158091 2025-08-14T21:24:12.9433528Z * [new tag] ciflow/trunk/158104 -> ciflow/trunk/158104 2025-08-14T21:24:12.9433641Z * [new tag] ciflow/trunk/158219 -> ciflow/trunk/158219 2025-08-14T21:24:12.9433755Z * [new tag] ciflow/trunk/158220 -> ciflow/trunk/158220 2025-08-14T21:24:12.9433863Z * [new tag] ciflow/trunk/158224 -> ciflow/trunk/158224 2025-08-14T21:24:12.9433966Z * [new tag] ciflow/trunk/158529 -> ciflow/trunk/158529 2025-08-14T21:24:12.9434078Z * [new tag] ciflow/trunk/158647 -> ciflow/trunk/158647 2025-08-14T21:24:12.9434182Z * [new tag] ciflow/trunk/158810 -> ciflow/trunk/158810 2025-08-14T21:24:12.9434300Z * [new tag] ciflow/trunk/158812 -> ciflow/trunk/158812 2025-08-14T21:24:12.9434422Z * [new tag] ciflow/trunk/158863 -> ciflow/trunk/158863 2025-08-14T21:24:12.9434694Z * [new tag] ciflow/trunk/158864 -> ciflow/trunk/158864 2025-08-14T21:24:12.9435009Z * [new tag] ciflow/trunk/158883 -> ciflow/trunk/158883 2025-08-14T21:24:12.9435477Z * [new tag] ciflow/trunk/158914 -> ciflow/trunk/158914 2025-08-14T21:24:12.9435605Z * [new tag] ciflow/trunk/158965 -> ciflow/trunk/158965 2025-08-14T21:24:12.9435722Z * [new tag] ciflow/trunk/158987 -> ciflow/trunk/158987 2025-08-14T21:24:12.9435825Z * [new tag] ciflow/trunk/159033 -> ciflow/trunk/159033 2025-08-14T21:24:12.9435929Z * [new tag] ciflow/trunk/159140 -> ciflow/trunk/159140 2025-08-14T21:24:12.9436040Z * [new tag] ciflow/trunk/159158 -> ciflow/trunk/159158 2025-08-14T21:24:12.9436160Z * [new tag] ciflow/trunk/159553 -> ciflow/trunk/159553 2025-08-14T21:24:12.9436269Z * [new tag] ciflow/trunk/159562 -> ciflow/trunk/159562 2025-08-14T21:24:12.9436382Z * [new tag] ciflow/trunk/159682 -> ciflow/trunk/159682 2025-08-14T21:24:12.9436483Z * [new tag] ciflow/trunk/159691 -> ciflow/trunk/159691 2025-08-14T21:24:12.9436592Z * [new tag] ciflow/trunk/159842 -> ciflow/trunk/159842 2025-08-14T21:24:12.9436858Z * [new tag] ciflow/trunk/159889 -> ciflow/trunk/159889 2025-08-14T21:24:12.9438837Z * [new tag] ciflow/trunk/159923 -> ciflow/trunk/159923 2025-08-14T21:24:12.9438972Z * [new tag] ciflow/trunk/160004 -> ciflow/trunk/160004 2025-08-14T21:24:12.9439076Z * [new tag] ciflow/trunk/160113 -> ciflow/trunk/160113 2025-08-14T21:24:12.9439325Z * [new tag] ciflow/trunk/160161 -> ciflow/trunk/160161 2025-08-14T21:24:12.9439629Z * [new tag] ciflow/trunk/160168 -> ciflow/trunk/160168 2025-08-14T21:24:12.9439890Z * [new tag] ciflow/trunk/160181 -> ciflow/trunk/160181 2025-08-14T21:24:12.9440010Z * [new tag] ciflow/trunk/160183 -> ciflow/trunk/160183 2025-08-14T21:24:12.9440190Z * [new tag] ciflow/trunk/160190 -> ciflow/trunk/160190 2025-08-14T21:24:12.9440318Z * [new tag] ciflow/trunk/160198 -> ciflow/trunk/160198 2025-08-14T21:24:12.9440421Z * [new tag] ciflow/trunk/160205 -> ciflow/trunk/160205 2025-08-14T21:24:12.9440651Z * [new tag] ciflow/trunk/160219 -> ciflow/trunk/160219 2025-08-14T21:24:12.9441076Z * [new tag] ciflow/trunk/160224 -> ciflow/trunk/160224 2025-08-14T21:24:12.9441229Z * [new tag] ciflow/trunk/160250 -> ciflow/trunk/160250 2025-08-14T21:24:12.9441579Z * [new tag] ciflow/trunk/160253 -> ciflow/trunk/160253 2025-08-14T21:24:12.9442080Z * [new tag] ciflow/trunk/160335 -> ciflow/trunk/160335 2025-08-14T21:24:12.9443292Z * [new tag] ciflow/trunk/160338 -> ciflow/trunk/160338 2025-08-14T21:24:12.9443444Z * [new tag] ciflow/trunk/160383 -> ciflow/trunk/160383 2025-08-14T21:24:12.9443695Z * [new tag] ciflow/trunk/160401 -> ciflow/trunk/160401 2025-08-14T21:24:12.9444145Z * [new tag] ciflow/trunk/160403 -> ciflow/trunk/160403 2025-08-14T21:24:12.9444515Z * [new tag] ciflow/trunk/160430 -> ciflow/trunk/160430 2025-08-14T21:24:12.9444946Z * [new tag] ciflow/trunk/160431 -> ciflow/trunk/160431 2025-08-14T21:24:12.9445729Z * [new tag] ciflow/trunk/160439 -> ciflow/trunk/160439 2025-08-14T21:24:12.9446171Z * [new tag] ciflow/trunk/160449 -> ciflow/trunk/160449 2025-08-14T21:24:12.9446695Z * [new tag] ciflow/trunk/160454 -> ciflow/trunk/160454 2025-08-14T21:24:12.9447127Z * [new tag] ciflow/trunk/160468 -> ciflow/trunk/160468 2025-08-14T21:24:12.9447554Z * [new tag] ciflow/trunk/160481 -> ciflow/trunk/160481 2025-08-14T21:24:12.9450820Z * [new tag] ciflow/trunk/160485 -> ciflow/trunk/160485 2025-08-14T21:24:12.9450998Z * [new tag] ciflow/trunk/160519 -> ciflow/trunk/160519 2025-08-14T21:24:12.9451217Z * [new tag] ciflow/trunk/160527 -> ciflow/trunk/160527 2025-08-14T21:24:12.9451350Z * [new tag] ciflow/trunk/160560 -> ciflow/trunk/160560 2025-08-14T21:24:12.9451537Z * [new tag] ciflow/trunk/160578 -> ciflow/trunk/160578 2025-08-14T21:24:12.9451830Z * [new tag] ciflow/trunk/160589 -> ciflow/trunk/160589 2025-08-14T21:24:12.9452141Z * [new tag] ciflow/trunk/160592 -> ciflow/trunk/160592 2025-08-14T21:24:12.9452284Z * [new tag] ciflow/trunk/160649 -> ciflow/trunk/160649 2025-08-14T21:24:12.9452401Z * [new tag] ciflow/trunk/160656 -> ciflow/trunk/160656 2025-08-14T21:24:12.9452556Z * [new tag] ciflow/unstable/123 -> ciflow/unstable/123 2025-08-14T21:24:12.9455537Z * [new tag] ciflow/vllm/160116 -> ciflow/vllm/160116 2025-08-14T21:24:12.9455690Z * [new tag] ciflow/vllm/160583 -> ciflow/vllm/160583 2025-08-14T21:24:12.9455810Z * [new tag] ciflow/vllm/160619 -> ciflow/vllm/160619 2025-08-14T21:24:12.9455920Z * [new tag] ciflow/vllm/160625 -> ciflow/vllm/160625 2025-08-14T21:24:12.9456034Z * [new tag] ciflow/vllm/160627 -> ciflow/vllm/160627 2025-08-14T21:24:12.9456338Z * [new tag] ciflow/win-arm64/156049 -> ciflow/win-arm64/156049 2025-08-14T21:24:12.9456491Z * [new tag] ciflow/win-arm64/158104 -> ciflow/win-arm64/158104 2025-08-14T21:24:12.9456784Z * [new tag] ciflow/win-arm64/159553 -> ciflow/win-arm64/159553 2025-08-14T21:24:12.9456923Z * [new tag] ciflow/win-arm64/159562 -> ciflow/win-arm64/159562 2025-08-14T21:24:12.9457077Z * [new tag] ciflow/win-arm64/159777 -> ciflow/win-arm64/159777 2025-08-14T21:24:12.9457582Z * [new tag] ciflow/win-arm64/159780 -> ciflow/win-arm64/159780 2025-08-14T21:24:12.9457971Z * [new tag] ciflow/win-arm64/159842 -> ciflow/win-arm64/159842 2025-08-14T21:24:12.9458519Z * [new tag] ciflow/win-arm64/160250 -> ciflow/win-arm64/160250 2025-08-14T21:24:12.9459242Z * [new tag] ciflow/win-arm64/160253 -> ciflow/win-arm64/160253 2025-08-14T21:24:12.9459530Z * [new tag] ciflow/win-arm64/160454 -> ciflow/win-arm64/160454 2025-08-14T21:24:12.9459857Z * [new tag] ciflow/win-arm64/160560 -> ciflow/win-arm64/160560 2025-08-14T21:24:12.9460187Z * [new tag] ciflow/xpu/138996 -> ciflow/xpu/138996 2025-08-14T21:24:12.9460560Z * [new tag] ciflow/xpu/139971 -> ciflow/xpu/139971 2025-08-14T21:24:12.9461428Z * [new tag] ciflow/xpu/140972 -> ciflow/xpu/140972 2025-08-14T21:24:12.9461548Z * [new tag] ciflow/xpu/143553 -> ciflow/xpu/143553 2025-08-14T21:24:12.9464972Z * [new tag] ciflow/xpu/156272 -> ciflow/xpu/156272 2025-08-14T21:24:12.9465118Z * [new tag] ciflow/xpu/156812 -> ciflow/xpu/156812 2025-08-14T21:24:12.9465231Z * [new tag] ciflow/xpu/157699 -> ciflow/xpu/157699 2025-08-14T21:24:12.9465331Z * [new tag] ciflow/xpu/157994 -> ciflow/xpu/157994 2025-08-14T21:24:12.9465445Z * [new tag] ciflow/xpu/158336 -> ciflow/xpu/158336 2025-08-14T21:24:12.9465704Z * [new tag] ciflow/xpu/158733 -> ciflow/xpu/158733 2025-08-14T21:24:12.9465805Z * [new tag] ciflow/xpu/159033 -> ciflow/xpu/159033 2025-08-14T21:24:12.9465918Z * [new tag] ciflow/xpu/159118 -> ciflow/xpu/159118 2025-08-14T21:24:12.9466019Z * [new tag] ciflow/xpu/159140 -> ciflow/xpu/159140 2025-08-14T21:24:12.9466344Z * [new tag] ciflow/xpu/159241 -> ciflow/xpu/159241 2025-08-14T21:24:12.9469162Z * [new tag] ciflow/xpu/159473 -> ciflow/xpu/159473 2025-08-14T21:24:12.9469456Z * [new tag] ciflow/xpu/159474 -> ciflow/xpu/159474 2025-08-14T21:24:12.9469590Z * [new tag] ciflow/xpu/159553 -> ciflow/xpu/159553 2025-08-14T21:24:12.9469714Z * [new tag] ciflow/xpu/159944 -> ciflow/xpu/159944 2025-08-14T21:24:12.9471001Z * [new tag] ciflow/xpu/160062 -> ciflow/xpu/160062 2025-08-14T21:24:12.9471140Z * [new tag] ciflow/xpu/160067 -> ciflow/xpu/160067 2025-08-14T21:24:12.9471250Z * [new tag] ciflow/xpu/160158 -> ciflow/xpu/160158 2025-08-14T21:24:12.9471350Z * [new tag] ciflow/xpu/160173 -> ciflow/xpu/160173 2025-08-14T21:24:12.9471452Z * [new tag] ciflow/xpu/160183 -> ciflow/xpu/160183 2025-08-14T21:24:12.9471562Z * [new tag] ciflow/xpu/160301 -> ciflow/xpu/160301 2025-08-14T21:24:12.9471675Z * [new tag] ciflow/xpu/160403 -> ciflow/xpu/160403 2025-08-14T21:24:12.9471899Z * [new tag] ciflow/xpu/160606 -> ciflow/xpu/160606 2025-08-14T21:24:12.9472198Z * [new tag] cslpull75 -> cslpull75 2025-08-14T21:24:12.9472968Z * [new tag] cslpull76 -> cslpull76 2025-08-14T21:24:12.9473222Z * [new tag] cslpull77 -> cslpull77 2025-08-14T21:24:12.9474081Z * [new tag] cslpull78 -> cslpull78 2025-08-14T21:24:12.9474416Z * [new tag] cslpull79 -> cslpull79 2025-08-14T21:24:12.9477271Z * [new tag] cslpull80 -> cslpull80 2025-08-14T21:24:12.9477421Z * [new tag] cslpull81 -> cslpull81 2025-08-14T21:24:12.9477525Z * [new tag] cslpull82 -> cslpull82 2025-08-14T21:24:12.9477633Z * [new tag] cslpull83 -> cslpull83 2025-08-14T21:24:12.9477734Z * [new tag] cslpull84 -> cslpull84 2025-08-14T21:24:12.9477838Z * [new tag] cslpull85 -> cslpull85 2025-08-14T21:24:12.9478587Z * [new tag] cslpull86 -> cslpull86 2025-08-14T21:24:12.9478892Z * [new tag] cslpull87 -> cslpull87 2025-08-14T21:24:12.9479820Z * [new tag] cslpull88 -> cslpull88 2025-08-14T21:24:12.9480818Z * [new tag] cslpull89 -> cslpull89 2025-08-14T21:24:12.9481047Z * [new tag] cslpull90 -> cslpull90 2025-08-14T21:24:12.9482227Z * [new tag] cslpull91 -> cslpull91 2025-08-14T21:24:12.9482580Z * [new tag] cslpull92 -> cslpull92 2025-08-14T21:24:12.9483446Z * [new tag] flight_5 -> flight_5 2025-08-14T21:24:12.9483827Z * [new tag] flight_5.1 -> flight_5.1 2025-08-14T21:24:12.9484685Z * [new tag] flight_5.2 -> flight_5.2 2025-08-14T21:24:12.9484901Z * [new tag] flight_5.3 -> flight_5.3 2025-08-14T21:24:12.9485723Z * [new tag] forpull1 -> forpull1 2025-08-14T21:24:12.9486947Z * [new tag] malfet/tag-2ef5611 -> malfet/tag-2ef5611 2025-08-14T21:24:12.9487311Z * [new tag] malfet/tag-317b1a0 -> malfet/tag-317b1a0 2025-08-14T21:24:12.9488246Z * [new tag] malfet/tag-ec6f767 -> malfet/tag-ec6f767 2025-08-14T21:24:12.9488503Z * [new tag] nightly-binary -> nightly-binary 2025-08-14T21:24:12.9489044Z * [new tag] sqzhang_flight4_plus -> sqzhang_flight4_plus 2025-08-14T21:24:12.9489623Z * [new tag] sqzhang_flight_3 -> sqzhang_flight_3 2025-08-14T21:24:12.9490695Z * [new tag] trunk/01584d2a7d029c9749eb73678cf1dc313cc35df6 -> trunk/01584d2a7d029c9749eb73678cf1dc313cc35df6 2025-08-14T21:24:12.9490936Z * [new tag] trunk/017259f9c65b6fad55fb9597d7077e2543eaae46 -> trunk/017259f9c65b6fad55fb9597d7077e2543eaae46 2025-08-14T21:24:12.9493704Z * [new tag] trunk/01bcf9a40dea937637d2cdd530bed2652510943d -> trunk/01bcf9a40dea937637d2cdd530bed2652510943d 2025-08-14T21:24:12.9498227Z * [new tag] trunk/01f66d08d93365015f4af005a252f439c4d4013a -> trunk/01f66d08d93365015f4af005a252f439c4d4013a 2025-08-14T21:24:12.9502234Z * [new tag] trunk/03b254e49f2d4c092e6ca712e5702cf2895aa47e -> trunk/03b254e49f2d4c092e6ca712e5702cf2895aa47e 2025-08-14T21:24:12.9507035Z * [new tag] trunk/05029ad1c30865d3f7e7fd13384db9d826e563eb -> trunk/05029ad1c30865d3f7e7fd13384db9d826e563eb 2025-08-14T21:24:12.9509229Z * [new tag] trunk/05c19d1acecc01b0d2512364183058a6885b9869 -> trunk/05c19d1acecc01b0d2512364183058a6885b9869 2025-08-14T21:24:12.9509662Z * [new tag] trunk/05c417715f791875fbf28cfc3fc86142de1a3206 -> trunk/05c417715f791875fbf28cfc3fc86142de1a3206 2025-08-14T21:24:12.9515465Z * [new tag] trunk/06824f3c7268bb807a422b663047cd0900ddd126 -> trunk/06824f3c7268bb807a422b663047cd0900ddd126 2025-08-14T21:24:12.9520054Z * [new tag] trunk/077cb389746a7d61cfc018aad2ba29a8aa195610 -> trunk/077cb389746a7d61cfc018aad2ba29a8aa195610 2025-08-14T21:24:12.9520546Z * [new tag] trunk/089c4a1ba007ed4abb3e5e0eafd97b7584566057 -> trunk/089c4a1ba007ed4abb3e5e0eafd97b7584566057 2025-08-14T21:24:12.9520813Z * [new tag] trunk/09381f5dacda7bbbfa361f5df76bde5cd309adc1 -> trunk/09381f5dacda7bbbfa361f5df76bde5cd309adc1 2025-08-14T21:24:12.9521050Z * [new tag] trunk/0bd3af4fb87445f4de3a1f9b823e399c8b3cefde -> trunk/0bd3af4fb87445f4de3a1f9b823e399c8b3cefde 2025-08-14T21:24:12.9521291Z * [new tag] trunk/0d3461bac0fb5177e35152d980b301ea3a0aa2c4 -> trunk/0d3461bac0fb5177e35152d980b301ea3a0aa2c4 2025-08-14T21:24:12.9521533Z * [new tag] trunk/0d40ff3b496e68193bc16d5391fa2e3623709f81 -> trunk/0d40ff3b496e68193bc16d5391fa2e3623709f81 2025-08-14T21:24:12.9521805Z * [new tag] trunk/0d71ca2c46753bb268bfdcf815c14415c122a289 -> trunk/0d71ca2c46753bb268bfdcf815c14415c122a289 2025-08-14T21:24:12.9522050Z * [new tag] trunk/0d88593dd826544c9e7bd4aa615ef86847a78d2b -> trunk/0d88593dd826544c9e7bd4aa615ef86847a78d2b 2025-08-14T21:24:12.9522299Z * [new tag] trunk/0e3e377bd5126cfcc69d70c4d77b352d3404cc11 -> trunk/0e3e377bd5126cfcc69d70c4d77b352d3404cc11 2025-08-14T21:24:12.9522544Z * [new tag] trunk/0f3b10b8eebe68e3c75d473d499b87dfe14a2eca -> trunk/0f3b10b8eebe68e3c75d473d499b87dfe14a2eca 2025-08-14T21:24:12.9522784Z * [new tag] trunk/101276f81b4d2a8c31bfd6796b986d4c1bfdf483 -> trunk/101276f81b4d2a8c31bfd6796b986d4c1bfdf483 2025-08-14T21:24:12.9523017Z * [new tag] trunk/1028c5e2d50e121865bf98307e7c035f549a24b2 -> trunk/1028c5e2d50e121865bf98307e7c035f549a24b2 2025-08-14T21:24:12.9523261Z * [new tag] trunk/10bc36fe840cb3510fab84d2ea22663b76702f1e -> trunk/10bc36fe840cb3510fab84d2ea22663b76702f1e 2025-08-14T21:24:12.9523557Z * [new tag] trunk/10e3514c962b58cbbee994257872a626ff76d51b -> trunk/10e3514c962b58cbbee994257872a626ff76d51b 2025-08-14T21:24:12.9523796Z * [new tag] trunk/1128f4c2a822cbe34a9d966306af15097179ffe1 -> trunk/1128f4c2a822cbe34a9d966306af15097179ffe1 2025-08-14T21:24:12.9524040Z * [new tag] trunk/114a6c40434bfb9cfa5abc30e9e34d81300d743e -> trunk/114a6c40434bfb9cfa5abc30e9e34d81300d743e 2025-08-14T21:24:12.9524279Z * [new tag] trunk/118bc97b14c24ac88a4b0c0750a9e7bf93154c76 -> trunk/118bc97b14c24ac88a4b0c0750a9e7bf93154c76 2025-08-14T21:24:12.9524526Z * [new tag] trunk/1196bb1c2e4d5a7edc09f2260e3034132f0c6c91 -> trunk/1196bb1c2e4d5a7edc09f2260e3034132f0c6c91 2025-08-14T21:24:12.9524764Z * [new tag] trunk/11a3565f1872bbad9c253a127e8d4ce7a1b40ec8 -> trunk/11a3565f1872bbad9c253a127e8d4ce7a1b40ec8 2025-08-14T21:24:12.9525002Z * [new tag] trunk/15e49f61643e4c0eef420f0981609709ef55b848 -> trunk/15e49f61643e4c0eef420f0981609709ef55b848 2025-08-14T21:24:12.9525238Z * [new tag] trunk/16d15445f8bd8740095b23de4af89d757af793ca -> trunk/16d15445f8bd8740095b23de4af89d757af793ca 2025-08-14T21:24:12.9525469Z * [new tag] trunk/178515d0ff6833c8e9221482b2a650ab31e00019 -> trunk/178515d0ff6833c8e9221482b2a650ab31e00019 2025-08-14T21:24:12.9525787Z * [new tag] trunk/182efe31dbe43376e7eef7338356aaf94d5bcabe -> trunk/182efe31dbe43376e7eef7338356aaf94d5bcabe 2025-08-14T21:24:12.9526070Z * [new tag] trunk/194fcfcfbdad0add1a1b695321e31a576058f4cf -> trunk/194fcfcfbdad0add1a1b695321e31a576058f4cf 2025-08-14T21:24:12.9526317Z * [new tag] trunk/195b5c2e27eb8f21cbc8ad1e90f42db5a8cfccca -> trunk/195b5c2e27eb8f21cbc8ad1e90f42db5a8cfccca 2025-08-14T21:24:12.9526564Z * [new tag] trunk/198b5fd2d47fa3d5110ceba6827a3b18e0064014 -> trunk/198b5fd2d47fa3d5110ceba6827a3b18e0064014 2025-08-14T21:24:12.9526901Z * [new tag] trunk/199e9abb6a366bbd27c39d1da7c3123b4eea9b5a -> trunk/199e9abb6a366bbd27c39d1da7c3123b4eea9b5a 2025-08-14T21:24:12.9527143Z * [new tag] trunk/19b4283884b2d9b3a0eb364da10b1540d14ab7a7 -> trunk/19b4283884b2d9b3a0eb364da10b1540d14ab7a7 2025-08-14T21:24:12.9527436Z * [new tag] trunk/1c2587119152cec3905647a47c65d3d26619c5a8 -> trunk/1c2587119152cec3905647a47c65d3d26619c5a8 2025-08-14T21:24:12.9527691Z * [new tag] trunk/1c26c53851c212a7c90a325549a72f0571613a8c -> trunk/1c26c53851c212a7c90a325549a72f0571613a8c 2025-08-14T21:24:12.9527939Z * [new tag] trunk/1c2cba17eab2b09d87142883da2bdbdbcf018613 -> trunk/1c2cba17eab2b09d87142883da2bdbdbcf018613 2025-08-14T21:24:12.9528182Z * [new tag] trunk/1d80d697a269234b47ec7ede192faf3bb9b159e3 -> trunk/1d80d697a269234b47ec7ede192faf3bb9b159e3 2025-08-14T21:24:12.9528438Z * [new tag] trunk/1ea688f9a2602fbcde32c0302b822526ca4219dc -> trunk/1ea688f9a2602fbcde32c0302b822526ca4219dc 2025-08-14T21:24:12.9528677Z * [new tag] trunk/1f4057c11ac941fb324386ca594d0a6882185aad -> trunk/1f4057c11ac941fb324386ca594d0a6882185aad 2025-08-14T21:24:12.9528933Z * [new tag] trunk/1fc683cf17c8c673044538d10266c00f92987be2 -> trunk/1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:24:12.9529208Z * [new tag] trunk/1febab2a89302464f6c7d69cfbef7a24c421ea65 -> trunk/1febab2a89302464f6c7d69cfbef7a24c421ea65 2025-08-14T21:24:12.9529446Z * [new tag] trunk/206c1eef6571f906c2792d899a09136b3fce9673 -> trunk/206c1eef6571f906c2792d899a09136b3fce9673 2025-08-14T21:24:12.9529691Z * [new tag] trunk/20bdabbb3c5d6b118a94b2e045c777662563d5bb -> trunk/20bdabbb3c5d6b118a94b2e045c777662563d5bb 2025-08-14T21:24:12.9529903Z * [new tag] trunk/21392c0e06ac2b2621950455975ca6332f0bf641 -> trunk/21392c0e06ac2b2621950455975ca6332f0bf641 2025-08-14T21:24:12.9530125Z * [new tag] trunk/2247aa6d1d43e256255f5c74a781c3190a4387b6 -> trunk/2247aa6d1d43e256255f5c74a781c3190a4387b6 2025-08-14T21:24:12.9530386Z * [new tag] trunk/2259dbed4e0d3f2a8174b5847fd0741aed42451d -> trunk/2259dbed4e0d3f2a8174b5847fd0741aed42451d 2025-08-14T21:24:12.9530599Z * [new tag] trunk/231c72240d80091f099c95e326d3600cba866eee -> trunk/231c72240d80091f099c95e326d3600cba866eee 2025-08-14T21:24:12.9530823Z * [new tag] trunk/24257f5bfaa37795f74d9f64c1b43584128d4b8c -> trunk/24257f5bfaa37795f74d9f64c1b43584128d4b8c 2025-08-14T21:24:12.9531069Z * [new tag] trunk/24f43d0da7ad9c6e95a09a2fee610387728cc1cd -> trunk/24f43d0da7ad9c6e95a09a2fee610387728cc1cd 2025-08-14T21:24:12.9531332Z * [new tag] trunk/2898d3f965e5cd9d02fc2ecdab7c580fd457fea9 -> trunk/2898d3f965e5cd9d02fc2ecdab7c580fd457fea9 2025-08-14T21:24:12.9531573Z * [new tag] trunk/28ccc9e7247798980fe00a11bcd64a8016b5f227 -> trunk/28ccc9e7247798980fe00a11bcd64a8016b5f227 2025-08-14T21:24:12.9531817Z * [new tag] trunk/29712314dd5cf500a8ea3d1c69483a3cb768ca72 -> trunk/29712314dd5cf500a8ea3d1c69483a3cb768ca72 2025-08-14T21:24:12.9532129Z * [new tag] trunk/29d20d49f0b7f4e362e1cefdcdc4b5659969312c -> trunk/29d20d49f0b7f4e362e1cefdcdc4b5659969312c 2025-08-14T21:24:12.9532352Z * [new tag] trunk/2c5e10a5fceb208b11c3d569ae02e348b5893b31 -> trunk/2c5e10a5fceb208b11c3d569ae02e348b5893b31 2025-08-14T21:24:12.9532597Z * [new tag] trunk/2d0cdee394bccadcd0abe19dd4623ed978a331ad -> trunk/2d0cdee394bccadcd0abe19dd4623ed978a331ad 2025-08-14T21:24:12.9532835Z * [new tag] trunk/2e4e5ab4be9e0aeffd9c49b5b2f9f820bd0895b1 -> trunk/2e4e5ab4be9e0aeffd9c49b5b2f9f820bd0895b1 2025-08-14T21:24:12.9533054Z * [new tag] trunk/2ea40fba841b3af8103f332ba62e54f350ba9a51 -> trunk/2ea40fba841b3af8103f332ba62e54f350ba9a51 2025-08-14T21:24:12.9533301Z * [new tag] trunk/2ee22e435131369a7e4f8cc4732579acc29a941b -> trunk/2ee22e435131369a7e4f8cc4732579acc29a941b 2025-08-14T21:24:12.9533510Z * [new tag] trunk/2f4c2226175512af787725c4d5ad7313c60d4db1 -> trunk/2f4c2226175512af787725c4d5ad7313c60d4db1 2025-08-14T21:24:12.9533728Z * [new tag] trunk/3008d985a8fc155eb89374afff50cb33a6bd10d5 -> trunk/3008d985a8fc155eb89374afff50cb33a6bd10d5 2025-08-14T21:24:12.9533940Z * [new tag] trunk/3028fa6ce9d9c96671722ab8213a1a30670d7cf2 -> trunk/3028fa6ce9d9c96671722ab8213a1a30670d7cf2 2025-08-14T21:24:12.9534160Z * [new tag] trunk/303c614f3df95ae2b659c5f6c1838b14e4776ce6 -> trunk/303c614f3df95ae2b659c5f6c1838b14e4776ce6 2025-08-14T21:24:12.9534371Z * [new tag] trunk/305fa2239365ad17ac9c534a68bba8a149c42d67 -> trunk/305fa2239365ad17ac9c534a68bba8a149c42d67 2025-08-14T21:24:12.9534600Z * [new tag] trunk/31c9ac4319c0cc2ed8c6be701c6ccf73f6cb4706 -> trunk/31c9ac4319c0cc2ed8c6be701c6ccf73f6cb4706 2025-08-14T21:24:12.9534824Z * [new tag] trunk/32099961d588fc19ead8afe805d6b5108de75669 -> trunk/32099961d588fc19ead8afe805d6b5108de75669 2025-08-14T21:24:12.9535036Z * [new tag] trunk/32e5e2f596d55bb9441d5d53f3c58bcb55828047 -> trunk/32e5e2f596d55bb9441d5d53f3c58bcb55828047 2025-08-14T21:24:12.9535247Z * [new tag] trunk/334b38ccc4427b1d14981c48a3a0b92180d58225 -> trunk/334b38ccc4427b1d14981c48a3a0b92180d58225 2025-08-14T21:24:12.9535463Z * [new tag] trunk/334ecbd4ffe11858cae7d23d1190ddb4777c2513 -> trunk/334ecbd4ffe11858cae7d23d1190ddb4777c2513 2025-08-14T21:24:12.9535666Z * [new tag] trunk/33d94018668951611b318b7515ae96f04e48eac0 -> trunk/33d94018668951611b318b7515ae96f04e48eac0 2025-08-14T21:24:12.9535889Z * [new tag] trunk/34358f335d95213d96b6cca6a83e7bf3af6a9fcb -> trunk/34358f335d95213d96b6cca6a83e7bf3af6a9fcb 2025-08-14T21:24:12.9536112Z * [new tag] trunk/34ec5ed275f8aa875c80daa97b3e82af0b06f673 -> trunk/34ec5ed275f8aa875c80daa97b3e82af0b06f673 2025-08-14T21:24:12.9536361Z * [new tag] trunk/355462e1278d818deb9ef4a184073d5b66074816 -> trunk/355462e1278d818deb9ef4a184073d5b66074816 2025-08-14T21:24:12.9536721Z * [new tag] trunk/3626ba711b34397d1fbf0a9b1979f85cbf68b919 -> trunk/3626ba711b34397d1fbf0a9b1979f85cbf68b919 2025-08-14T21:24:12.9536956Z * [new tag] trunk/36f46d082a4954921cb8493223f000f2aab79ed7 -> trunk/36f46d082a4954921cb8493223f000f2aab79ed7 2025-08-14T21:24:12.9537166Z * [new tag] trunk/39aa3d1471549b7829c207d634dfdc1d26e346a2 -> trunk/39aa3d1471549b7829c207d634dfdc1d26e346a2 2025-08-14T21:24:12.9537883Z * [new tag] trunk/3a562374401113187ce2566b87e3f1d87d7c53aa -> trunk/3a562374401113187ce2566b87e3f1d87d7c53aa 2025-08-14T21:24:12.9538539Z * [new tag] trunk/3ac86e728dfaa7383ff7f865e9e7d33486188dae -> trunk/3ac86e728dfaa7383ff7f865e9e7d33486188dae 2025-08-14T21:24:12.9538809Z * [new tag] trunk/3be70dc30e893b552fc0f23ca06cd8f7949b6d08 -> trunk/3be70dc30e893b552fc0f23ca06cd8f7949b6d08 2025-08-14T21:24:12.9539084Z * [new tag] trunk/3cec82a7e9aea040a34dd7a2587ae6d3bd65dba0 -> trunk/3cec82a7e9aea040a34dd7a2587ae6d3bd65dba0 2025-08-14T21:24:12.9539646Z * [new tag] trunk/3cf7b4024ef83e44e9ae223dbff7c7ab68240cb2 -> trunk/3cf7b4024ef83e44e9ae223dbff7c7ab68240cb2 2025-08-14T21:24:12.9540188Z * [new tag] trunk/3ef2e1ef769582a82c6ddf150e9d11bf4bf1c44f -> trunk/3ef2e1ef769582a82c6ddf150e9d11bf4bf1c44f 2025-08-14T21:24:12.9540753Z * [new tag] trunk/3f1636ebef9b45e8a3cb0eb20d327ee6acb74be0 -> trunk/3f1636ebef9b45e8a3cb0eb20d327ee6acb74be0 2025-08-14T21:24:12.9541274Z * [new tag] trunk/3faee0a6318afcbbbb48687009a459214910d820 -> trunk/3faee0a6318afcbbbb48687009a459214910d820 2025-08-14T21:24:12.9541917Z * [new tag] trunk/3fcd79e023da7156ac584992ebab29205d3b7881 -> trunk/3fcd79e023da7156ac584992ebab29205d3b7881 2025-08-14T21:24:12.9542364Z * [new tag] trunk/3fe19a7a0af3f4d692af30476c320be18c7e8ae6 -> trunk/3fe19a7a0af3f4d692af30476c320be18c7e8ae6 2025-08-14T21:24:12.9544421Z * [new tag] trunk/41673110cd7c5960824cc74a6fcaeda1a8bc7a23 -> trunk/41673110cd7c5960824cc74a6fcaeda1a8bc7a23 2025-08-14T21:24:12.9544681Z * [new tag] trunk/4183d4ff3dcc1d87400326a9a7998c3f9e966f60 -> trunk/4183d4ff3dcc1d87400326a9a7998c3f9e966f60 2025-08-14T21:24:12.9544920Z * [new tag] trunk/422bd6808bb98cbbac31d157d9c82ad11ba9732d -> trunk/422bd6808bb98cbbac31d157d9c82ad11ba9732d 2025-08-14T21:24:12.9545138Z * [new tag] trunk/42e51cd4b3973a053fcfa80878a3f346fd158e9f -> trunk/42e51cd4b3973a053fcfa80878a3f346fd158e9f 2025-08-14T21:24:12.9545506Z * [new tag] trunk/4416433c7c625127b7f975c92f8ec98ea4c67fd3 -> trunk/4416433c7c625127b7f975c92f8ec98ea4c67fd3 2025-08-14T21:24:12.9545846Z * [new tag] trunk/45ba7ecda876685b083cbbe932450560c566826b -> trunk/45ba7ecda876685b083cbbe932450560c566826b 2025-08-14T21:24:12.9546579Z * [new tag] trunk/47a1db823dfcdacdb99f317428fc3791a18c5812 -> trunk/47a1db823dfcdacdb99f317428fc3791a18c5812 2025-08-14T21:24:12.9547209Z * [new tag] trunk/4a773e1e867f28a8ff0b15203e5cd9548f74fcee -> trunk/4a773e1e867f28a8ff0b15203e5cd9548f74fcee 2025-08-14T21:24:12.9547783Z * [new tag] trunk/4a90dc0c1f68d1f98832b169f792ed1bb195a0f3 -> trunk/4a90dc0c1f68d1f98832b169f792ed1bb195a0f3 2025-08-14T21:24:12.9548095Z * [new tag] trunk/4cde0acc0e4e795e1a12cbdd9b93c8c04c1fa05d -> trunk/4cde0acc0e4e795e1a12cbdd9b93c8c04c1fa05d 2025-08-14T21:24:12.9548611Z * [new tag] trunk/4d419a74610c32b1372f8802dcc61893740a23cf -> trunk/4d419a74610c32b1372f8802dcc61893740a23cf 2025-08-14T21:24:12.9550848Z * [new tag] trunk/4d5b3f2d5af7c8e4f41da4ffca53fafe8bb86235 -> trunk/4d5b3f2d5af7c8e4f41da4ffca53fafe8bb86235 2025-08-14T21:24:12.9551265Z * [new tag] trunk/4e2ddb5db67617f9f5309c8bba0c17adc84cadbc -> trunk/4e2ddb5db67617f9f5309c8bba0c17adc84cadbc 2025-08-14T21:24:12.9551870Z * [new tag] trunk/50a8c118754a6c5a46968f5c8e215ccba6831d42 -> trunk/50a8c118754a6c5a46968f5c8e215ccba6831d42 2025-08-14T21:24:12.9552215Z * [new tag] trunk/50f23ff6f883db5021dd6bab4c146434f98dd15d -> trunk/50f23ff6f883db5021dd6bab4c146434f98dd15d 2025-08-14T21:24:12.9552582Z * [new tag] trunk/515cb70367e84fcbad23fcc5b39eb1d7706df2aa -> trunk/515cb70367e84fcbad23fcc5b39eb1d7706df2aa 2025-08-14T21:24:12.9552904Z * [new tag] trunk/53e39494958b7e2278cc8176f63636e812e8945f -> trunk/53e39494958b7e2278cc8176f63636e812e8945f 2025-08-14T21:24:12.9553207Z * [new tag] trunk/556e2a73f4f0643f7c2aeb5c7dddda43388a40ce -> trunk/556e2a73f4f0643f7c2aeb5c7dddda43388a40ce 2025-08-14T21:24:12.9553543Z * [new tag] trunk/5665dc9ab76b84d7c90d845ffb0f6349b3621919 -> trunk/5665dc9ab76b84d7c90d845ffb0f6349b3621919 2025-08-14T21:24:12.9553848Z * [new tag] trunk/566c6d52ef1411c8262d7b9cf85e2044fdfbe1a3 -> trunk/566c6d52ef1411c8262d7b9cf85e2044fdfbe1a3 2025-08-14T21:24:12.9554257Z * [new tag] trunk/56c828bef93eada0e18d2cc013207831ca80cc99 -> trunk/56c828bef93eada0e18d2cc013207831ca80cc99 2025-08-14T21:24:12.9554783Z * [new tag] trunk/5737372862253a0ac0292407a5844796f02380ad -> trunk/5737372862253a0ac0292407a5844796f02380ad 2025-08-14T21:24:12.9555655Z * [new tag] trunk/57f738b6357cc8fcdde479a0948e723809a1a44d -> trunk/57f738b6357cc8fcdde479a0948e723809a1a44d 2025-08-14T21:24:12.9556251Z * [new tag] trunk/5a40c5784482255b9baf14086cc4b9349fc6d512 -> trunk/5a40c5784482255b9baf14086cc4b9349fc6d512 2025-08-14T21:24:12.9556683Z * [new tag] trunk/5a9c4cfce42b9eb87da0de40c5633f083115c307 -> trunk/5a9c4cfce42b9eb87da0de40c5633f083115c307 2025-08-14T21:24:12.9557187Z * [new tag] trunk/5ace061254af71aa83d1baae81aa1864c9746add -> trunk/5ace061254af71aa83d1baae81aa1864c9746add 2025-08-14T21:24:12.9557683Z * [new tag] trunk/5dddcd5b07c6644efca8d613f4eca1dc95daa87f -> trunk/5dddcd5b07c6644efca8d613f4eca1dc95daa87f 2025-08-14T21:24:12.9558179Z * [new tag] trunk/5ed4f9177907fe403ec4c4499d0d0e9be6b68fcf -> trunk/5ed4f9177907fe403ec4c4499d0d0e9be6b68fcf 2025-08-14T21:24:12.9559813Z * [new tag] trunk/5f1010fbb3850d99c8fdf9a9de2f79260cdc586a -> trunk/5f1010fbb3850d99c8fdf9a9de2f79260cdc586a 2025-08-14T21:24:12.9560223Z * [new tag] trunk/5f5f508aa836a46dfe88857fb223049616b94e93 -> trunk/5f5f508aa836a46dfe88857fb223049616b94e93 2025-08-14T21:24:12.9560539Z * [new tag] trunk/62bac0798100e0e06a86b7a4cee1788413e3d0ca -> trunk/62bac0798100e0e06a86b7a4cee1788413e3d0ca 2025-08-14T21:24:12.9560846Z * [new tag] trunk/63654ba4c5178fd12220cfc9d1c878af2fdd07cc -> trunk/63654ba4c5178fd12220cfc9d1c878af2fdd07cc 2025-08-14T21:24:12.9561165Z * [new tag] trunk/639778b3ee3b80e0894367fdc4442b58ae1b3a62 -> trunk/639778b3ee3b80e0894367fdc4442b58ae1b3a62 2025-08-14T21:24:12.9562268Z * [new tag] trunk/641ee7478150f26969968f49d8b358e199679a8a -> trunk/641ee7478150f26969968f49d8b358e199679a8a 2025-08-14T21:24:12.9562943Z * [new tag] trunk/65053c03a3d209060cb239d20a229dac37cf9dd1 -> trunk/65053c03a3d209060cb239d20a229dac37cf9dd1 2025-08-14T21:24:12.9563306Z * [new tag] trunk/652a6f5954d039d61dc6e6575ccf89d385d74537 -> trunk/652a6f5954d039d61dc6e6575ccf89d385d74537 2025-08-14T21:24:12.9563791Z * [new tag] trunk/685f15dbea66e8ffa8564752f81ad2f6cb447a14 -> trunk/685f15dbea66e8ffa8564752f81ad2f6cb447a14 2025-08-14T21:24:12.9564267Z * [new tag] trunk/68a4b4b2e336cfd4451ce6546d900568e5ddf96c -> trunk/68a4b4b2e336cfd4451ce6546d900568e5ddf96c 2025-08-14T21:24:12.9564910Z * [new tag] trunk/69a0a9aa7f5e320a02e97fa789d2f72baff1554f -> trunk/69a0a9aa7f5e320a02e97fa789d2f72baff1554f 2025-08-14T21:24:12.9565664Z * [new tag] trunk/6be6d06295c870c77a6eb69f96b3170d983520d5 -> trunk/6be6d06295c870c77a6eb69f96b3170d983520d5 2025-08-14T21:24:12.9566212Z * [new tag] trunk/6c05ea6475beaf3acc05e1bda0f3f8fe3bdc1d49 -> trunk/6c05ea6475beaf3acc05e1bda0f3f8fe3bdc1d49 2025-08-14T21:24:12.9567002Z * [new tag] trunk/6da11d9aafc0d84dc7f66030c181608ff2614f66 -> trunk/6da11d9aafc0d84dc7f66030c181608ff2614f66 2025-08-14T21:24:12.9567461Z * [new tag] trunk/6e8865fbc161270e2ffc52817e6c667df417a3f7 -> trunk/6e8865fbc161270e2ffc52817e6c667df417a3f7 2025-08-14T21:24:12.9571397Z * [new tag] trunk/6ea8376f84232048d6be0f7b2edf82aec1b61d58 -> trunk/6ea8376f84232048d6be0f7b2edf82aec1b61d58 2025-08-14T21:24:12.9571794Z * [new tag] trunk/6ee175195ac7853734d64704171993cc6265eb38 -> trunk/6ee175195ac7853734d64704171993cc6265eb38 2025-08-14T21:24:12.9572172Z * [new tag] trunk/6f0f4e0c3eacd479864319127915f869f64e1935 -> trunk/6f0f4e0c3eacd479864319127915f869f64e1935 2025-08-14T21:24:12.9572527Z * [new tag] trunk/70ccdec44b89e355a2cb03ba14a634284f7750f8 -> trunk/70ccdec44b89e355a2cb03ba14a634284f7750f8 2025-08-14T21:24:12.9573230Z * [new tag] trunk/72009ec6bebca7714f99c18449183787f202af4d -> trunk/72009ec6bebca7714f99c18449183787f202af4d 2025-08-14T21:24:12.9573499Z * [new tag] trunk/731ee31f7b6ba19307daab323f6196172b71aaf8 -> trunk/731ee31f7b6ba19307daab323f6196172b71aaf8 2025-08-14T21:24:12.9573726Z * [new tag] trunk/76a0609b6bddb2bc40f1eb4ade12885023653d59 -> trunk/76a0609b6bddb2bc40f1eb4ade12885023653d59 2025-08-14T21:24:12.9573951Z * [new tag] trunk/781e9a7724c47496e3d38a81e6dd6194cf098c41 -> trunk/781e9a7724c47496e3d38a81e6dd6194cf098c41 2025-08-14T21:24:12.9574332Z * [new tag] trunk/78a2fe1d42edeaa2ef7020b0fa0ac82ee4a640e4 -> trunk/78a2fe1d42edeaa2ef7020b0fa0ac82ee4a640e4 2025-08-14T21:24:12.9574587Z * [new tag] trunk/7a974a88f2c529a614baeabe4debd00fc8a3b299 -> trunk/7a974a88f2c529a614baeabe4debd00fc8a3b299 2025-08-14T21:24:12.9574979Z * [new tag] trunk/7ae0629d64b404e0ef5d9c931433ad25e65d6114 -> trunk/7ae0629d64b404e0ef5d9c931433ad25e65d6114 2025-08-14T21:24:12.9575327Z * [new tag] trunk/7d2ec704e47f4b740cdecda5534b305e8e1875ef -> trunk/7d2ec704e47f4b740cdecda5534b305e8e1875ef 2025-08-14T21:24:12.9575634Z * [new tag] trunk/7d87e358ac8440f666fabbfd99058bb5342be6ac -> trunk/7d87e358ac8440f666fabbfd99058bb5342be6ac 2025-08-14T21:24:12.9575938Z * [new tag] trunk/7e27347fd353928c99620495c8c531a5eba7d56b -> trunk/7e27347fd353928c99620495c8c531a5eba7d56b 2025-08-14T21:24:12.9576445Z * [new tag] trunk/7e91394955721c77645fcdb75a5d47a255d65020 -> trunk/7e91394955721c77645fcdb75a5d47a255d65020 2025-08-14T21:24:12.9577066Z * [new tag] trunk/7f4cb4a3e018a621add2a37a3a2f67b982d51001 -> trunk/7f4cb4a3e018a621add2a37a3a2f67b982d51001 2025-08-14T21:24:12.9577579Z * [new tag] trunk/7fbc22855c17741ae016992803b2e147a13aa22d -> trunk/7fbc22855c17741ae016992803b2e147a13aa22d 2025-08-14T21:24:12.9578162Z * [new tag] trunk/8047421fbb607d70ede13b9cd5a60b7b8bdfe348 -> trunk/8047421fbb607d70ede13b9cd5a60b7b8bdfe348 2025-08-14T21:24:12.9578584Z * [new tag] trunk/8088cfa592504a2897b4c78f8a46fe658ab5c2c2 -> trunk/8088cfa592504a2897b4c78f8a46fe658ab5c2c2 2025-08-14T21:24:12.9581077Z * [new tag] trunk/80cca8307943ba64168208b54028f55b2c71daff -> trunk/80cca8307943ba64168208b54028f55b2c71daff 2025-08-14T21:24:12.9581481Z * [new tag] trunk/8147370733bbdcd034cad54e9212e51885a11892 -> trunk/8147370733bbdcd034cad54e9212e51885a11892 2025-08-14T21:24:12.9581848Z * [new tag] trunk/83875cdb5594ccb3c9206b8eb5745fe1d011cf26 -> trunk/83875cdb5594ccb3c9206b8eb5745fe1d011cf26 2025-08-14T21:24:12.9582344Z * [new tag] trunk/8399cf88ce8399d2be93355f29d4cb69f51c0654 -> trunk/8399cf88ce8399d2be93355f29d4cb69f51c0654 2025-08-14T21:24:12.9583123Z * [new tag] trunk/842cc77ab9aafd518593c2fce077d6abb42a5b7f -> trunk/842cc77ab9aafd518593c2fce077d6abb42a5b7f 2025-08-14T21:24:12.9583392Z * [new tag] trunk/85db508af533649d0b3447ff3f0d5fe083150c84 -> trunk/85db508af533649d0b3447ff3f0d5fe083150c84 2025-08-14T21:24:12.9583603Z * [new tag] trunk/86eb65f7f06016bcd5d7951dc9d74bc3993a827a -> trunk/86eb65f7f06016bcd5d7951dc9d74bc3993a827a 2025-08-14T21:24:12.9583810Z * [new tag] trunk/87e6c4079d8ec7d04aff00ed82096b39836a8367 -> trunk/87e6c4079d8ec7d04aff00ed82096b39836a8367 2025-08-14T21:24:12.9584033Z * [new tag] trunk/89654db1abccf7e5f261989a150db4d1619ea2aa -> trunk/89654db1abccf7e5f261989a150db4d1619ea2aa 2025-08-14T21:24:12.9584259Z * [new tag] trunk/8a37f0c90392a2c38b7c5955471fa49edcaf5cb1 -> trunk/8a37f0c90392a2c38b7c5955471fa49edcaf5cb1 2025-08-14T21:24:12.9584573Z * [new tag] trunk/8ab5868a2199fe485c2d66533b9244ccb97e487d -> trunk/8ab5868a2199fe485c2d66533b9244ccb97e487d 2025-08-14T21:24:12.9585022Z * [new tag] trunk/8ae4d2652f64b8444b3d5314b9232bd2119bcde6 -> trunk/8ae4d2652f64b8444b3d5314b9232bd2119bcde6 2025-08-14T21:24:12.9585571Z * [new tag] trunk/8c41cb800ae0411f02ea5da34bd5ccc3790633b0 -> trunk/8c41cb800ae0411f02ea5da34bd5ccc3790633b0 2025-08-14T21:24:12.9587234Z * [new tag] trunk/8cb91e20bc205b1416648d0ffd98d1ba1f3a6fc4 -> trunk/8cb91e20bc205b1416648d0ffd98d1ba1f3a6fc4 2025-08-14T21:24:12.9587647Z * [new tag] trunk/8cfaf51d4e29c9bd9f49ecc98d955ed53df1a13d -> trunk/8cfaf51d4e29c9bd9f49ecc98d955ed53df1a13d 2025-08-14T21:24:12.9587995Z * [new tag] trunk/8d1cf529229dce7cd5ea04abb0faac83b87ca6d1 -> trunk/8d1cf529229dce7cd5ea04abb0faac83b87ca6d1 2025-08-14T21:24:12.9588441Z * [new tag] trunk/8d3d1c844303cb1d46123a1caa76d4cf83973347 -> trunk/8d3d1c844303cb1d46123a1caa76d4cf83973347 2025-08-14T21:24:12.9588777Z * [new tag] trunk/8d6d3246316e1767a57d5e855acd6208da753b75 -> trunk/8d6d3246316e1767a57d5e855acd6208da753b75 2025-08-14T21:24:12.9589084Z * [new tag] trunk/8e6a3138581152ab827a0997f34c470271399f5e -> trunk/8e6a3138581152ab827a0997f34c470271399f5e 2025-08-14T21:24:12.9589722Z * [new tag] trunk/8eee08d2279b98af2522debb6512d37e837e89e3 -> trunk/8eee08d2279b98af2522debb6512d37e837e89e3 2025-08-14T21:24:12.9589999Z * [new tag] trunk/90b78ee50f73b5c963996076a3d54b74b1b965be -> trunk/90b78ee50f73b5c963996076a3d54b74b1b965be 2025-08-14T21:24:12.9590400Z * [new tag] trunk/94b91a876327820a4bb6f5d39d156f13f2553ab6 -> trunk/94b91a876327820a4bb6f5d39d156f13f2553ab6 2025-08-14T21:24:12.9591503Z * [new tag] trunk/95210cc409dd578988c7116b47725c304dea54c7 -> trunk/95210cc409dd578988c7116b47725c304dea54c7 2025-08-14T21:24:12.9591817Z * [new tag] trunk/96bd33b2de79598566df395f32e27c4d33673f05 -> trunk/96bd33b2de79598566df395f32e27c4d33673f05 2025-08-14T21:24:12.9592289Z * [new tag] trunk/9708fcf92db88b80b9010c68662d634434da3106 -> trunk/9708fcf92db88b80b9010c68662d634434da3106 2025-08-14T21:24:12.9592890Z * [new tag] trunk/97c8c98f8dcb9c5c188b691d156e0043dba6c7f8 -> trunk/97c8c98f8dcb9c5c188b691d156e0043dba6c7f8 2025-08-14T21:24:12.9593403Z * [new tag] trunk/9903ca4f70bdc1653016256f5b4fd74fdfc609f8 -> trunk/9903ca4f70bdc1653016256f5b4fd74fdfc609f8 2025-08-14T21:24:12.9595513Z * [new tag] trunk/99bc2f94c1955657e950ebdad5f77e518785ccbd -> trunk/99bc2f94c1955657e950ebdad5f77e518785ccbd 2025-08-14T21:24:12.9595771Z * [new tag] trunk/9a06e6d0310da9d8a59ae05e8ec9c0201b55cacd -> trunk/9a06e6d0310da9d8a59ae05e8ec9c0201b55cacd 2025-08-14T21:24:12.9596028Z * [new tag] trunk/9a0f7a3bb01b235ea04581ee540970a098071b72 -> trunk/9a0f7a3bb01b235ea04581ee540970a098071b72 2025-08-14T21:24:12.9596419Z * [new tag] trunk/9b803cdbe298009f08340c1aaccb25aafbca95d8 -> trunk/9b803cdbe298009f08340c1aaccb25aafbca95d8 2025-08-14T21:24:12.9596823Z * [new tag] trunk/9ccd0f5e31ea54fcf42101dfbaacc103494e34df -> trunk/9ccd0f5e31ea54fcf42101dfbaacc103494e34df 2025-08-14T21:24:12.9597152Z * [new tag] trunk/9d37c960a4fc44d5ac334ca8bf775f85b95d76fc -> trunk/9d37c960a4fc44d5ac334ca8bf775f85b95d76fc 2025-08-14T21:24:12.9597536Z * [new tag] trunk/9e07673deb212c87b1c6fea23799a97474c476ed -> trunk/9e07673deb212c87b1c6fea23799a97474c476ed 2025-08-14T21:24:12.9597998Z * [new tag] trunk/9eedd2a20b64302d0d116ea2802b50948d2ebb09 -> trunk/9eedd2a20b64302d0d116ea2802b50948d2ebb09 2025-08-14T21:24:12.9599382Z * [new tag] trunk/9fa8ce26cf638504469852cbc3e7d04579fc8674 -> trunk/9fa8ce26cf638504469852cbc3e7d04579fc8674 2025-08-14T21:24:12.9600001Z * [new tag] trunk/a06ec54d40013c97fbffc174ea8f524ea5a95715 -> trunk/a06ec54d40013c97fbffc174ea8f524ea5a95715 2025-08-14T21:24:12.9600278Z * [new tag] trunk/a288b15ea9f87ddd665f249d492e0fb0861f5a69 -> trunk/a288b15ea9f87ddd665f249d492e0fb0861f5a69 2025-08-14T21:24:12.9600886Z * [new tag] trunk/a2fd106d670bb4990cebfd00f25ecbae4145e76c -> trunk/a2fd106d670bb4990cebfd00f25ecbae4145e76c 2025-08-14T21:24:12.9601568Z * [new tag] trunk/a354fa91e26b376d96385a2206c5ff5b42aa4600 -> trunk/a354fa91e26b376d96385a2206c5ff5b42aa4600 2025-08-14T21:24:12.9602075Z * [new tag] trunk/a4f69a5da08eace1c1e6469dec6a18aa842da73b -> trunk/a4f69a5da08eace1c1e6469dec6a18aa842da73b 2025-08-14T21:24:12.9602776Z * [new tag] trunk/a53d14d5f846ac44f6c205abb1c5bc4d2f3126ae -> trunk/a53d14d5f846ac44f6c205abb1c5bc4d2f3126ae 2025-08-14T21:24:12.9603446Z * [new tag] trunk/a5652407e4f3d772fc44486ac2abf756decf0861 -> trunk/a5652407e4f3d772fc44486ac2abf756decf0861 2025-08-14T21:24:12.9604150Z * [new tag] trunk/a7abf57aabec0ce686092e2d66e53ba185dbc56b -> trunk/a7abf57aabec0ce686092e2d66e53ba185dbc56b 2025-08-14T21:24:12.9604755Z * [new tag] trunk/a84b60c0c4016785fd93b7b8a0c04f2d0770d332 -> trunk/a84b60c0c4016785fd93b7b8a0c04f2d0770d332 2025-08-14T21:24:12.9605358Z * [new tag] trunk/aa75e917bdb0f95bb6dee81853c2d3c4ab3e1883 -> trunk/aa75e917bdb0f95bb6dee81853c2d3c4ab3e1883 2025-08-14T21:24:12.9606039Z * [new tag] trunk/adcca7d9a1c053495e99012de801b2ea237faad0 -> trunk/adcca7d9a1c053495e99012de801b2ea237faad0 2025-08-14T21:24:12.9606691Z * [new tag] trunk/af10f1f86cc4effc93142a447693d8be55966615 -> trunk/af10f1f86cc4effc93142a447693d8be55966615 2025-08-14T21:24:12.9607331Z * [new tag] trunk/af3cabc55d5699f4da528e1ca39d83338f84ae8c -> trunk/af3cabc55d5699f4da528e1ca39d83338f84ae8c 2025-08-14T21:24:12.9607927Z * [new tag] trunk/b0df7715e8c590c0001d1f9cdb97057be80c9107 -> trunk/b0df7715e8c590c0001d1f9cdb97057be80c9107 2025-08-14T21:24:12.9608483Z * [new tag] trunk/b149c7204c218e7c4d6594a89dd74f72bd480ec5 -> trunk/b149c7204c218e7c4d6594a89dd74f72bd480ec5 2025-08-14T21:24:12.9609116Z * [new tag] trunk/b1a602762e6a6674b406a3137e7e7a678885a97b -> trunk/b1a602762e6a6674b406a3137e7e7a678885a97b 2025-08-14T21:24:12.9609646Z * [new tag] trunk/b1f43548cad8fc0e30bda250f6e196310fa7a4bc -> trunk/b1f43548cad8fc0e30bda250f6e196310fa7a4bc 2025-08-14T21:24:12.9610302Z * [new tag] trunk/b219ca2a00a305753c4f1ea4c9c5d23243d54753 -> trunk/b219ca2a00a305753c4f1ea4c9c5d23243d54753 2025-08-14T21:24:12.9610855Z * [new tag] trunk/b4596895b9d85a686c2cb978938b0a7797b3690a -> trunk/b4596895b9d85a686c2cb978938b0a7797b3690a 2025-08-14T21:24:12.9611548Z * [new tag] trunk/b5fd7223b1bf44720dc9183bda7dfcf7aeccff02 -> trunk/b5fd7223b1bf44720dc9183bda7dfcf7aeccff02 2025-08-14T21:24:12.9612137Z * [new tag] trunk/b602ea9cab7d43a7ee7b4051227090f23fbd3dbf -> trunk/b602ea9cab7d43a7ee7b4051227090f23fbd3dbf 2025-08-14T21:24:12.9613083Z * [new tag] trunk/b6b74aed604bd2e96389ff99aaaf39abc64fdc64 -> trunk/b6b74aed604bd2e96389ff99aaaf39abc64fdc64 2025-08-14T21:24:12.9613613Z * [new tag] trunk/b7db86600a2614adc71c92ca42d359a7ac534d78 -> trunk/b7db86600a2614adc71c92ca42d359a7ac534d78 2025-08-14T21:24:12.9613983Z * [new tag] trunk/b9003ed3d87699e81e436719625a21996a6654e5 -> trunk/b9003ed3d87699e81e436719625a21996a6654e5 2025-08-14T21:24:12.9614702Z * [new tag] trunk/b90feeac86bda00afc2789321bcd706015ff44e3 -> trunk/b90feeac86bda00afc2789321bcd706015ff44e3 2025-08-14T21:24:12.9619719Z * [new tag] trunk/b9d7de3a094598c3dc0dd52e57bce30eb684c9d8 -> trunk/b9d7de3a094598c3dc0dd52e57bce30eb684c9d8 2025-08-14T21:24:12.9620033Z * [new tag] trunk/ba47821f524eee50a214ed39fa2e7765d54aabf4 -> trunk/ba47821f524eee50a214ed39fa2e7765d54aabf4 2025-08-14T21:24:12.9620296Z * [new tag] trunk/ba4ccf5d67e3d237f435eacc2bce3c6025f08491 -> trunk/ba4ccf5d67e3d237f435eacc2bce3c6025f08491 2025-08-14T21:24:12.9620535Z * [new tag] trunk/bcf23ecc476df2bd7479f142567213e2623308ee -> trunk/bcf23ecc476df2bd7479f142567213e2623308ee 2025-08-14T21:24:12.9620783Z * [new tag] trunk/be53f609aaf6f01e2863f490975ea9eaac3ee9ff -> trunk/be53f609aaf6f01e2863f490975ea9eaac3ee9ff 2025-08-14T21:24:12.9621021Z * [new tag] trunk/beb4d7816dedc67a5de1f82e5a45b5910f407941 -> trunk/beb4d7816dedc67a5de1f82e5a45b5910f407941 2025-08-14T21:24:12.9621259Z * [new tag] trunk/bfc873d02ec413344717493e4175a902921359fd -> trunk/bfc873d02ec413344717493e4175a902921359fd 2025-08-14T21:24:12.9621647Z * [new tag] trunk/c184cb3852f0ff2d16a489d61abc3739c309e6ca -> trunk/c184cb3852f0ff2d16a489d61abc3739c309e6ca 2025-08-14T21:24:12.9621897Z * [new tag] trunk/c24ca7f4bf79f62fd623d76346ca27e53f731431 -> trunk/c24ca7f4bf79f62fd623d76346ca27e53f731431 2025-08-14T21:24:12.9622143Z * [new tag] trunk/c3dc8dc4122977893004c49d10e4676cd0a97da4 -> trunk/c3dc8dc4122977893004c49d10e4676cd0a97da4 2025-08-14T21:24:12.9622384Z * [new tag] trunk/c5ec5458a547f7a774468ea0eb2258d3de596492 -> trunk/c5ec5458a547f7a774468ea0eb2258d3de596492 2025-08-14T21:24:12.9622624Z * [new tag] trunk/c5efc5c8a66eca84865015058b3221013ebfe685 -> trunk/c5efc5c8a66eca84865015058b3221013ebfe685 2025-08-14T21:24:12.9622895Z * [new tag] trunk/c6563341208003f64c131854a9cf029555f786d2 -> trunk/c6563341208003f64c131854a9cf029555f786d2 2025-08-14T21:24:12.9623518Z * [new tag] trunk/c6d78d4dbda53837d298d23a5fbc09af90a42d9e -> trunk/c6d78d4dbda53837d298d23a5fbc09af90a42d9e 2025-08-14T21:24:12.9624109Z * [new tag] trunk/c8205cb35435f39d2c26f6c94b45e4adeb6dcb23 -> trunk/c8205cb35435f39d2c26f6c94b45e4adeb6dcb23 2025-08-14T21:24:12.9624617Z * [new tag] trunk/c859ba7114b1fcb49527e090745fa17091d1f8d5 -> trunk/c859ba7114b1fcb49527e090745fa17091d1f8d5 2025-08-14T21:24:12.9625142Z * [new tag] trunk/c86040a8e68f754b90a84099187d3624954c7f36 -> trunk/c86040a8e68f754b90a84099187d3624954c7f36 2025-08-14T21:24:12.9629437Z * [new tag] trunk/c9671dc865aa0fc1cb86df754e355b44d8e02bb4 -> trunk/c9671dc865aa0fc1cb86df754e355b44d8e02bb4 2025-08-14T21:24:12.9629725Z * [new tag] trunk/ca7315c17162ea21b1ca5ba23f4bf6168766c7b9 -> trunk/ca7315c17162ea21b1ca5ba23f4bf6168766c7b9 2025-08-14T21:24:12.9629968Z * [new tag] trunk/cae2b5e3d223829bdc553fc8601df4b1c1554cff -> trunk/cae2b5e3d223829bdc553fc8601df4b1c1554cff 2025-08-14T21:24:12.9630191Z * [new tag] trunk/cbffde774557752cf20447d42d99ec6102673c31 -> trunk/cbffde774557752cf20447d42d99ec6102673c31 2025-08-14T21:24:12.9630456Z * [new tag] trunk/cd8d8c18f5bafdc1c73d5ac0129e7b4d76ab45bc -> trunk/cd8d8c18f5bafdc1c73d5ac0129e7b4d76ab45bc 2025-08-14T21:24:12.9630854Z * [new tag] trunk/cf0a0dcb0afa5e84b95461cc542f862b51ca96bf -> trunk/cf0a0dcb0afa5e84b95461cc542f862b51ca96bf 2025-08-14T21:24:12.9631095Z * [new tag] trunk/cf4964be68fa9f4ffc334f01cce42d7424b1cc81 -> trunk/cf4964be68fa9f4ffc334f01cce42d7424b1cc81 2025-08-14T21:24:12.9631349Z * [new tag] trunk/d0e2240f680ea2a553f7ee8188f52482e130bfd0 -> trunk/d0e2240f680ea2a553f7ee8188f52482e130bfd0 2025-08-14T21:24:12.9631601Z * [new tag] trunk/d1950d4bb5cba8fb6b23e4d283eea5b9801737e2 -> trunk/d1950d4bb5cba8fb6b23e4d283eea5b9801737e2 2025-08-14T21:24:12.9631869Z * [new tag] trunk/d20c4c20e61adecf00335c4d8c22eb1ace472cd3 -> trunk/d20c4c20e61adecf00335c4d8c22eb1ace472cd3 2025-08-14T21:24:12.9632125Z * [new tag] trunk/d25c4f954d599ea512e2f70cd6df101c21479d4c -> trunk/d25c4f954d599ea512e2f70cd6df101c21479d4c 2025-08-14T21:24:12.9632908Z * [new tag] trunk/d3d359dbafa89173a371e2637f22b47398e94a24 -> trunk/d3d359dbafa89173a371e2637f22b47398e94a24 2025-08-14T21:24:12.9633559Z * [new tag] trunk/d46768db04499d07a5b0db984112a6d1b7d3b0c1 -> trunk/d46768db04499d07a5b0db984112a6d1b7d3b0c1 2025-08-14T21:24:12.9634234Z * [new tag] trunk/d4c1a08c89f37d249a0146ff511c82ecc5c53b8f -> trunk/d4c1a08c89f37d249a0146ff511c82ecc5c53b8f 2025-08-14T21:24:12.9634841Z * [new tag] trunk/d556586448f3caab85673c7da0978fe31c7748f7 -> trunk/d556586448f3caab85673c7da0978fe31c7748f7 2025-08-14T21:24:12.9635569Z * [new tag] trunk/d670304001429a1a833255a918ed788d7ec4989a -> trunk/d670304001429a1a833255a918ed788d7ec4989a 2025-08-14T21:24:12.9636091Z * [new tag] trunk/d6786741a77aba200c78002646cc069b7a1799b0 -> trunk/d6786741a77aba200c78002646cc069b7a1799b0 2025-08-14T21:24:12.9637074Z * [new tag] trunk/d68c323692dedcbb74e670801e3502944fd790ff -> trunk/d68c323692dedcbb74e670801e3502944fd790ff 2025-08-14T21:24:12.9637581Z * [new tag] trunk/d8cb3db5339b45e4b745b2b883ef3ecde9843e2c -> trunk/d8cb3db5339b45e4b745b2b883ef3ecde9843e2c 2025-08-14T21:24:12.9638127Z * [new tag] trunk/da1f608ca33f3062535d0a4866d95db19e72fcbd -> trunk/da1f608ca33f3062535d0a4866d95db19e72fcbd 2025-08-14T21:24:12.9638512Z * [new tag] trunk/db0b7f1cc9bb3fe71aaf8b964a644147ae8e1c35 -> trunk/db0b7f1cc9bb3fe71aaf8b964a644147ae8e1c35 2025-08-14T21:24:12.9639633Z * [new tag] trunk/db32b60662b2f2bdcad980127d5dc4b66b02a7e4 -> trunk/db32b60662b2f2bdcad980127d5dc4b66b02a7e4 2025-08-14T21:24:12.9639953Z * [new tag] trunk/db763b17175553ba09637362eb9773a91997a7ad -> trunk/db763b17175553ba09637362eb9773a91997a7ad 2025-08-14T21:24:12.9640694Z * [new tag] trunk/db78943a1ca13a32a3d6045eb15e2b719ee13a2f -> trunk/db78943a1ca13a32a3d6045eb15e2b719ee13a2f 2025-08-14T21:24:12.9641242Z * [new tag] trunk/dc0d18e023d9b7e314ebba0f234b6cb1579dbcfd -> trunk/dc0d18e023d9b7e314ebba0f234b6cb1579dbcfd 2025-08-14T21:24:12.9641716Z * [new tag] trunk/dd21c8a578038ab2841a7ba809a06921093ac9d8 -> trunk/dd21c8a578038ab2841a7ba809a06921093ac9d8 2025-08-14T21:24:12.9642397Z * [new tag] trunk/deea71a90e05eb320c04bebfead5317746637f0d -> trunk/deea71a90e05eb320c04bebfead5317746637f0d 2025-08-14T21:24:12.9642988Z * [new tag] trunk/df55ec7d4b35f6d21691e9dd41c82f27de762948 -> trunk/df55ec7d4b35f6d21691e9dd41c82f27de762948 2025-08-14T21:24:12.9643518Z * [new tag] trunk/e1cf0d496ea85d1807c8c740f296e77bf7bdc1df -> trunk/e1cf0d496ea85d1807c8c740f296e77bf7bdc1df 2025-08-14T21:24:12.9644281Z * [new tag] trunk/e248719ac03c103767ab72034f6b9fd56855bf98 -> trunk/e248719ac03c103767ab72034f6b9fd56855bf98 2025-08-14T21:24:12.9644949Z * [new tag] trunk/e49762026070f66be41bfa6537fbcf9bfc24e558 -> trunk/e49762026070f66be41bfa6537fbcf9bfc24e558 2025-08-14T21:24:12.9645422Z * [new tag] trunk/e4de93f6a3e342bab34d3757cf90ec0ccc87e168 -> trunk/e4de93f6a3e342bab34d3757cf90ec0ccc87e168 2025-08-14T21:24:12.9646046Z * [new tag] trunk/e619c6bb90b9dedaccd3cbeed86a288993a4e33f -> trunk/e619c6bb90b9dedaccd3cbeed86a288993a4e33f 2025-08-14T21:24:12.9646706Z * [new tag] trunk/e63c2b21c186a7d2ab8a8953b8aa1535f2e96e58 -> trunk/e63c2b21c186a7d2ab8a8953b8aa1535f2e96e58 2025-08-14T21:24:12.9647272Z * [new tag] trunk/e7152ff8a6a929a0db7f3f4a72a5b6d471769cd3 -> trunk/e7152ff8a6a929a0db7f3f4a72a5b6d471769cd3 2025-08-14T21:24:12.9651185Z * [new tag] trunk/e96c7c4bb0f6aeae2ab3b6f040f7d67edbec199a -> trunk/e96c7c4bb0f6aeae2ab3b6f040f7d67edbec199a 2025-08-14T21:24:12.9651447Z * [new tag] trunk/e9eb2096a59a79e7a94c3e28a0715e040369f34c -> trunk/e9eb2096a59a79e7a94c3e28a0715e040369f34c 2025-08-14T21:24:12.9651686Z * [new tag] trunk/eac2d9d695a32dd456050f45cac35134ec3809f4 -> trunk/eac2d9d695a32dd456050f45cac35134ec3809f4 2025-08-14T21:24:12.9651922Z * [new tag] trunk/ecde76c764752540edf9ef62a97936c86d984b17 -> trunk/ecde76c764752540edf9ef62a97936c86d984b17 2025-08-14T21:24:12.9652139Z * [new tag] trunk/ecea81117b2fdc52907c97b3c32d779e07b5d55b -> trunk/ecea81117b2fdc52907c97b3c32d779e07b5d55b 2025-08-14T21:24:12.9652359Z * [new tag] trunk/edaa151d0d5a4e75fbec9843f49cc78770eb61fb -> trunk/edaa151d0d5a4e75fbec9843f49cc78770eb61fb 2025-08-14T21:24:12.9652579Z * [new tag] trunk/ee1b0412b919dfb358d5a697b3be49621497fbc2 -> trunk/ee1b0412b919dfb358d5a697b3be49621497fbc2 2025-08-14T21:24:12.9652951Z * [new tag] trunk/ee1fb43450c2e985657f95a91b68328d6f20f24e -> trunk/ee1fb43450c2e985657f95a91b68328d6f20f24e 2025-08-14T21:24:12.9654059Z * [new tag] trunk/ee89cc7a0acd69de25f98fe4ef828546db7b444c -> trunk/ee89cc7a0acd69de25f98fe4ef828546db7b444c 2025-08-14T21:24:12.9654417Z * [new tag] trunk/ee9f8ba11d664b871a9e0c7933fdc8571635b78c -> trunk/ee9f8ba11d664b871a9e0c7933fdc8571635b78c 2025-08-14T21:24:12.9654715Z * [new tag] trunk/eed9dbf70f43ee529fec78ac00ed9a4fd74c6e76 -> trunk/eed9dbf70f43ee529fec78ac00ed9a4fd74c6e76 2025-08-14T21:24:12.9655006Z * [new tag] trunk/f077c2402e4eb5b0ed562b4ee5b7a0503f26ef94 -> trunk/f077c2402e4eb5b0ed562b4ee5b7a0503f26ef94 2025-08-14T21:24:12.9655261Z * [new tag] trunk/f0980fc0bbd656d6c02d23ad97e945353b314f35 -> trunk/f0980fc0bbd656d6c02d23ad97e945353b314f35 2025-08-14T21:24:12.9655817Z * [new tag] trunk/f15ada5c6fad97a7dcbfa4673f067b6942dda640 -> trunk/f15ada5c6fad97a7dcbfa4673f067b6942dda640 2025-08-14T21:24:12.9656322Z * [new tag] trunk/f27232a2134150cb5e55d26a74d8c36c6a961ca5 -> trunk/f27232a2134150cb5e55d26a74d8c36c6a961ca5 2025-08-14T21:24:12.9656865Z * [new tag] trunk/f33ce40bc062a281e1a1f57e8c1926d0a7d155cc -> trunk/f33ce40bc062a281e1a1f57e8c1926d0a7d155cc 2025-08-14T21:24:12.9659037Z * [new tag] trunk/f341077ce4710172da20cfad916ee37159bfe9fe -> trunk/f341077ce4710172da20cfad916ee37159bfe9fe 2025-08-14T21:24:12.9659449Z * [new tag] trunk/f3a4d742ece08de4cb0e59dcc62e0093a7d0b0c7 -> trunk/f3a4d742ece08de4cb0e59dcc62e0093a7d0b0c7 2025-08-14T21:24:12.9659806Z * [new tag] trunk/f3f159ff8c4bad2edec99c68a941c628e983d04c -> trunk/f3f159ff8c4bad2edec99c68a941c628e983d04c 2025-08-14T21:24:12.9660047Z * [new tag] trunk/f60454cce8b93e5bbf67f2f3c88c8ac01ed65457 -> trunk/f60454cce8b93e5bbf67f2f3c88c8ac01ed65457 2025-08-14T21:24:12.9660267Z * [new tag] trunk/f7b2f3314cf7aede67d5fa5c75e4243208484344 -> trunk/f7b2f3314cf7aede67d5fa5c75e4243208484344 2025-08-14T21:24:12.9660555Z * [new tag] trunk/f8f0414a5983ff481a2188e0c18594150430c8c5 -> trunk/f8f0414a5983ff481a2188e0c18594150430c8c5 2025-08-14T21:24:12.9661478Z * [new tag] trunk/f95b58c2844b3444cd8446fed8570729dc4216eb -> trunk/f95b58c2844b3444cd8446fed8570729dc4216eb 2025-08-14T21:24:12.9662053Z * [new tag] trunk/f990490a23815ea6ee27e487c70ba2cf513ba43d -> trunk/f990490a23815ea6ee27e487c70ba2cf513ba43d 2025-08-14T21:24:12.9662617Z * [new tag] trunk/fb887c3bb588cfe782615e67f6c26db636b8539b -> trunk/fb887c3bb588cfe782615e67f6c26db636b8539b 2025-08-14T21:24:12.9664959Z * [new tag] trunk/fc25c68f20f772290927a7031b998b92615259cf -> trunk/fc25c68f20f772290927a7031b998b92615259cf 2025-08-14T21:24:12.9665365Z * [new tag] trunk/fc80f6859e0ccf66513a40f04b9e735e759d4ddb -> trunk/fc80f6859e0ccf66513a40f04b9e735e759d4ddb 2025-08-14T21:24:12.9665725Z * [new tag] trunk/fdfd69bb05488d76123db9cc1cdd90ac4137bbfb -> trunk/fdfd69bb05488d76123db9cc1cdd90ac4137bbfb 2025-08-14T21:24:12.9666058Z * [new tag] trunk/fe3f5fe4ea2ff6f56406dc5d954636ebb08d0a08 -> trunk/fe3f5fe4ea2ff6f56406dc5d954636ebb08d0a08 2025-08-14T21:24:12.9666413Z * [new tag] trunk/fea7e9dd37c02c334b130f6624af6163fde6b2ab -> trunk/fea7e9dd37c02c334b130f6624af6163fde6b2ab 2025-08-14T21:24:12.9667112Z * [new tag] trunk/ff0d56d03592aa03f3ced8359241d21df1783393 -> trunk/ff0d56d03592aa03f3ced8359241d21df1783393 2025-08-14T21:24:12.9667261Z * [new tag] v0.1.1 -> v0.1.1 2025-08-14T21:24:12.9670165Z * [new tag] v0.1.10 -> v0.1.10 2025-08-14T21:24:12.9670430Z * [new tag] v0.1.11 -> v0.1.11 2025-08-14T21:24:12.9670533Z * [new tag] v0.1.12 -> v0.1.12 2025-08-14T21:24:12.9670635Z * [new tag] v0.1.2 -> v0.1.2 2025-08-14T21:24:12.9670720Z * [new tag] v0.1.3 -> v0.1.3 2025-08-14T21:24:12.9671108Z * [new tag] v0.1.4 -> v0.1.4 2025-08-14T21:24:12.9671349Z * [new tag] v0.1.5 -> v0.1.5 2025-08-14T21:24:12.9671452Z * [new tag] v0.1.6 -> v0.1.6 2025-08-14T21:24:12.9671847Z * [new tag] v0.1.7 -> v0.1.7 2025-08-14T21:24:12.9672485Z * [new tag] v0.1.8 -> v0.1.8 2025-08-14T21:24:12.9672802Z * [new tag] v0.1.9 -> v0.1.9 2025-08-14T21:24:12.9675503Z * [new tag] v0.2.0 -> v0.2.0 2025-08-14T21:24:12.9675783Z * [new tag] v0.3.0 -> v0.3.0 2025-08-14T21:24:12.9675886Z * [new tag] v0.3.1 -> v0.3.1 2025-08-14T21:24:12.9675980Z * [new tag] v0.4.0 -> v0.4.0 2025-08-14T21:24:12.9676063Z * [new tag] v0.4.1 -> v0.4.1 2025-08-14T21:24:12.9676300Z * [new tag] v1.0.0 -> v1.0.0 2025-08-14T21:24:12.9676424Z * [new tag] v1.0.0a0 -> v1.0.0a0 2025-08-14T21:24:12.9677140Z * [new tag] v1.0.1 -> v1.0.1 2025-08-14T21:24:12.9677535Z * [new tag] v1.0rc0 -> v1.0rc0 2025-08-14T21:24:12.9677907Z * [new tag] v1.0rc1 -> v1.0rc1 2025-08-14T21:24:12.9680649Z * [new tag] v1.1.0 -> v1.1.0 2025-08-14T21:24:12.9680776Z * [new tag] v1.1.0a0 -> v1.1.0a0 2025-08-14T21:24:12.9680883Z * [new tag] v1.10.0 -> v1.10.0 2025-08-14T21:24:12.9680984Z * [new tag] v1.10.0-rc1 -> v1.10.0-rc1 2025-08-14T21:24:12.9681083Z * [new tag] v1.10.0-rc2 -> v1.10.0-rc2 2025-08-14T21:24:12.9681438Z * [new tag] v1.10.0-rc3 -> v1.10.0-rc3 2025-08-14T21:24:12.9681817Z * [new tag] v1.10.1 -> v1.10.1 2025-08-14T21:24:12.9682251Z * [new tag] v1.10.1-rc1 -> v1.10.1-rc1 2025-08-14T21:24:12.9682621Z * [new tag] v1.10.2 -> v1.10.2 2025-08-14T21:24:12.9683120Z * [new tag] v1.10.2-rc1 -> v1.10.2-rc1 2025-08-14T21:24:12.9683894Z * [new tag] v1.11.0 -> v1.11.0 2025-08-14T21:24:12.9684411Z * [new tag] v1.11.0-rc1 -> v1.11.0-rc1 2025-08-14T21:24:12.9685309Z * [new tag] v1.11.0-rc2 -> v1.11.0-rc2 2025-08-14T21:24:12.9685778Z * [new tag] v1.11.0-rc3 -> v1.11.0-rc3 2025-08-14T21:24:12.9687023Z * [new tag] v1.11.0-rc4 -> v1.11.0-rc4 2025-08-14T21:24:12.9687251Z * [new tag] v1.11.0-rc5 -> v1.11.0-rc5 2025-08-14T21:24:12.9687614Z * [new tag] v1.11.0-rc6 -> v1.11.0-rc6 2025-08-14T21:24:12.9687989Z * [new tag] v1.11.0-rc7 -> v1.11.0-rc7 2025-08-14T21:24:12.9690256Z * [new tag] v1.12.0 -> v1.12.0 2025-08-14T21:24:12.9690401Z * [new tag] v1.12.0-rc1 -> v1.12.0-rc1 2025-08-14T21:24:12.9690505Z * [new tag] v1.12.0-rc2 -> v1.12.0-rc2 2025-08-14T21:24:12.9690619Z * [new tag] v1.12.0-rc3 -> v1.12.0-rc3 2025-08-14T21:24:12.9691076Z * [new tag] v1.12.0-rc4 -> v1.12.0-rc4 2025-08-14T21:24:12.9692628Z * [new tag] v1.12.0-rc5 -> v1.12.0-rc5 2025-08-14T21:24:12.9692921Z * [new tag] v1.12.0-rc6 -> v1.12.0-rc6 2025-08-14T21:24:12.9693039Z * [new tag] v1.12.0-rc7 -> v1.12.0-rc7 2025-08-14T21:24:12.9693328Z * [new tag] v1.12.0-rc8 -> v1.12.0-rc8 2025-08-14T21:24:12.9693970Z * [new tag] v1.12.1 -> v1.12.1 2025-08-14T21:24:12.9694707Z * [new tag] v1.12.1-rc1 -> v1.12.1-rc1 2025-08-14T21:24:12.9694875Z * [new tag] v1.12.1-rc2 -> v1.12.1-rc2 2025-08-14T21:24:12.9697514Z * [new tag] v1.12.1-rc3 -> v1.12.1-rc3 2025-08-14T21:24:12.9697799Z * [new tag] v1.12.1-rc4 -> v1.12.1-rc4 2025-08-14T21:24:12.9697914Z * [new tag] v1.12.1-rc5 -> v1.12.1-rc5 2025-08-14T21:24:12.9698025Z * [new tag] v1.13.0 -> v1.13.0 2025-08-14T21:24:12.9698127Z * [new tag] v1.13.0-rc1 -> v1.13.0-rc1 2025-08-14T21:24:12.9699746Z * [new tag] v1.13.0-rc2 -> v1.13.0-rc2 2025-08-14T21:24:12.9700042Z * [new tag] v1.13.0-rc3 -> v1.13.0-rc3 2025-08-14T21:24:12.9700169Z * [new tag] v1.13.0-rc4 -> v1.13.0-rc4 2025-08-14T21:24:12.9700295Z * [new tag] v1.13.0-rc5 -> v1.13.0-rc5 2025-08-14T21:24:12.9700742Z * [new tag] v1.13.0-rc6 -> v1.13.0-rc6 2025-08-14T21:24:12.9703460Z * [new tag] v1.13.1 -> v1.13.1 2025-08-14T21:24:12.9703758Z * [new tag] v1.13.1-rc1 -> v1.13.1-rc1 2025-08-14T21:24:12.9703878Z * [new tag] v1.2.0 -> v1.2.0 2025-08-14T21:24:12.9704065Z * [new tag] v1.2.0a0 -> v1.2.0a0 2025-08-14T21:24:12.9704243Z * [new tag] v1.3.0 -> v1.3.0 2025-08-14T21:24:12.9704351Z * [new tag] v1.3.0a0 -> v1.3.0a0 2025-08-14T21:24:12.9704476Z * [new tag] v1.3.1 -> v1.3.1 2025-08-14T21:24:12.9704973Z * [new tag] v1.4.0 -> v1.4.0 2025-08-14T21:24:12.9705670Z * [new tag] v1.4.0a0 -> v1.4.0a0 2025-08-14T21:24:12.9705833Z * [new tag] v1.4.1 -> v1.4.1 2025-08-14T21:24:12.9708877Z * [new tag] v1.5.0 -> v1.5.0 2025-08-14T21:24:12.9709175Z * [new tag] v1.5.0-rc1 -> v1.5.0-rc1 2025-08-14T21:24:12.9709298Z * [new tag] v1.5.0-rc2 -> v1.5.0-rc2 2025-08-14T21:24:12.9709394Z * [new tag] v1.5.0-rc3 -> v1.5.0-rc3 2025-08-14T21:24:12.9709498Z * [new tag] v1.5.0-rc4 -> v1.5.0-rc4 2025-08-14T21:24:12.9709721Z * [new tag] v1.5.0-rc5 -> v1.5.0-rc5 2025-08-14T21:24:12.9709836Z * [new tag] v1.5.1 -> v1.5.1 2025-08-14T21:24:12.9710336Z * [new tag] v1.5.1-rc1 -> v1.5.1-rc1 2025-08-14T21:24:12.9710658Z * [new tag] v1.6.0 -> v1.6.0 2025-08-14T21:24:12.9713682Z * [new tag] v1.6.0-rc1 -> v1.6.0-rc1 2025-08-14T21:24:12.9713816Z * [new tag] v1.6.0-rc2 -> v1.6.0-rc2 2025-08-14T21:24:12.9713912Z * [new tag] v1.6.0-rc3 -> v1.6.0-rc3 2025-08-14T21:24:12.9714011Z * [new tag] v1.6.0-rc4 -> v1.6.0-rc4 2025-08-14T21:24:12.9714102Z * [new tag] v1.6.0-rc5 -> v1.6.0-rc5 2025-08-14T21:24:12.9714206Z * [new tag] v1.6.0-rc6 -> v1.6.0-rc6 2025-08-14T21:24:12.9714447Z * [new tag] v1.6.0-rc7 -> v1.6.0-rc7 2025-08-14T21:24:12.9715944Z * [new tag] v1.7.0 -> v1.7.0 2025-08-14T21:24:12.9716243Z * [new tag] v1.7.0-rc1 -> v1.7.0-rc1 2025-08-14T21:24:12.9716376Z * [new tag] v1.7.0-rc2 -> v1.7.0-rc2 2025-08-14T21:24:12.9716921Z * [new tag] v1.7.0-rc3 -> v1.7.0-rc3 2025-08-14T21:24:12.9717041Z * [new tag] v1.7.0-rc4 -> v1.7.0-rc4 2025-08-14T21:24:12.9718031Z * [new tag] v1.7.1 -> v1.7.1 2025-08-14T21:24:12.9718317Z * [new tag] v1.7.1-rc1 -> v1.7.1-rc1 2025-08-14T21:24:12.9719225Z * [new tag] v1.7.1-rc2 -> v1.7.1-rc2 2025-08-14T21:24:12.9719484Z * [new tag] v1.7.1-rc3 -> v1.7.1-rc3 2025-08-14T21:24:12.9720021Z * [new tag] v1.8.0 -> v1.8.0 2025-08-14T21:24:12.9720335Z * [new tag] v1.8.0-rc1 -> v1.8.0-rc1 2025-08-14T21:24:12.9720990Z * [new tag] v1.8.0-rc2 -> v1.8.0-rc2 2025-08-14T21:24:12.9721454Z * [new tag] v1.8.0-rc3 -> v1.8.0-rc3 2025-08-14T21:24:12.9722078Z * [new tag] v1.8.0-rc4 -> v1.8.0-rc4 2025-08-14T21:24:12.9722429Z * [new tag] v1.8.0-rc5 -> v1.8.0-rc5 2025-08-14T21:24:12.9722852Z * [new tag] v1.8.1 -> v1.8.1 2025-08-14T21:24:12.9723734Z * [new tag] v1.8.1-rc1 -> v1.8.1-rc1 2025-08-14T21:24:12.9723957Z * [new tag] v1.8.1-rc2 -> v1.8.1-rc2 2025-08-14T21:24:12.9724300Z * [new tag] v1.8.1-rc3 -> v1.8.1-rc3 2025-08-14T21:24:12.9725756Z * [new tag] v1.8.2 -> v1.8.2 2025-08-14T21:24:12.9726062Z * [new tag] v1.8.2-rc1 -> v1.8.2-rc1 2025-08-14T21:24:12.9726197Z * [new tag] v1.9.0 -> v1.9.0 2025-08-14T21:24:12.9727152Z * [new tag] v1.9.0-rc1 -> v1.9.0-rc1 2025-08-14T21:24:12.9727485Z * [new tag] v1.9.0-rc2 -> v1.9.0-rc2 2025-08-14T21:24:12.9728395Z * [new tag] v1.9.0-rc3 -> v1.9.0-rc3 2025-08-14T21:24:12.9728491Z * [new tag] v1.9.0-rc4 -> v1.9.0-rc4 2025-08-14T21:24:12.9730651Z * [new tag] v1.9.1 -> v1.9.1 2025-08-14T21:24:12.9730789Z * [new tag] v1.9.1-rc1 -> v1.9.1-rc1 2025-08-14T21:24:12.9730900Z * [new tag] v1.9.1-rc2 -> v1.9.1-rc2 2025-08-14T21:24:12.9731320Z * [new tag] v2.0.0 -> v2.0.0 2025-08-14T21:24:12.9731763Z * [new tag] v2.0.0-rc1 -> v2.0.0-rc1 2025-08-14T21:24:12.9732535Z * [new tag] v2.0.0-rc2 -> v2.0.0-rc2 2025-08-14T21:24:12.9733198Z * [new tag] v2.0.0-rc3 -> v2.0.0-rc3 2025-08-14T21:24:12.9733480Z * [new tag] v2.0.0-rc4 -> v2.0.0-rc4 2025-08-14T21:24:12.9734739Z * [new tag] v2.0.0-rc5 -> v2.0.0-rc5 2025-08-14T21:24:12.9735052Z * [new tag] v2.0.0-rc6 -> v2.0.0-rc6 2025-08-14T21:24:12.9735228Z * [new tag] v2.0.1 -> v2.0.1 2025-08-14T21:24:12.9735926Z * [new tag] v2.0.1-rc1 -> v2.0.1-rc1 2025-08-14T21:24:12.9736040Z * [new tag] v2.0.1-rc2 -> v2.0.1-rc2 2025-08-14T21:24:12.9738742Z * [new tag] v2.0.1-rc3 -> v2.0.1-rc3 2025-08-14T21:24:12.9738871Z * [new tag] v2.0.1-rc4 -> v2.0.1-rc4 2025-08-14T21:24:12.9738977Z * [new tag] v2.1.0 -> v2.1.0 2025-08-14T21:24:12.9739084Z * [new tag] v2.1.0-rc1 -> v2.1.0-rc1 2025-08-14T21:24:12.9739554Z * [new tag] v2.1.0-rc2 -> v2.1.0-rc2 2025-08-14T21:24:12.9739962Z * [new tag] v2.1.0-rc3 -> v2.1.0-rc3 2025-08-14T21:24:12.9741187Z * [new tag] v2.1.0-rc4 -> v2.1.0-rc4 2025-08-14T21:24:12.9741422Z * [new tag] v2.1.0-rc5 -> v2.1.0-rc5 2025-08-14T21:24:12.9741597Z * [new tag] v2.1.0-rc6 -> v2.1.0-rc6 2025-08-14T21:24:12.9744073Z * [new tag] v2.1.1 -> v2.1.1 2025-08-14T21:24:12.9744236Z * [new tag] v2.1.1-rc1 -> v2.1.1-rc1 2025-08-14T21:24:12.9744341Z * [new tag] v2.1.1-rc2 -> v2.1.1-rc2 2025-08-14T21:24:12.9744558Z * [new tag] v2.1.1-rc3 -> v2.1.1-rc3 2025-08-14T21:24:12.9744751Z * [new tag] v2.1.1-rc4 -> v2.1.1-rc4 2025-08-14T21:24:12.9745127Z * [new tag] v2.1.1-rc5 -> v2.1.1-rc5 2025-08-14T21:24:12.9745569Z * [new tag] v2.1.1-rc6 -> v2.1.1-rc6 2025-08-14T21:24:12.9746195Z * [new tag] v2.1.2 -> v2.1.2 2025-08-14T21:24:12.9749117Z * [new tag] v2.1.2-rc1 -> v2.1.2-rc1 2025-08-14T21:24:12.9749379Z * [new tag] v2.1.2-rc2 -> v2.1.2-rc2 2025-08-14T21:24:12.9749695Z * [new tag] v2.1.2-rc3 -> v2.1.2-rc3 2025-08-14T21:24:12.9749921Z * [new tag] v2.2.0 -> v2.2.0 2025-08-14T21:24:12.9750035Z * [new tag] v2.2.0-rc1 -> v2.2.0-rc1 2025-08-14T21:24:12.9750133Z * [new tag] v2.2.0-rc2 -> v2.2.0-rc2 2025-08-14T21:24:12.9750223Z * [new tag] v2.2.0-rc3 -> v2.2.0-rc3 2025-08-14T21:24:12.9750568Z * [new tag] v2.2.0-rc4 -> v2.2.0-rc4 2025-08-14T21:24:12.9751076Z * [new tag] v2.2.0-rc5 -> v2.2.0-rc5 2025-08-14T21:24:12.9751871Z * [new tag] v2.2.0-rc6 -> v2.2.0-rc6 2025-08-14T21:24:12.9751974Z * [new tag] v2.2.0-rc7 -> v2.2.0-rc7 2025-08-14T21:24:12.9755203Z * [new tag] v2.2.0-rc8 -> v2.2.0-rc8 2025-08-14T21:24:12.9755347Z * [new tag] v2.2.1 -> v2.2.1 2025-08-14T21:24:12.9755448Z * [new tag] v2.2.1-rc1 -> v2.2.1-rc1 2025-08-14T21:24:12.9755545Z * [new tag] v2.2.1-rc2 -> v2.2.1-rc2 2025-08-14T21:24:12.9755636Z * [new tag] v2.2.1-rc3 -> v2.2.1-rc3 2025-08-14T21:24:12.9755725Z * [new tag] v2.2.2 -> v2.2.2 2025-08-14T21:24:12.9755851Z * [new tag] v2.2.2-rc1 -> v2.2.2-rc1 2025-08-14T21:24:12.9756110Z * [new tag] v2.2.2-rc2 -> v2.2.2-rc2 2025-08-14T21:24:12.9756287Z * [new tag] v2.2.2-rc3 -> v2.2.2-rc3 2025-08-14T21:24:12.9756714Z * [new tag] v2.3.0 -> v2.3.0 2025-08-14T21:24:12.9757522Z * [new tag] v2.3.0-rc1 -> v2.3.0-rc1 2025-08-14T21:24:12.9757830Z * [new tag] v2.3.0-rc10 -> v2.3.0-rc10 2025-08-14T21:24:12.9760215Z * [new tag] v2.3.0-rc11 -> v2.3.0-rc11 2025-08-14T21:24:12.9760344Z * [new tag] v2.3.0-rc12 -> v2.3.0-rc12 2025-08-14T21:24:12.9760446Z * [new tag] v2.3.0-rc2 -> v2.3.0-rc2 2025-08-14T21:24:12.9760541Z * [new tag] v2.3.0-rc3 -> v2.3.0-rc3 2025-08-14T21:24:12.9760648Z * [new tag] v2.3.0-rc4 -> v2.3.0-rc4 2025-08-14T21:24:12.9761641Z * [new tag] v2.3.0-rc5 -> v2.3.0-rc5 2025-08-14T21:24:12.9761769Z * [new tag] v2.3.0-rc6 -> v2.3.0-rc6 2025-08-14T21:24:12.9762155Z * [new tag] v2.3.0-rc7 -> v2.3.0-rc7 2025-08-14T21:24:12.9764032Z * [new tag] v2.3.0-rc8 -> v2.3.0-rc8 2025-08-14T21:24:12.9764166Z * [new tag] v2.3.0-rc9 -> v2.3.0-rc9 2025-08-14T21:24:12.9764267Z * [new tag] v2.3.1 -> v2.3.1 2025-08-14T21:24:12.9764363Z * [new tag] v2.3.1-rc1 -> v2.3.1-rc1 2025-08-14T21:24:12.9764845Z * [new tag] v2.3.1-rc2 -> v2.3.1-rc2 2025-08-14T21:24:12.9769095Z * [new tag] v2.3.1-rc3 -> v2.3.1-rc3 2025-08-14T21:24:12.9769366Z * [new tag] v2.4.0 -> v2.4.0 2025-08-14T21:24:12.9769498Z * [new tag] v2.4.0-rc1 -> v2.4.0-rc1 2025-08-14T21:24:12.9769675Z * [new tag] v2.4.0-rc2 -> v2.4.0-rc2 2025-08-14T21:24:12.9769778Z * [new tag] v2.4.0-rc3 -> v2.4.0-rc3 2025-08-14T21:24:12.9769902Z * [new tag] v2.4.0-rc4 -> v2.4.0-rc4 2025-08-14T21:24:12.9770007Z * [new tag] v2.4.0-rc5 -> v2.4.0-rc5 2025-08-14T21:24:12.9770097Z * [new tag] v2.4.0-rc6 -> v2.4.0-rc6 2025-08-14T21:24:12.9770387Z * [new tag] v2.4.0-rc7 -> v2.4.0-rc7 2025-08-14T21:24:12.9771523Z * [new tag] v2.4.0-rc8 -> v2.4.0-rc8 2025-08-14T21:24:12.9771806Z * [new tag] v2.4.0-rc9 -> v2.4.0-rc9 2025-08-14T21:24:12.9771924Z * [new tag] v2.4.1 -> v2.4.1 2025-08-14T21:24:12.9774286Z * [new tag] v2.4.1-rc1 -> v2.4.1-rc1 2025-08-14T21:24:12.9774721Z * [new tag] v2.4.1-rc2 -> v2.4.1-rc2 2025-08-14T21:24:12.9774842Z * [new tag] v2.4.1-rc3 -> v2.4.1-rc3 2025-08-14T21:24:12.9775017Z * [new tag] v2.5.0 -> v2.5.0 2025-08-14T21:24:12.9775226Z * [new tag] v2.5.0-rc1 -> v2.5.0-rc1 2025-08-14T21:24:12.9775327Z * [new tag] v2.5.0-rc10 -> v2.5.0-rc10 2025-08-14T21:24:12.9775754Z * [new tag] v2.5.0-rc2 -> v2.5.0-rc2 2025-08-14T21:24:12.9777310Z * [new tag] v2.5.0-rc3 -> v2.5.0-rc3 2025-08-14T21:24:12.9777612Z * [new tag] v2.5.0-rc4 -> v2.5.0-rc4 2025-08-14T21:24:12.9777733Z * [new tag] v2.5.0-rc5 -> v2.5.0-rc5 2025-08-14T21:24:12.9778177Z * [new tag] v2.5.0-rc6 -> v2.5.0-rc6 2025-08-14T21:24:12.9780546Z * [new tag] v2.5.0-rc7 -> v2.5.0-rc7 2025-08-14T21:24:12.9780813Z * [new tag] v2.5.0-rc8 -> v2.5.0-rc8 2025-08-14T21:24:12.9780931Z * [new tag] v2.5.0-rc9 -> v2.5.0-rc9 2025-08-14T21:24:12.9781028Z * [new tag] v2.5.1 -> v2.5.1 2025-08-14T21:24:12.9781237Z * [new tag] v2.5.1-rc1 -> v2.5.1-rc1 2025-08-14T21:24:12.9781347Z * [new tag] v2.6.0 -> v2.6.0 2025-08-14T21:24:12.9781721Z * [new tag] v2.6.0-rc1 -> v2.6.0-rc1 2025-08-14T21:24:12.9782979Z * [new tag] v2.6.0-rc2 -> v2.6.0-rc2 2025-08-14T21:24:12.9783195Z * [new tag] v2.6.0-rc3 -> v2.6.0-rc3 2025-08-14T21:24:12.9783408Z * [new tag] v2.6.0-rc4 -> v2.6.0-rc4 2025-08-14T21:24:12.9785676Z * [new tag] v2.6.0-rc5 -> v2.6.0-rc5 2025-08-14T21:24:12.9785984Z * [new tag] v2.6.0-rc6 -> v2.6.0-rc6 2025-08-14T21:24:12.9786142Z * [new tag] v2.6.0-rc7 -> v2.6.0-rc7 2025-08-14T21:24:12.9786254Z * [new tag] v2.6.0-rc8 -> v2.6.0-rc8 2025-08-14T21:24:12.9786755Z * [new tag] v2.6.0-rc9 -> v2.6.0-rc9 2025-08-14T21:24:12.9787770Z * [new tag] v2.7.0 -> v2.7.0 2025-08-14T21:24:12.9787885Z * [new tag] v2.7.0-rc1 -> v2.7.0-rc1 2025-08-14T21:24:12.9790568Z * [new tag] v2.7.0-rc10 -> v2.7.0-rc10 2025-08-14T21:24:12.9790824Z * [new tag] v2.7.0-rc2 -> v2.7.0-rc2 2025-08-14T21:24:12.9791044Z * [new tag] v2.7.0-rc3 -> v2.7.0-rc3 2025-08-14T21:24:12.9791151Z * [new tag] v2.7.0-rc4 -> v2.7.0-rc4 2025-08-14T21:24:12.9791384Z * [new tag] v2.7.0-rc5 -> v2.7.0-rc5 2025-08-14T21:24:12.9791483Z * [new tag] v2.7.0-rc6 -> v2.7.0-rc6 2025-08-14T21:24:12.9793038Z * [new tag] v2.7.0-rc7 -> v2.7.0-rc7 2025-08-14T21:24:12.9793160Z * [new tag] v2.7.0-rc8 -> v2.7.0-rc8 2025-08-14T21:24:12.9793256Z * [new tag] v2.7.0-rc9 -> v2.7.0-rc9 2025-08-14T21:24:12.9796392Z * [new tag] v2.7.1 -> v2.7.1 2025-08-14T21:24:12.9796670Z * [new tag] v2.7.1-rc1 -> v2.7.1-rc1 2025-08-14T21:24:12.9796789Z * [new tag] v2.7.1-rc2 -> v2.7.1-rc2 2025-08-14T21:24:12.9796886Z * [new tag] v2.7.1-rc3 -> v2.7.1-rc3 2025-08-14T21:24:12.9796986Z * [new tag] v2.7.1-rc4 -> v2.7.1-rc4 2025-08-14T21:24:12.9797364Z * [new tag] v2.7.1-rc5 -> v2.7.1-rc5 2025-08-14T21:24:12.9797474Z * [new tag] v2.8.0 -> v2.8.0 2025-08-14T21:24:12.9797687Z * [new tag] v2.8.0-rc1 -> v2.8.0-rc1 2025-08-14T21:24:12.9798608Z * [new tag] v2.8.0-rc2 -> v2.8.0-rc2 2025-08-14T21:24:12.9798787Z * [new tag] v2.8.0-rc3 -> v2.8.0-rc3 2025-08-14T21:24:12.9799760Z * [new tag] v2.8.0-rc4 -> v2.8.0-rc4 2025-08-14T21:24:12.9800337Z * [new tag] v2.8.0-rc5 -> v2.8.0-rc5 2025-08-14T21:24:12.9800908Z * [new tag] v2.8.0-rc6 -> v2.8.0-rc6 2025-08-14T21:24:12.9801246Z * [new tag] v2.8.0-rc7 -> v2.8.0-rc7 2025-08-14T21:24:12.9802086Z * [new tag] v2.8.0-rc8 -> v2.8.0-rc8 2025-08-14T21:24:12.9802947Z * [new tag] whc_flight_1 -> whc_flight_1 2025-08-14T21:24:12.9803063Z * [new tag] whc_flight_2 -> whc_flight_2 2025-08-14T21:24:12.9805110Z * [new tag] whc_flight_4 -> whc_flight_4 2025-08-14T21:24:13.0274306Z [command]/usr/bin/git rev-parse --verify --quiet 1fc683cf17c8c673044538d10266c00f92987be2^{object} 2025-08-14T21:24:13.0302163Z 1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:24:13.0306674Z ##[endgroup] 2025-08-14T21:24:13.0306914Z ##[group]Determining the checkout info 2025-08-14T21:24:13.0307088Z ##[endgroup] 2025-08-14T21:24:13.0310957Z [command]/usr/bin/git sparse-checkout disable 2025-08-14T21:24:13.0358887Z [command]/usr/bin/git config --local --unset-all extensions.worktreeConfig 2025-08-14T21:24:13.0386668Z ##[group]Checking out the ref 2025-08-14T21:24:13.0390597Z [command]/usr/bin/git checkout --progress --force 1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:24:14.0485029Z Note: switching to '1fc683cf17c8c673044538d10266c00f92987be2'. 2025-08-14T21:24:14.0485331Z 2025-08-14T21:24:14.0485494Z You are in 'detached HEAD' state. You can look around, make experimental 2025-08-14T21:24:14.0486133Z changes and commit them, and you can discard any commits you make in this 2025-08-14T21:24:14.0486491Z state without impacting any branches by switching back to a branch. 2025-08-14T21:24:14.0486699Z 2025-08-14T21:24:14.0486852Z If you want to create a new branch to retain commits you create, you may 2025-08-14T21:24:14.0487168Z do so (now or later) by using -c with the switch command. Example: 2025-08-14T21:24:14.0487337Z 2025-08-14T21:24:14.0487428Z git switch -c 2025-08-14T21:24:14.0487563Z 2025-08-14T21:24:14.0487643Z Or undo this operation with: 2025-08-14T21:24:14.0487762Z 2025-08-14T21:24:14.0487834Z git switch - 2025-08-14T21:24:14.0487928Z 2025-08-14T21:24:14.0488104Z Turn off this advice by setting config variable advice.detachedHead to false 2025-08-14T21:24:14.0488367Z 2025-08-14T21:24:14.0488650Z HEAD is now at 1fc683cf17c [Inductor] Allow indexing a flexible layout for extract_input_node_reduction_ranges (#160645) 2025-08-14T21:24:14.0536736Z ##[endgroup] 2025-08-14T21:24:14.0537111Z ##[group]Setting up auth for fetching submodules 2025-08-14T21:24:14.0544752Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic *** 2025-08-14T21:24:14.0605017Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf 2025-08-14T21:24:14.0631096Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com: 2025-08-14T21:24:14.0660800Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com: 2025-08-14T21:24:14.0683572Z ##[endgroup] 2025-08-14T21:24:14.0683906Z ##[group]Fetching submodules 2025-08-14T21:24:14.0686975Z [command]/usr/bin/git submodule sync --recursive 2025-08-14T21:24:14.1017319Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive 2025-08-14T21:24:14.1345249Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni' 2025-08-14T21:24:14.1667358Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16' 2025-08-14T21:24:14.1669973Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv' 2025-08-14T21:24:14.1670578Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK' 2025-08-14T21:24:14.1671174Z Submodule 'third_party/NVTX' (https://github.com/NVIDIA/NVTX.git) registered for path 'third_party/NVTX' 2025-08-14T21:24:14.1671954Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator' 2025-08-14T21:24:14.1687846Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK' 2025-08-14T21:24:14.1691671Z Submodule 'third_party/aiter' (https://github.com/ROCm/aiter.git) registered for path 'third_party/aiter' 2025-08-14T21:24:14.1692252Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark' 2025-08-14T21:24:14.1693553Z Submodule 'third_party/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/composable_kernel' 2025-08-14T21:24:14.1698770Z Submodule 'third_party/cpp-httplib' (https://github.com/yhirose/cpp-httplib.git) registered for path 'third_party/cpp-httplib' 2025-08-14T21:24:14.1699432Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo' 2025-08-14T21:24:14.1719115Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend' 2025-08-14T21:24:14.1720920Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass' 2025-08-14T21:24:14.1721452Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm' 2025-08-14T21:24:14.1726251Z Submodule 'third_party/flash-attention' (https://github.com/Dao-AILab/flash-attention.git) registered for path 'third_party/flash-attention' 2025-08-14T21:24:14.1726949Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers' 2025-08-14T21:24:14.1742408Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt' 2025-08-14T21:24:14.1747838Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp' 2025-08-14T21:24:14.1752708Z Submodule 'third_party/gloo' (https://github.com/pytorch/gloo) registered for path 'third_party/gloo' 2025-08-14T21:24:14.1755008Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest' 2025-08-14T21:24:14.1755746Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep' 2025-08-14T21:24:14.1760845Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi' 2025-08-14T21:24:14.1772483Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto' 2025-08-14T21:24:14.1773161Z Submodule 'third_party/kleidiai' (https://github.com/ARM-software/kleidiai.git) registered for path 'third_party/kleidiai' 2025-08-14T21:24:14.1777411Z Submodule 'third_party/mimalloc' (https://github.com/microsoft/mimalloc.git) registered for path 'third_party/mimalloc' 2025-08-14T21:24:14.1782765Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann' 2025-08-14T21:24:14.1787558Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx' 2025-08-14T21:24:14.1801381Z Submodule 'third_party/opentelemetry-cpp' (https://github.com/open-telemetry/opentelemetry-cpp.git) registered for path 'third_party/opentelemetry-cpp' 2025-08-14T21:24:14.1802554Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft' 2025-08-14T21:24:14.1805971Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf' 2025-08-14T21:24:14.1809301Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd' 2025-08-14T21:24:14.1812456Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool' 2025-08-14T21:24:14.1816316Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11' 2025-08-14T21:24:14.1836389Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy' 2025-08-14T21:24:14.1837050Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef' 2025-08-14T21:24:14.1839646Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe' 2025-08-14T21:24:14.1876140Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'... 2025-08-14T21:24:14.4385909Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'... 2025-08-14T21:24:14.4386423Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'... 2025-08-14T21:24:14.4386873Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'... 2025-08-14T21:24:14.4412672Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'... 2025-08-14T21:24:15.6420933Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'... 2025-08-14T21:24:15.6421726Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'... 2025-08-14T21:24:15.6424022Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'... 2025-08-14T21:24:15.6425011Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'... 2025-08-14T21:24:15.6425879Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'... 2025-08-14T21:24:15.6426788Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'... 2025-08-14T21:24:15.6427636Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'... 2025-08-14T21:24:15.6428598Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kleidiai'... 2025-08-14T21:24:15.6429484Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'... 2025-08-14T21:24:15.6430277Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NVTX'... 2025-08-14T21:24:15.6431321Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'... 2025-08-14T21:24:15.6432279Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention'... 2025-08-14T21:24:15.6433388Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpp-httplib'... 2025-08-14T21:24:15.6434314Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'... 2025-08-14T21:24:15.6435203Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'... 2025-08-14T21:24:15.6436043Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/mimalloc'... 2025-08-14T21:24:15.6437062Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'... 2025-08-14T21:24:15.6438629Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'... 2025-08-14T21:24:15.7422962Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'... 2025-08-14T21:24:15.9665832Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'... 2025-08-14T21:24:15.9666343Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'... 2025-08-14T21:24:15.9666777Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'... 2025-08-14T21:24:16.0307624Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'... 2025-08-14T21:24:27.5724031Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'... 2025-08-14T21:24:27.5725404Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'... 2025-08-14T21:24:27.5726007Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'... 2025-08-14T21:24:27.5726440Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'... 2025-08-14T21:24:27.5726912Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/composable_kernel'... 2025-08-14T21:24:27.5727366Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter'... 2025-08-14T21:24:27.5727827Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp'... 2025-08-14T21:24:27.5728290Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'... 2025-08-14T21:24:27.5728726Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'... 2025-08-14T21:24:27.5855727Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f' 2025-08-14T21:24:27.5973150Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3' 2025-08-14T21:24:27.6056487Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1' 2025-08-14T21:24:27.6259344Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73' 2025-08-14T21:24:27.6917271Z Submodule path 'third_party/NVTX': checked out '2942f167cc30c5e3a44a2aecd5b0d9c07ff61a07' 2025-08-14T21:24:27.7378303Z Submodule path 'third_party/VulkanMemoryAllocator': checked out '1d8f600fd424278486eade7ed3e877c99f0846b1' 2025-08-14T21:24:28.2573736Z Submodule path 'third_party/XNNPACK': checked out '51a0103656eff6fc9bfd39a4597923c4b542c883' 2025-08-14T21:24:28.3761681Z Submodule path 'third_party/aiter': checked out '01aae101b9e5e94d6c16a9514c9fb8df99c93150' 2025-08-14T21:24:28.3779996Z Submodule '3rdparty/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T21:24:28.3807138Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/aiter/3rdparty/composable_kernel'... 2025-08-14T21:24:31.7651389Z Submodule path 'third_party/aiter/3rdparty/composable_kernel': checked out 'cffe8fa2a442ac8e80dd236a1a5d24fe3d7e0cbf' 2025-08-14T21:24:31.7862269Z Submodule path 'third_party/benchmark': checked out '299e5928955cc62af9968370293b916f5130916f' 2025-08-14T21:24:32.0286699Z Submodule path 'third_party/composable_kernel': checked out '7fe50dc3da2069d6645d9deb8c017a876472a977' 2025-08-14T21:24:32.0735797Z Submodule path 'third_party/cpp-httplib': checked out '3af7f2c16147f3fbc6e4d717032daf505dc1652c' 2025-08-14T21:24:32.1616432Z Submodule path 'third_party/cpuinfo': checked out '5e3d2445e6a84d9599bee2bf78edbb4d80865e1d' 2025-08-14T21:24:32.2020367Z Submodule path 'third_party/cudnn_frontend': checked out 'f937055efc6d414d11f4c6577e3977fe74f35fb6' 2025-08-14T21:24:32.7423750Z Submodule path 'third_party/cutlass': checked out 'e51efbfe18fe4f4cbb66ab814c55bf4aa0185491' 2025-08-14T21:24:32.8588386Z Submodule path 'third_party/fbgemm': checked out '21c7d30c526c0f1ad873ecc632dca6cfa8a69067' 2025-08-14T21:24:32.8607708Z Submodule 'external/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/external/asmjit' 2025-08-14T21:24:32.8613268Z Submodule 'external/composable_kernel' (https://github.com/jwfromm/composable_kernel.git) registered for path 'third_party/fbgemm/external/composable_kernel' 2025-08-14T21:24:32.8618080Z Submodule 'external/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/external/cpuinfo' 2025-08-14T21:24:32.8623472Z Submodule 'external/cutlass' (https://github.com/jwfromm/cutlass) registered for path 'third_party/fbgemm/external/cutlass' 2025-08-14T21:24:32.8628685Z Submodule 'external/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/external/googletest' 2025-08-14T21:24:32.8629517Z Submodule 'external/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/external/hipify_torch' 2025-08-14T21:24:32.8630244Z Submodule 'external/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/fbgemm/external/json' 2025-08-14T21:24:32.8646350Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/asmjit'... 2025-08-14T21:24:34.0875565Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/hipify_torch'... 2025-08-14T21:24:34.0878729Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cpuinfo'... 2025-08-14T21:24:34.0879315Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/googletest'... 2025-08-14T21:24:34.0879947Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/composable_kernel'... 2025-08-14T21:24:34.1876349Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/cutlass'... 2025-08-14T21:24:35.1771401Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/external/json'... 2025-08-14T21:24:39.2907427Z Submodule path 'third_party/fbgemm/external/asmjit': checked out 'a3199e8857792cd10b7589ff5d58343d2c9008ea' 2025-08-14T21:24:39.4783158Z Submodule path 'third_party/fbgemm/external/composable_kernel': checked out 'b1281b8b08d973a7064f864f47eeb30f3e2596e9' 2025-08-14T21:24:39.5637222Z Submodule path 'third_party/fbgemm/external/cpuinfo': checked out '6543fec09b2f04ac4a666882998b534afc9c1349' 2025-08-14T21:24:40.0557686Z Submodule path 'third_party/fbgemm/external/cutlass': checked out 'b40777404c174b9694a870bff5c13ce6b7f656ad' 2025-08-14T21:24:40.0959038Z Submodule path 'third_party/fbgemm/external/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-08-14T21:24:40.1071157Z Submodule path 'third_party/fbgemm/external/hipify_torch': checked out 'a4337c69fe0e2552a7b7b0669178926beeed828c' 2025-08-14T21:24:40.1924319Z Submodule path 'third_party/fbgemm/external/json': checked out '9cca280a4d0ccf0c08f47a99aa71d1b0e52f8d03' 2025-08-14T21:24:40.2500690Z Submodule path 'third_party/flash-attention': checked out '979702c87a8713a8e0a5e9fee122b90d2ef13be5' 2025-08-14T21:24:40.2522277Z Submodule 'csrc/composable_kernel' (https://github.com/ROCm/composable_kernel.git) registered for path 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T21:24:40.2525197Z Submodule 'csrc/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/flash-attention/csrc/cutlass' 2025-08-14T21:24:40.2552837Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/composable_kernel'... 2025-08-14T21:24:43.3797105Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flash-attention/csrc/cutlass'... 2025-08-14T21:24:43.5604670Z Submodule path 'third_party/flash-attention/csrc/composable_kernel': checked out '888317e698e9803c62bd38568abc9e05d7709f33' 2025-08-14T21:24:44.0193567Z Submodule path 'third_party/flash-attention/csrc/cutlass': checked out 'c506e16788cb08416a4a57e11a9067beeee29420' 2025-08-14T21:24:44.1247423Z Submodule path 'third_party/flatbuffers': checked out 'a2cd1ea3b6d3fee220106b5fed3f7ce8da9eb757' 2025-08-14T21:24:44.1533551Z Submodule path 'third_party/fmt': checked out '40626af88bd7df9a5fb80be7b25ac85b122d6c21' 2025-08-14T21:24:44.1863960Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350' 2025-08-14T21:24:44.2069188Z Submodule path 'third_party/gloo': checked out 'c7b7b022c124d9643957d9bd55f57ac59fce8fa2' 2025-08-14T21:24:44.2489800Z Submodule path 'third_party/googletest': checked out '52eb8108c5bdec04579160ae17225d66034bd723' 2025-08-14T21:24:44.2610492Z Submodule path 'third_party/ideep': checked out '719d8e6cd7f7a0e01b155657526d693acf97c2b3' 2025-08-14T21:24:44.2628345Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn' 2025-08-14T21:24:44.2659289Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'... 2025-08-14T21:24:55.3592415Z Submodule path 'third_party/ideep/mkl-dnn': checked out '8d263e693366ef8db40acc569cc7d8edf644556d' 2025-08-14T21:24:55.3761713Z Submodule path 'third_party/ittapi': checked out 'dec1d23ca65ab069d225dfe40dea14f455170959' 2025-08-14T21:24:55.4580478Z Submodule path 'third_party/kineto': checked out '5e7501833f1021ce6f618572d3baf657b6319658' 2025-08-14T21:24:55.4597192Z Submodule 'libkineto/third_party/dynolog' (https://github.com/facebookincubator/dynolog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T21:24:55.4597879Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T21:24:55.4602268Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T21:24:55.4626263Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog'... 2025-08-14T21:24:56.1002329Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'... 2025-08-14T21:24:56.7358568Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'... 2025-08-14T21:24:56.8064428Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog': checked out '7d04a0053a845370ae06ce317a22a48e9edcc74e' 2025-08-14T21:24:56.8083593Z Submodule 'third_party/DCGM' (https://github.com/NVIDIA/DCGM.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T21:24:56.8084390Z Submodule 'third_party/cpr' (https://github.com/libcpr/cpr.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T21:24:56.8085157Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T21:24:56.8086138Z Submodule 'third_party/gflags' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T21:24:56.8086911Z Submodule 'third_party/glog' (https://github.com/google/glog.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T21:24:56.8087704Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T21:24:56.8094515Z Submodule 'third_party/json' (https://github.com/nlohmann/json.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T21:24:56.8096929Z Submodule 'third_party/pfs' (https://github.com/dtrugman/pfs.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T21:24:56.8120351Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM'... 2025-08-14T21:24:58.0923546Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/pfs'... 2025-08-14T21:24:58.0924248Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags'... 2025-08-14T21:24:58.0924891Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/cpr'... 2025-08-14T21:24:58.0925794Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/glog'... 2025-08-14T21:24:58.0926586Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/googletest'... 2025-08-14T21:24:58.0927218Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/fmt'... 2025-08-14T21:24:58.1925352Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/json'... 2025-08-14T21:25:03.9403199Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM': checked out 'ffde4e54bc7249a6039a5e6b45b395141e1217f9' 2025-08-14T21:25:03.9555843Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr': checked out '871ed52d350214a034f6ef8a3b8f51c5ce1bd400' 2025-08-14T21:25:03.9865013Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt': checked out 'cd4af11efc9c622896a3e4cb599fa28668ca3d05' 2025-08-14T21:25:03.9986421Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags': checked out 'e171aa2d15ed9eb17054558e0b3a6a413bb01067' 2025-08-14T21:25:03.9996100Z Submodule 'doc' (https://github.com/gflags/gflags.git) registered for path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T21:25:04.0020409Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc'... 2025-08-14T21:25:04.6258673Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc': checked out '8411df715cf522606e3b1aca386ddfc0b63d34b4' 2025-08-14T21:25:04.6420058Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog': checked out 'b33e3bad4c46c8a6345525fd822af355e5ef9446' 2025-08-14T21:25:04.6773418Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest': checked out '58d77fa8070e8cec2dc1ed015d66b454c8d78850' 2025-08-14T21:25:04.7618160Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/json': checked out '4f8fba14066156b73f1189a2b8bd568bde5284c5' 2025-08-14T21:25:04.7771063Z Submodule path 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs': checked out 'f68a2fa8ea36c783bdd760371411fcb495aa3150' 2025-08-14T21:25:04.8072590Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '0041a40c1350ba702d475b9c4ad62da77caea164' 2025-08-14T21:25:04.8564721Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347' 2025-08-14T21:25:04.8929789Z Submodule path 'third_party/kleidiai': checked out 'cca02c2f69dd18e1f12647c1c0bdc8cf90e680c7' 2025-08-14T21:25:04.9261204Z Submodule path 'third_party/mimalloc': checked out 'fbd8b99c2b828428947d70fdc046bb55609be93e' 2025-08-14T21:25:05.0176403Z Submodule path 'third_party/nlohmann': checked out '55f93686c01528224f448c19128836e7df245f72' 2025-08-14T21:25:05.2930123Z Submodule path 'third_party/onnx': checked out 'e709452ef2bbc1d113faf678c24e6d3467696e83' 2025-08-14T21:25:05.2961554Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11' 2025-08-14T21:25:05.2987000Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'... 2025-08-14T21:25:07.2300422Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2025-08-14T21:25:07.2841191Z Submodule path 'third_party/opentelemetry-cpp': checked out 'a799f4aed9c94b765dcdaabaeab7d5e7e2310878' 2025-08-14T21:25:07.2861260Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark) registered for path 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T21:25:07.2864300Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T21:25:07.2865023Z Submodule 'third_party/ms-gsl' (https://github.com/microsoft/GSL) registered for path 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T21:25:07.2865649Z Submodule 'third_party/nlohmann-json' (https://github.com/nlohmann/json) registered for path 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T21:25:07.2866404Z Submodule 'third_party/opentelemetry-proto' (https://github.com/open-telemetry/opentelemetry-proto) registered for path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T21:25:07.2867226Z Submodule 'third_party/opentracing-cpp' (https://github.com/opentracing/opentracing-cpp.git) registered for path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T21:25:07.2869027Z Submodule 'third_party/prometheus-cpp' (https://github.com/jupp0r/prometheus-cpp) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T21:25:07.2869671Z Submodule 'tools/vcpkg' (https://github.com/Microsoft/vcpkg) registered for path 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T21:25:07.2896091Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/benchmark'... 2025-08-14T21:25:07.7351778Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentracing-cpp'... 2025-08-14T21:25:07.7353393Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/opentelemetry-proto'... 2025-08-14T21:25:07.7354119Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp'... 2025-08-14T21:25:07.7354746Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/ms-gsl'... 2025-08-14T21:25:07.8354302Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/googletest'... 2025-08-14T21:25:08.3314616Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/nlohmann-json'... 2025-08-14T21:25:15.3772586Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/tools/vcpkg'... 2025-08-14T21:25:15.8775178Z Submodule path 'third_party/opentelemetry-cpp/third_party/benchmark': checked out 'd572f4777349d43653b21d6c2fc63020ab326db2' 2025-08-14T21:25:15.9120128Z Submodule path 'third_party/opentelemetry-cpp/third_party/googletest': checked out 'b796f7d44681514f58a683a3a71ff17c94edb0c1' 2025-08-14T21:25:15.9274296Z Submodule path 'third_party/opentelemetry-cpp/third_party/ms-gsl': checked out '6f4529395c5b7c2d661812257cd6780c67e54afa' 2025-08-14T21:25:16.0145297Z Submodule path 'third_party/opentelemetry-cpp/third_party/nlohmann-json': checked out 'bc889afb4c5bf1c0d8ee29ef35eaaf4c8bef8a5d' 2025-08-14T21:25:16.0275225Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto': checked out '4ca4f0335c63cda7ab31ea7ed70d6553aee14dce' 2025-08-14T21:25:16.0402982Z Submodule path 'third_party/opentelemetry-cpp/third_party/opentracing-cpp': checked out '06b57f48ded1fa3bdd3d4346f6ef29e40e08eaf5' 2025-08-14T21:25:16.0535573Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp': checked out 'c9ffcdda9086ffd9e1283ea7a0276d831f3c8a8d' 2025-08-14T21:25:16.0553191Z Submodule 'civetweb' (https://github.com/civetweb/civetweb.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T21:25:16.0557813Z Submodule 'googletest' (https://github.com/google/googletest.git) registered for path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T21:25:16.0580264Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb'... 2025-08-14T21:25:17.9082322Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest'... 2025-08-14T21:25:18.1261637Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb': checked out 'eefb26f82b233268fc98577d265352720d477ba4' 2025-08-14T21:25:18.1652145Z Submodule path 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929' 2025-08-14T21:25:18.4863882Z Submodule path 'third_party/opentelemetry-cpp/tools/vcpkg': checked out '8eb57355a4ffb410a2e94c07b4dca2dffbee8e50' 2025-08-14T21:25:18.4977519Z Submodule path 'third_party/pocketfft': checked out '0fa0ef591e38c2758e3184c6c23e497b9f732ffa' 2025-08-14T21:25:18.7121781Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a' 2025-08-14T21:25:18.7139620Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark' 2025-08-14T21:25:18.7140473Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest' 2025-08-14T21:25:18.7168830Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'... 2025-08-14T21:25:19.2494751Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'... 2025-08-14T21:25:19.6442801Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8' 2025-08-14T21:25:19.7084514Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081' 2025-08-14T21:25:19.7174509Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900' 2025-08-14T21:25:19.7285009Z Submodule path 'third_party/pthreadpool': checked out '4fe0e1e183925bf8cfa6aae24237e724a96479b8' 2025-08-14T21:25:19.7598028Z Submodule path 'third_party/pybind11': checked out 'a2e59f0e7065404b44dfe92a28aca47ba1378dc4' 2025-08-14T21:25:19.7853713Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67' 2025-08-14T21:25:19.8227980Z Submodule path 'third_party/sleef': checked out '5a1d179df9cf652951b59010a2d2075372d67f68' 2025-08-14T21:25:19.8457974Z Submodule path 'third_party/tensorpipe': checked out 'dacda0567d9f23d4bc503e1c4f84aa65f33ac38a' 2025-08-14T21:25:19.8472100Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest' 2025-08-14T21:25:19.8472925Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop' 2025-08-14T21:25:19.8477756Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv' 2025-08-14T21:25:19.8478580Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T21:25:19.8506926Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'... 2025-08-14T21:25:20.7762721Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'... 2025-08-14T21:25:20.7763365Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'... 2025-08-14T21:25:20.8765204Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'... 2025-08-14T21:25:21.0601282Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e' 2025-08-14T21:25:21.0749045Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281' 2025-08-14T21:25:21.1383070Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '5152db2cbfeb5582e9c27c5ea1dba2cd9e10759b' 2025-08-14T21:25:21.1645090Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef' 2025-08-14T21:25:21.1663957Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T21:25:21.1686562Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'... 2025-08-14T21:25:21.3653293Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5' 2025-08-14T21:25:21.3692812Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0 2025-08-14T21:25:21.4015458Z Entering 'android/libs/fbjni' 2025-08-14T21:25:21.4059580Z Entering 'third_party/FP16' 2025-08-14T21:25:21.4098512Z Entering 'third_party/FXdiv' 2025-08-14T21:25:21.4138525Z Entering 'third_party/NNPACK' 2025-08-14T21:25:21.4181124Z Entering 'third_party/NVTX' 2025-08-14T21:25:21.4224235Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-14T21:25:21.4262212Z Entering 'third_party/XNNPACK' 2025-08-14T21:25:21.4318382Z Entering 'third_party/aiter' 2025-08-14T21:25:21.4353661Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T21:25:21.4403899Z Entering 'third_party/benchmark' 2025-08-14T21:25:21.4442417Z Entering 'third_party/composable_kernel' 2025-08-14T21:25:21.4490536Z Entering 'third_party/cpp-httplib' 2025-08-14T21:25:21.4529777Z Entering 'third_party/cpuinfo' 2025-08-14T21:25:21.4574786Z Entering 'third_party/cudnn_frontend' 2025-08-14T21:25:21.4620271Z Entering 'third_party/cutlass' 2025-08-14T21:25:21.4665530Z Entering 'third_party/fbgemm' 2025-08-14T21:25:21.4705608Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-14T21:25:21.4746938Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-14T21:25:21.4792691Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-14T21:25:21.4837311Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-14T21:25:21.4886493Z Entering 'third_party/fbgemm/external/googletest' 2025-08-14T21:25:21.4928776Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-14T21:25:21.4970055Z Entering 'third_party/fbgemm/external/json' 2025-08-14T21:25:21.5013840Z Entering 'third_party/flash-attention' 2025-08-14T21:25:21.5054338Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T21:25:21.5098964Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-14T21:25:21.5152309Z Entering 'third_party/flatbuffers' 2025-08-14T21:25:21.5195275Z Entering 'third_party/fmt' 2025-08-14T21:25:21.5231760Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-14T21:25:21.5275512Z Entering 'third_party/gloo' 2025-08-14T21:25:21.5320986Z Entering 'third_party/googletest' 2025-08-14T21:25:21.5360543Z Entering 'third_party/ideep' 2025-08-14T21:25:21.5393985Z Entering 'third_party/ideep/mkl-dnn' 2025-08-14T21:25:21.5440726Z Entering 'third_party/ittapi' 2025-08-14T21:25:21.5490847Z Entering 'third_party/kineto' 2025-08-14T21:25:21.5522037Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T21:25:21.5563757Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T21:25:21.5610089Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T21:25:21.5644356Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T21:25:21.5684059Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T21:25:21.5727257Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T21:25:21.5772275Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T21:25:21.5815292Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T21:25:21.5858304Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T21:25:21.5897246Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T21:25:21.5934646Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T21:25:21.5981019Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T21:25:21.6024074Z Entering 'third_party/kleidiai' 2025-08-14T21:25:21.6066931Z Entering 'third_party/mimalloc' 2025-08-14T21:25:21.6102566Z Entering 'third_party/nlohmann' 2025-08-14T21:25:21.6145323Z Entering 'third_party/onnx' 2025-08-14T21:25:21.6197511Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-14T21:25:21.6240305Z Entering 'third_party/opentelemetry-cpp' 2025-08-14T21:25:21.6282824Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T21:25:21.6318776Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T21:25:21.6361924Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T21:25:21.6400709Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T21:25:21.6438004Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T21:25:21.6481510Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T21:25:21.6518642Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T21:25:21.6567748Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T21:25:21.6608323Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T21:25:21.6648316Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T21:25:21.6702922Z Entering 'third_party/pocketfft' 2025-08-14T21:25:21.6740752Z Entering 'third_party/protobuf' 2025-08-14T21:25:21.6784860Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-14T21:25:21.6824193Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-14T21:25:21.6872551Z Entering 'third_party/psimd' 2025-08-14T21:25:21.6914520Z Entering 'third_party/pthreadpool' 2025-08-14T21:25:21.6957849Z Entering 'third_party/pybind11' 2025-08-14T21:25:21.6999713Z Entering 'third_party/python-peachpy' 2025-08-14T21:25:21.7037563Z Entering 'third_party/sleef' 2025-08-14T21:25:21.7078221Z Entering 'third_party/tensorpipe' 2025-08-14T21:25:21.7116408Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-14T21:25:21.7158091Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-14T21:25:21.7199753Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-14T21:25:21.7242811Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T21:25:21.7289739Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T21:25:21.7343779Z ##[endgroup] 2025-08-14T21:25:21.7344134Z ##[group]Persisting credentials for submodules 2025-08-14T21:25:21.7354983Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :" 2025-08-14T21:25:21.7672848Z Entering 'android/libs/fbjni' 2025-08-14T21:25:21.7725778Z Entering 'third_party/FP16' 2025-08-14T21:25:21.7784524Z Entering 'third_party/FXdiv' 2025-08-14T21:25:21.7843013Z Entering 'third_party/NNPACK' 2025-08-14T21:25:21.7901708Z Entering 'third_party/NVTX' 2025-08-14T21:25:21.7956973Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-14T21:25:21.8007116Z Entering 'third_party/XNNPACK' 2025-08-14T21:25:21.8076844Z Entering 'third_party/aiter' 2025-08-14T21:25:21.8128778Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T21:25:21.8193809Z Entering 'third_party/benchmark' 2025-08-14T21:25:21.8251907Z Entering 'third_party/composable_kernel' 2025-08-14T21:25:21.8310970Z Entering 'third_party/cpp-httplib' 2025-08-14T21:25:21.8363470Z Entering 'third_party/cpuinfo' 2025-08-14T21:25:21.8421874Z Entering 'third_party/cudnn_frontend' 2025-08-14T21:25:21.8480406Z Entering 'third_party/cutlass' 2025-08-14T21:25:21.8553178Z Entering 'third_party/fbgemm' 2025-08-14T21:25:21.8602634Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-14T21:25:21.8661666Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-14T21:25:21.8720816Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-14T21:25:21.8776871Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-14T21:25:21.8839066Z Entering 'third_party/fbgemm/external/googletest' 2025-08-14T21:25:21.8896107Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-14T21:25:21.8948312Z Entering 'third_party/fbgemm/external/json' 2025-08-14T21:25:21.9005976Z Entering 'third_party/flash-attention' 2025-08-14T21:25:21.9061246Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T21:25:21.9116337Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-14T21:25:21.9173290Z Entering 'third_party/flatbuffers' 2025-08-14T21:25:21.9229602Z Entering 'third_party/fmt' 2025-08-14T21:25:21.9284562Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-14T21:25:21.9337822Z Entering 'third_party/gloo' 2025-08-14T21:25:21.9390017Z Entering 'third_party/googletest' 2025-08-14T21:25:21.9443546Z Entering 'third_party/ideep' 2025-08-14T21:25:21.9498328Z Entering 'third_party/ideep/mkl-dnn' 2025-08-14T21:25:21.9560540Z Entering 'third_party/ittapi' 2025-08-14T21:25:21.9624578Z Entering 'third_party/kineto' 2025-08-14T21:25:21.9681841Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T21:25:21.9733519Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T21:25:21.9798485Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T21:25:21.9850223Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T21:25:21.9905401Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T21:25:21.9962039Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T21:25:22.0016281Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T21:25:22.0072840Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T21:25:22.0126204Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T21:25:22.0184473Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T21:25:22.0238598Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T21:25:22.0290187Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T21:25:22.0352338Z Entering 'third_party/kleidiai' 2025-08-14T21:25:22.0401916Z Entering 'third_party/mimalloc' 2025-08-14T21:25:22.0459108Z Entering 'third_party/nlohmann' 2025-08-14T21:25:22.0515262Z Entering 'third_party/onnx' 2025-08-14T21:25:22.0581368Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-14T21:25:22.0636943Z Entering 'third_party/opentelemetry-cpp' 2025-08-14T21:25:22.0697947Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T21:25:22.0754741Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T21:25:22.0811311Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T21:25:22.0859710Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T21:25:22.0917069Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T21:25:22.0973785Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T21:25:22.1027607Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T21:25:22.1084650Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T21:25:22.1136217Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T21:25:22.1199273Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T21:25:22.1273601Z Entering 'third_party/pocketfft' 2025-08-14T21:25:22.1331056Z Entering 'third_party/protobuf' 2025-08-14T21:25:22.1392373Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-14T21:25:22.1447259Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-14T21:25:22.1505739Z Entering 'third_party/psimd' 2025-08-14T21:25:22.1565242Z Entering 'third_party/pthreadpool' 2025-08-14T21:25:22.1616546Z Entering 'third_party/pybind11' 2025-08-14T21:25:22.1673190Z Entering 'third_party/python-peachpy' 2025-08-14T21:25:22.1729967Z Entering 'third_party/sleef' 2025-08-14T21:25:22.1786759Z Entering 'third_party/tensorpipe' 2025-08-14T21:25:22.1839704Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-14T21:25:22.1892947Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-14T21:25:22.1946923Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-14T21:25:22.2000679Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T21:25:22.2052985Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T21:25:22.2135052Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url" 2025-08-14T21:25:22.2449365Z Entering 'android/libs/fbjni' 2025-08-14T21:25:22.2496928Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config remote.origin.url 2025-08-14T21:25:22.2510739Z Entering 'third_party/FP16' 2025-08-14T21:25:22.2560801Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config remote.origin.url 2025-08-14T21:25:22.2575863Z Entering 'third_party/FXdiv' 2025-08-14T21:25:22.2623193Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config remote.origin.url 2025-08-14T21:25:22.2637422Z Entering 'third_party/NNPACK' 2025-08-14T21:25:22.2690001Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config remote.origin.url 2025-08-14T21:25:22.2709592Z Entering 'third_party/NVTX' 2025-08-14T21:25:22.2757346Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NVTX/config remote.origin.url 2025-08-14T21:25:22.2779413Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-14T21:25:22.2828595Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config remote.origin.url 2025-08-14T21:25:22.2839977Z Entering 'third_party/XNNPACK' 2025-08-14T21:25:22.2888033Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config remote.origin.url 2025-08-14T21:25:22.2913942Z Entering 'third_party/aiter' 2025-08-14T21:25:22.2963392Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/config remote.origin.url 2025-08-14T21:25:22.2981558Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T21:25:22.3025586Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/aiter/modules/3rdparty/composable_kernel/config remote.origin.url 2025-08-14T21:25:22.3051554Z Entering 'third_party/benchmark' 2025-08-14T21:25:22.3092128Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config remote.origin.url 2025-08-14T21:25:22.3108869Z Entering 'third_party/composable_kernel' 2025-08-14T21:25:22.3154107Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/composable_kernel/config remote.origin.url 2025-08-14T21:25:22.3179328Z Entering 'third_party/cpp-httplib' 2025-08-14T21:25:22.3223633Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpp-httplib/config remote.origin.url 2025-08-14T21:25:22.3235075Z Entering 'third_party/cpuinfo' 2025-08-14T21:25:22.3284532Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config remote.origin.url 2025-08-14T21:25:22.3301565Z Entering 'third_party/cudnn_frontend' 2025-08-14T21:25:22.3348705Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config remote.origin.url 2025-08-14T21:25:22.3366286Z Entering 'third_party/cutlass' 2025-08-14T21:25:22.3415340Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config remote.origin.url 2025-08-14T21:25:22.3435743Z Entering 'third_party/fbgemm' 2025-08-14T21:25:22.3487091Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config remote.origin.url 2025-08-14T21:25:22.3500677Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-14T21:25:22.3550181Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/asmjit/config remote.origin.url 2025-08-14T21:25:22.3565207Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-14T21:25:22.3617431Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/composable_kernel/config remote.origin.url 2025-08-14T21:25:22.3641586Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-14T21:25:22.3692428Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cpuinfo/config remote.origin.url 2025-08-14T21:25:22.3704743Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-14T21:25:22.3752573Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/cutlass/config remote.origin.url 2025-08-14T21:25:22.3779765Z Entering 'third_party/fbgemm/external/googletest' 2025-08-14T21:25:22.3829122Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/googletest/config remote.origin.url 2025-08-14T21:25:22.3850755Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-14T21:25:22.3891746Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/hipify_torch/config remote.origin.url 2025-08-14T21:25:22.3908201Z Entering 'third_party/fbgemm/external/json' 2025-08-14T21:25:22.3955703Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/external/json/config remote.origin.url 2025-08-14T21:25:22.3977092Z Entering 'third_party/flash-attention' 2025-08-14T21:25:22.4028723Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/config remote.origin.url 2025-08-14T21:25:22.4045418Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T21:25:22.4090144Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/composable_kernel/config remote.origin.url 2025-08-14T21:25:22.4110867Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-14T21:25:22.4158375Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flash-attention/modules/csrc/cutlass/config remote.origin.url 2025-08-14T21:25:22.4181417Z Entering 'third_party/flatbuffers' 2025-08-14T21:25:22.4231976Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config remote.origin.url 2025-08-14T21:25:22.4251901Z Entering 'third_party/fmt' 2025-08-14T21:25:22.4301725Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config remote.origin.url 2025-08-14T21:25:22.4320186Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-14T21:25:22.4372519Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config remote.origin.url 2025-08-14T21:25:22.4385812Z Entering 'third_party/gloo' 2025-08-14T21:25:22.4439572Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config remote.origin.url 2025-08-14T21:25:22.4456717Z Entering 'third_party/googletest' 2025-08-14T21:25:22.4499921Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config remote.origin.url 2025-08-14T21:25:22.4514781Z Entering 'third_party/ideep' 2025-08-14T21:25:22.4562642Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config remote.origin.url 2025-08-14T21:25:22.4581594Z Entering 'third_party/ideep/mkl-dnn' 2025-08-14T21:25:22.4628164Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config remote.origin.url 2025-08-14T21:25:22.4654545Z Entering 'third_party/ittapi' 2025-08-14T21:25:22.4706720Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config remote.origin.url 2025-08-14T21:25:22.4719711Z Entering 'third_party/kineto' 2025-08-14T21:25:22.4774265Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config remote.origin.url 2025-08-14T21:25:22.4793685Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T21:25:22.4840611Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/config remote.origin.url 2025-08-14T21:25:22.4856671Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T21:25:22.4902262Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/DCGM/config remote.origin.url 2025-08-14T21:25:22.4921617Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T21:25:22.4977116Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/cpr/config remote.origin.url 2025-08-14T21:25:22.4993268Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T21:25:22.5038815Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/fmt/config remote.origin.url 2025-08-14T21:25:22.5058014Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T21:25:22.5114430Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/config remote.origin.url 2025-08-14T21:25:22.5116452Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T21:25:22.5164133Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/gflags/modules/doc/config remote.origin.url 2025-08-14T21:25:22.5183928Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T21:25:22.5231292Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/glog/config remote.origin.url 2025-08-14T21:25:22.5246587Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T21:25:22.5293487Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/googletest/config remote.origin.url 2025-08-14T21:25:22.5311226Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T21:25:22.5361266Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/json/config remote.origin.url 2025-08-14T21:25:22.5378124Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T21:25:22.5427643Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/dynolog/modules/third_party/pfs/config remote.origin.url 2025-08-14T21:25:22.5443999Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T21:25:22.5492827Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config remote.origin.url 2025-08-14T21:25:22.5513175Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T21:25:22.5560038Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config remote.origin.url 2025-08-14T21:25:22.5584114Z Entering 'third_party/kleidiai' 2025-08-14T21:25:22.5633333Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kleidiai/config remote.origin.url 2025-08-14T21:25:22.5654177Z Entering 'third_party/mimalloc' 2025-08-14T21:25:22.5700390Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/mimalloc/config remote.origin.url 2025-08-14T21:25:22.5721626Z Entering 'third_party/nlohmann' 2025-08-14T21:25:22.5766174Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config remote.origin.url 2025-08-14T21:25:22.5783560Z Entering 'third_party/onnx' 2025-08-14T21:25:22.5831034Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config remote.origin.url 2025-08-14T21:25:22.5859865Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-14T21:25:22.5912507Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config remote.origin.url 2025-08-14T21:25:22.5930951Z Entering 'third_party/opentelemetry-cpp' 2025-08-14T21:25:22.5980202Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/config remote.origin.url 2025-08-14T21:25:22.5999291Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T21:25:22.6045053Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/benchmark/config remote.origin.url 2025-08-14T21:25:22.6063481Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T21:25:22.6111545Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/googletest/config remote.origin.url 2025-08-14T21:25:22.6129104Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T21:25:22.6175853Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/ms-gsl/config remote.origin.url 2025-08-14T21:25:22.6191656Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T21:25:22.6236002Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/nlohmann-json/config remote.origin.url 2025-08-14T21:25:22.6252580Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T21:25:22.6301112Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentelemetry-proto/config remote.origin.url 2025-08-14T21:25:22.6316733Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T21:25:22.6366727Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/opentracing-cpp/config remote.origin.url 2025-08-14T21:25:22.6384610Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T21:25:22.6432456Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/config remote.origin.url 2025-08-14T21:25:22.6447715Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T21:25:22.6499876Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/civetweb/config remote.origin.url 2025-08-14T21:25:22.6515707Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T21:25:22.6563786Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/third_party/prometheus-cpp/modules/googletest/config remote.origin.url 2025-08-14T21:25:22.6586879Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T21:25:22.6635665Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/opentelemetry-cpp/modules/tools/vcpkg/config remote.origin.url 2025-08-14T21:25:22.6673662Z Entering 'third_party/pocketfft' 2025-08-14T21:25:22.6720706Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config remote.origin.url 2025-08-14T21:25:22.6742013Z Entering 'third_party/protobuf' 2025-08-14T21:25:22.6791461Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config remote.origin.url 2025-08-14T21:25:22.6807122Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-14T21:25:22.6854687Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config remote.origin.url 2025-08-14T21:25:22.6868088Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-14T21:25:22.6918211Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config remote.origin.url 2025-08-14T21:25:22.6935063Z Entering 'third_party/psimd' 2025-08-14T21:25:22.6987410Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config remote.origin.url 2025-08-14T21:25:22.7002453Z Entering 'third_party/pthreadpool' 2025-08-14T21:25:22.7059231Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config remote.origin.url 2025-08-14T21:25:22.7075935Z Entering 'third_party/pybind11' 2025-08-14T21:25:22.7121211Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config remote.origin.url 2025-08-14T21:25:22.7141096Z Entering 'third_party/python-peachpy' 2025-08-14T21:25:22.7191450Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config remote.origin.url 2025-08-14T21:25:22.7205706Z Entering 'third_party/sleef' 2025-08-14T21:25:22.7257943Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config remote.origin.url 2025-08-14T21:25:22.7269819Z Entering 'third_party/tensorpipe' 2025-08-14T21:25:22.7318669Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config remote.origin.url 2025-08-14T21:25:22.7335919Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-14T21:25:22.7385814Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config remote.origin.url 2025-08-14T21:25:22.7403349Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-14T21:25:22.7449932Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config remote.origin.url 2025-08-14T21:25:22.7466294Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-14T21:25:22.7513314Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config remote.origin.url 2025-08-14T21:25:22.7525053Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T21:25:22.7575213Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config remote.origin.url 2025-08-14T21:25:22.7590248Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T21:25:22.7640951Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config remote.origin.url 2025-08-14T21:25:22.8929617Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:' 2025-08-14T21:25:22.9242638Z Entering 'android/libs/fbjni' 2025-08-14T21:25:22.9286306Z Entering 'third_party/FP16' 2025-08-14T21:25:22.9321801Z Entering 'third_party/FXdiv' 2025-08-14T21:25:22.9366837Z Entering 'third_party/NNPACK' 2025-08-14T21:25:22.9404446Z Entering 'third_party/NVTX' 2025-08-14T21:25:22.9450789Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-14T21:25:22.9490526Z Entering 'third_party/XNNPACK' 2025-08-14T21:25:22.9534669Z Entering 'third_party/aiter' 2025-08-14T21:25:22.9579651Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T21:25:22.9623553Z Entering 'third_party/benchmark' 2025-08-14T21:25:22.9670272Z Entering 'third_party/composable_kernel' 2025-08-14T21:25:22.9715635Z Entering 'third_party/cpp-httplib' 2025-08-14T21:25:22.9754339Z Entering 'third_party/cpuinfo' 2025-08-14T21:25:22.9796210Z Entering 'third_party/cudnn_frontend' 2025-08-14T21:25:22.9830823Z Entering 'third_party/cutlass' 2025-08-14T21:25:22.9886633Z Entering 'third_party/fbgemm' 2025-08-14T21:25:22.9923506Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-14T21:25:22.9966139Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-14T21:25:23.0014873Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-14T21:25:23.0051907Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-14T21:25:23.0096981Z Entering 'third_party/fbgemm/external/googletest' 2025-08-14T21:25:23.0133424Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-14T21:25:23.0176773Z Entering 'third_party/fbgemm/external/json' 2025-08-14T21:25:23.0217745Z Entering 'third_party/flash-attention' 2025-08-14T21:25:23.0257813Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T21:25:23.0305082Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-14T21:25:23.0357054Z Entering 'third_party/flatbuffers' 2025-08-14T21:25:23.0397333Z Entering 'third_party/fmt' 2025-08-14T21:25:23.0434004Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-14T21:25:23.0475992Z Entering 'third_party/gloo' 2025-08-14T21:25:23.0517518Z Entering 'third_party/googletest' 2025-08-14T21:25:23.0558117Z Entering 'third_party/ideep' 2025-08-14T21:25:23.0596626Z Entering 'third_party/ideep/mkl-dnn' 2025-08-14T21:25:23.0640478Z Entering 'third_party/ittapi' 2025-08-14T21:25:23.0681587Z Entering 'third_party/kineto' 2025-08-14T21:25:23.0725072Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T21:25:23.0770824Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T21:25:23.0807175Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T21:25:23.0845547Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T21:25:23.0885703Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T21:25:23.0924582Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T21:25:23.0977576Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T21:25:23.1019145Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T21:25:23.1069666Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T21:25:23.1115322Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T21:25:23.1157392Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T21:25:23.1198748Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T21:25:23.1240503Z Entering 'third_party/kleidiai' 2025-08-14T21:25:23.1285244Z Entering 'third_party/mimalloc' 2025-08-14T21:25:23.1325082Z Entering 'third_party/nlohmann' 2025-08-14T21:25:23.1372794Z Entering 'third_party/onnx' 2025-08-14T21:25:23.1421781Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-14T21:25:23.1468154Z Entering 'third_party/opentelemetry-cpp' 2025-08-14T21:25:23.1512998Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T21:25:23.1552388Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T21:25:23.1590530Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T21:25:23.1626564Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T21:25:23.1673170Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T21:25:23.1713994Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T21:25:23.1754455Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T21:25:23.1794733Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T21:25:23.1837281Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T21:25:23.1884120Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T21:25:23.1942508Z Entering 'third_party/pocketfft' 2025-08-14T21:25:23.1984946Z Entering 'third_party/protobuf' 2025-08-14T21:25:23.2025571Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-14T21:25:23.2067496Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-14T21:25:23.2116477Z Entering 'third_party/psimd' 2025-08-14T21:25:23.2159777Z Entering 'third_party/pthreadpool' 2025-08-14T21:25:23.2201792Z Entering 'third_party/pybind11' 2025-08-14T21:25:23.2237935Z Entering 'third_party/python-peachpy' 2025-08-14T21:25:23.2283044Z Entering 'third_party/sleef' 2025-08-14T21:25:23.2325638Z Entering 'third_party/tensorpipe' 2025-08-14T21:25:23.2370646Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-14T21:25:23.2404095Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-14T21:25:23.2444147Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-14T21:25:23.2484290Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T21:25:23.2524289Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T21:25:23.2591014Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:' 2025-08-14T21:25:23.2922864Z Entering 'android/libs/fbjni' 2025-08-14T21:25:23.2966611Z Entering 'third_party/FP16' 2025-08-14T21:25:23.3012890Z Entering 'third_party/FXdiv' 2025-08-14T21:25:23.3052120Z Entering 'third_party/NNPACK' 2025-08-14T21:25:23.3094462Z Entering 'third_party/NVTX' 2025-08-14T21:25:23.3144501Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-14T21:25:23.3183079Z Entering 'third_party/XNNPACK' 2025-08-14T21:25:23.3228635Z Entering 'third_party/aiter' 2025-08-14T21:25:23.3274606Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T21:25:23.3322674Z Entering 'third_party/benchmark' 2025-08-14T21:25:23.3366236Z Entering 'third_party/composable_kernel' 2025-08-14T21:25:23.3415278Z Entering 'third_party/cpp-httplib' 2025-08-14T21:25:23.3462306Z Entering 'third_party/cpuinfo' 2025-08-14T21:25:23.3504087Z Entering 'third_party/cudnn_frontend' 2025-08-14T21:25:23.3545701Z Entering 'third_party/cutlass' 2025-08-14T21:25:23.3593014Z Entering 'third_party/fbgemm' 2025-08-14T21:25:23.3634815Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-14T21:25:23.3674122Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-14T21:25:23.3723252Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-14T21:25:23.3766061Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-14T21:25:23.3811301Z Entering 'third_party/fbgemm/external/googletest' 2025-08-14T21:25:23.3845297Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-14T21:25:23.3883453Z Entering 'third_party/fbgemm/external/json' 2025-08-14T21:25:23.3926237Z Entering 'third_party/flash-attention' 2025-08-14T21:25:23.3976117Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T21:25:23.4016514Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-14T21:25:23.4065935Z Entering 'third_party/flatbuffers' 2025-08-14T21:25:23.4108990Z Entering 'third_party/fmt' 2025-08-14T21:25:23.4150376Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-14T21:25:23.4189355Z Entering 'third_party/gloo' 2025-08-14T21:25:23.4229770Z Entering 'third_party/googletest' 2025-08-14T21:25:23.4277135Z Entering 'third_party/ideep' 2025-08-14T21:25:23.4316271Z Entering 'third_party/ideep/mkl-dnn' 2025-08-14T21:25:23.4362672Z Entering 'third_party/ittapi' 2025-08-14T21:25:23.4404134Z Entering 'third_party/kineto' 2025-08-14T21:25:23.4443064Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T21:25:23.4479057Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T21:25:23.4521004Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T21:25:23.4562986Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T21:25:23.4601242Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T21:25:23.4636741Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T21:25:23.4682617Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T21:25:23.4722089Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T21:25:23.4762719Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T21:25:23.4810236Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T21:25:23.4846437Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T21:25:23.4884574Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T21:25:23.4928663Z Entering 'third_party/kleidiai' 2025-08-14T21:25:23.4972468Z Entering 'third_party/mimalloc' 2025-08-14T21:25:23.5014527Z Entering 'third_party/nlohmann' 2025-08-14T21:25:23.5060572Z Entering 'third_party/onnx' 2025-08-14T21:25:23.5107354Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-14T21:25:23.5154001Z Entering 'third_party/opentelemetry-cpp' 2025-08-14T21:25:23.5197512Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T21:25:23.5234277Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T21:25:23.5277681Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T21:25:23.5320636Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T21:25:23.5365001Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T21:25:23.5404859Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T21:25:23.5446412Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T21:25:23.5486596Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T21:25:23.5526292Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T21:25:23.5574687Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T21:25:23.5628377Z Entering 'third_party/pocketfft' 2025-08-14T21:25:23.5673875Z Entering 'third_party/protobuf' 2025-08-14T21:25:23.5743625Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-14T21:25:23.5776286Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-14T21:25:23.5822439Z Entering 'third_party/psimd' 2025-08-14T21:25:23.5862124Z Entering 'third_party/pthreadpool' 2025-08-14T21:25:23.5905340Z Entering 'third_party/pybind11' 2025-08-14T21:25:23.5950358Z Entering 'third_party/python-peachpy' 2025-08-14T21:25:23.5990514Z Entering 'third_party/sleef' 2025-08-14T21:25:23.6029973Z Entering 'third_party/tensorpipe' 2025-08-14T21:25:23.6073371Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-14T21:25:23.6115990Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-14T21:25:23.6155692Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-14T21:25:23.6196448Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T21:25:23.6236876Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T21:25:23.6309705Z ##[endgroup] 2025-08-14T21:25:23.6347184Z [command]/usr/bin/git log -1 --format=%H 2025-08-14T21:25:23.6376630Z 1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:25:23.6524570Z Prepare all required actions 2025-08-14T21:25:23.6525093Z Getting action download info 2025-08-14T21:25:23.7899700Z ##[group]Run ./.github/actions/setup-linux 2025-08-14T21:25:23.7899921Z env: 2025-08-14T21:25:23.7900075Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:25:23.7900240Z ##[endgroup] 2025-08-14T21:25:23.7931507Z ##[group]Run set -euo pipefail 2025-08-14T21:25:23.7931759Z set -euo pipefail 2025-08-14T21:25:23.7931964Z function get_ec2_metadata() { 2025-08-14T21:25:23.7932223Z  # Pulled from instance metadata endpoint for EC2 2025-08-14T21:25:23.7932640Z  # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html 2025-08-14T21:25:23.7933006Z  category=$1 2025-08-14T21:25:23.7933246Z  # If it is GCP runner (runner name contains gcp), do not run this 2025-08-14T21:25:23.7933513Z  runner_name_str=i-0851ccaad4f014969 2025-08-14T21:25:23.7933772Z  if [[ -f /.inarc ]]; then 2025-08-14T21:25:23.7933986Z  echo "ARC Runner, no info on ec2 metadata" 2025-08-14T21:25:23.7934242Z  elif [[ $runner_name_str == *"gcp"* ]]; then 2025-08-14T21:25:23.7934524Z  echo "Runner is from Google Cloud Platform, No info on ec2 metadata" 2025-08-14T21:25:23.7934770Z  else 2025-08-14T21:25:23.7935271Z  curl -H "X-aws-ec2-metadata-token: $(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 30")" -fsSL "http://169.254.169.254/latest/meta-data/${category}" 2025-08-14T21:25:23.7935759Z  fi 2025-08-14T21:25:23.7935908Z } 2025-08-14T21:25:23.7936074Z echo "ami-id: $(get_ec2_metadata ami-id)" 2025-08-14T21:25:23.7936331Z echo "instance-id: $(get_ec2_metadata instance-id)" 2025-08-14T21:25:23.7936612Z echo "instance-type: $(get_ec2_metadata instance-type)" 2025-08-14T21:25:23.7936858Z echo "system info $(uname -a)" 2025-08-14T21:25:23.7943344Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:25:23.7943589Z env: 2025-08-14T21:25:23.7943761Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:25:23.7943925Z ##[endgroup] 2025-08-14T21:25:23.8086276Z ami-id: ami-05ffe3c48a9991133 2025-08-14T21:25:23.8182327Z instance-id: i-0851ccaad4f014969 2025-08-14T21:25:23.8274671Z instance-type: m7i-flex.8xlarge 2025-08-14T21:25:23.8286906Z system info Linux ip-10-0-8-108.ec2.internal 6.1.141-155.222.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Jun 17 10:29:47 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux 2025-08-14T21:25:23.8307134Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-08-14T21:25:23.8307712Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-08-14T21:25:23.8312979Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:25:23.8313243Z env: 2025-08-14T21:25:23.8313526Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:25:23.8313700Z ##[endgroup] 2025-08-14T21:25:23.8366143Z ##[group]Run if systemctl is-active --quiet docker; then 2025-08-14T21:25:23.8366469Z if systemctl is-active --quiet docker; then 2025-08-14T21:25:23.8366758Z  echo "Docker daemon is running..."; 2025-08-14T21:25:23.8366989Z else 2025-08-14T21:25:23.8367239Z  echo "Starting docker daemon..." && sudo systemctl start docker; 2025-08-14T21:25:23.8367530Z fi 2025-08-14T21:25:23.8371765Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:25:23.8371999Z env: 2025-08-14T21:25:23.8372154Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:25:23.8372323Z ##[endgroup] 2025-08-14T21:25:23.8449943Z Docker daemon is running... 2025-08-14T21:25:23.8476885Z ##[group]Run nick-fields/retry@v3.0.0 2025-08-14T21:25:23.8477075Z with: 2025-08-14T21:25:23.8477218Z shell: bash 2025-08-14T21:25:23.8478094Z timeout_minutes: 5 2025-08-14T21:25:23.8478281Z max_attempts: 3 2025-08-14T21:25:23.8478441Z retry_wait_seconds: 30 2025-08-14T21:25:23.8479730Z command: AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\") aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" # For LF Runners we need to make sure we also login to Meta's ECR docker registry too. META_AWS_ACCOUNT_ID=308535385114 if [ "$AWS_ACCOUNT_ID" != "$META_AWS_ACCOUNT_ID" ] ; then aws ecr get-login-password --region "$AWS_DEFAULT_REGION" | docker login --username AWS \ --password-stdin "$META_AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com" fi 2025-08-14T21:25:23.8480930Z polling_interval_seconds: 1 2025-08-14T21:25:23.8481110Z warning_on_retry: true 2025-08-14T21:25:23.8481275Z continue_on_error: false 2025-08-14T21:25:23.8481433Z env: 2025-08-14T21:25:23.8481579Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:25:23.8481747Z AWS_RETRY_MODE: standard 2025-08-14T21:25:23.8481908Z AWS_MAX_ATTEMPTS: 5 2025-08-14T21:25:23.8482077Z AWS_DEFAULT_REGION: us-east-1 2025-08-14T21:25:23.8482251Z ##[endgroup] 2025-08-14T21:25:24.8188194Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-08-14T21:25:24.8188806Z Configure a credential helper to remove this warning. See 2025-08-14T21:25:24.8189319Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-08-14T21:25:24.8189564Z 2025-08-14T21:25:24.8189637Z Login Succeeded 2025-08-14T21:25:24.9160899Z Command completed after 1 attempt(s). 2025-08-14T21:25:24.9215720Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-08-14T21:25:24.9216092Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-08-14T21:25:24.9216396Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-08-14T21:25:24.9222558Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:25:24.9222804Z env: 2025-08-14T21:25:24.9222964Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:25:24.9223140Z ##[endgroup] 2025-08-14T21:25:24.9300528Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-08-14T21:25:24.9300869Z # ignore expansion of "docker ps -q" since it could be empty 2025-08-14T21:25:24.9301127Z # shellcheck disable=SC2046 2025-08-14T21:25:24.9301338Z docker stop $(docker ps -q) || true 2025-08-14T21:25:24.9301544Z # Prune all of the docker images 2025-08-14T21:25:24.9301747Z docker system prune -af 2025-08-14T21:25:24.9305951Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:25:24.9306180Z env: 2025-08-14T21:25:24.9306332Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:25:24.9306497Z ##[endgroup] 2025-08-14T21:25:24.9829534Z "docker stop" requires at least 1 argument. 2025-08-14T21:25:24.9829882Z See 'docker stop --help'. 2025-08-14T21:25:24.9830050Z 2025-08-14T21:25:24.9830181Z Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] 2025-08-14T21:25:24.9830729Z 2025-08-14T21:25:24.9830809Z Stop one or more running containers 2025-08-14T21:25:25.0109048Z Total reclaimed space: 0B 2025-08-14T21:25:25.0143394Z ##[group]Run set +e 2025-08-14T21:25:25.0143621Z set +e 2025-08-14T21:25:25.0143783Z set -x 2025-08-14T21:25:25.0143930Z  2025-08-14T21:25:25.0144093Z PT_DOMAIN=download.pytorch.org 2025-08-14T21:25:25.0144460Z # TODO: Flaky access to download.pytorch.org https://github.com/pytorch/pytorch/issues/100400, 2025-08-14T21:25:25.0144884Z # cleaning this up once the issue is fixed. There are more than one resolved IP here, the last 2025-08-14T21:25:25.0145189Z # one is returned at random 2025-08-14T21:25:25.0145432Z RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" | tail -n1) 2025-08-14T21:25:25.0145664Z  2025-08-14T21:25:25.0145985Z if [ -z "${RESOLVED_IP}" ]; then 2025-08-14T21:25:25.0146247Z  echo "Couldn't resolve ${PT_DOMAIN}, retrying with Google DNS..." 2025-08-14T21:25:25.0146572Z  RESOLVED_IP=$(dig -4 +short "${PT_DOMAIN}" @8.8.8.8 | tail -n1) 2025-08-14T21:25:25.0146809Z  2025-08-14T21:25:25.0146970Z  if [ -z "${RESOLVED_IP}" ]; then 2025-08-14T21:25:25.0147203Z  echo "Couldn't resolve ${PT_DOMAIN}, exiting..." 2025-08-14T21:25:25.0147431Z  exit 1 2025-08-14T21:25:25.0147586Z  fi 2025-08-14T21:25:25.0147725Z fi 2025-08-14T21:25:25.0147866Z  2025-08-14T21:25:25.0148036Z if grep -r "${PT_DOMAIN}" /etc/hosts; then 2025-08-14T21:25:25.0148254Z  # Clean up any old records first 2025-08-14T21:25:25.0148481Z  sudo sed -i "/${PT_DOMAIN}/d" /etc/hosts 2025-08-14T21:25:25.0148685Z fi 2025-08-14T21:25:25.0148826Z  2025-08-14T21:25:25.0149025Z echo "${RESOLVED_IP} ${PT_DOMAIN}" | sudo tee -a /etc/hosts 2025-08-14T21:25:25.0149274Z cat /etc/hosts 2025-08-14T21:25:25.0153955Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:25:25.0154184Z env: 2025-08-14T21:25:25.0154347Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:25:25.0154533Z ##[endgroup] 2025-08-14T21:25:25.0175232Z + PT_DOMAIN=download.pytorch.org 2025-08-14T21:25:25.0185947Z ++ dig -4 +short download.pytorch.org 2025-08-14T21:25:25.0186405Z ++ tail -n1 2025-08-14T21:25:25.1117342Z + RESOLVED_IP=18.160.10.28 2025-08-14T21:25:25.1117617Z + '[' -z 18.160.10.28 ']' 2025-08-14T21:25:25.1117870Z + grep -r download.pytorch.org /etc/hosts 2025-08-14T21:25:25.1133966Z + echo '18.160.10.28 download.pytorch.org' 2025-08-14T21:25:25.1134858Z + sudo tee -a /etc/hosts 2025-08-14T21:25:25.3884900Z 18.160.10.28 download.pytorch.org 2025-08-14T21:25:25.3926092Z + cat /etc/hosts 2025-08-14T21:25:25.3951935Z 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 2025-08-14T21:25:25.3961018Z ::1 localhost6 localhost6.localdomain6 2025-08-14T21:25:25.3961337Z 18.160.10.28 download.pytorch.org 2025-08-14T21:25:25.4066130Z ##[group]Run pytorch/test-infra/.github/actions/calculate-docker-image@main 2025-08-14T21:25:25.4066539Z with: 2025-08-14T21:25:25.4067106Z docker-image-name: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:25:25.4067667Z use-custom-docker-registry: true 2025-08-14T21:25:25.4067883Z docker-build-dir: .ci/docker 2025-08-14T21:25:25.4068086Z docker-build-script: ./build.sh 2025-08-14T21:25:25.4068277Z working-directory: . 2025-08-14T21:25:25.4068513Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:25:25.4068765Z force-push: false 2025-08-14T21:25:25.4068916Z env: 2025-08-14T21:25:25.4069071Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:25:25.4069248Z ##[endgroup] 2025-08-14T21:25:25.4082940Z ##[group]Run set -ex 2025-08-14T21:25:25.4083178Z set -ex 2025-08-14T21:25:25.4083462Z  2025-08-14T21:25:25.4083766Z # If the docker build directory or the build script doesn't exist, the action will 2025-08-14T21:25:25.4084187Z # gracefully return the docker image name as it is. Pulling docker image in Linux 2025-08-14T21:25:25.4084543Z # job could then download the pre-built image as usual 2025-08-14T21:25:25.4084983Z if [[ -d "${DOCKER_BUILD_DIR}" ]] && [[ -f "${DOCKER_BUILD_DIR}/${DOCKER_BUILD_SCRIPT}" ]] && [[ "${USE_CUSTOM_DOCKER_REGISTRY}" == "true" ]]; then 2025-08-14T21:25:25.4085388Z  echo "skip=false" >> "${GITHUB_OUTPUT}" 2025-08-14T21:25:25.4085784Z else 2025-08-14T21:25:25.4085978Z  echo "skip=true" >> "${GITHUB_OUTPUT}" 2025-08-14T21:25:25.4086278Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-08-14T21:25:25.4086553Z  2025-08-14T21:25:25.4086923Z  echo "Not using custom ECR registry. Either it was not requested or there is no Docker build script in the ${REPO_NAME} repo..." 2025-08-14T21:25:25.4087337Z  exit 0 2025-08-14T21:25:25.4087506Z fi 2025-08-14T21:25:25.4087658Z  2025-08-14T21:25:25.4087903Z if [[ "${DOCKER_IMAGE_NAME}" == *"${DOCKER_REGISTRY}/${REPO_NAME}"* ]]; then 2025-08-14T21:25:25.4088279Z  # The docker image name already includes the ECR prefix and tag, so we can just 2025-08-14T21:25:25.4088600Z  # use it as it is, but first let's extract the tag 2025-08-14T21:25:25.4088903Z  DOCKER_TAG=$(echo "${DOCKER_IMAGE_NAME}" | awk -F '[:,]' '{print $2}') 2025-08-14T21:25:25.4089233Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-08-14T21:25:25.4089546Z  echo "docker-image=${DOCKER_IMAGE_NAME}" >> "${GITHUB_OUTPUT}" 2025-08-14T21:25:25.4089807Z else 2025-08-14T21:25:25.4089995Z  if [[ "${DOCKER_IMAGE_NAME}" == *:* ]]; then 2025-08-14T21:25:25.4090265Z  CUSTOM_TAG_PREFIX=${DOCKER_IMAGE_NAME#*:} 2025-08-14T21:25:25.4090540Z  DOCKER_IMAGE_NAME=${DOCKER_IMAGE_NAME%%:*} 2025-08-14T21:25:25.4090764Z  fi 2025-08-14T21:25:25.4091073Z  DOCKER_TAG=${CUSTOM_TAG_PREFIX:+${CUSTOM_TAG_PREFIX}-}$(git rev-parse HEAD:"${DOCKER_BUILD_DIR}") 2025-08-14T21:25:25.4091476Z  echo "docker-tag=${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-08-14T21:25:25.4091876Z  echo "docker-image=${DOCKER_REGISTRY}/${REPO_NAME}/${DOCKER_IMAGE_NAME}:${DOCKER_TAG}" >> "${GITHUB_OUTPUT}" 2025-08-14T21:25:25.4092302Z  echo "custom-tag-prefix=${CUSTOM_TAG_PREFIX}" >> "${GITHUB_OUTPUT}" 2025-08-14T21:25:25.4092580Z fi 2025-08-14T21:25:25.4100195Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:25:25.4100456Z env: 2025-08-14T21:25:25.4100635Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:25:25.4100839Z REPO_NAME: pytorch 2025-08-14T21:25:25.4101554Z DOCKER_IMAGE_NAME: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:25:25.4102183Z DOCKER_BUILD_DIR: .ci/docker 2025-08-14T21:25:25.4102402Z DOCKER_BUILD_SCRIPT: ./build.sh 2025-08-14T21:25:25.4102679Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:25:25.4102954Z USE_CUSTOM_DOCKER_REGISTRY: true 2025-08-14T21:25:25.4103171Z CUSTOM_TAG_PREFIX: 2025-08-14T21:25:25.4103356Z ##[endgroup] 2025-08-14T21:25:25.4130162Z + [[ -d .ci/docker ]] 2025-08-14T21:25:25.4130387Z + [[ -f .ci/docker/./build.sh ]] 2025-08-14T21:25:25.4130600Z + [[ true == \t\r\u\e ]] 2025-08-14T21:25:25.4130780Z + echo skip=false 2025-08-14T21:25:25.4131512Z + [[ 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe == *\3\0\8\5\3\5\3\8\5\1\1\4\.\d\k\r\.\e\c\r\.\u\s\-\e\a\s\t\-\1\.\a\m\a\z\o\n\a\w\s\.\c\o\m\/\p\y\t\o\r\c\h* ]] 2025-08-14T21:25:25.4138545Z ++ echo 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:25:25.4139184Z ++ awk -F '[:,]' '{print $2}' 2025-08-14T21:25:25.4176337Z + DOCKER_TAG=pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:25:25.4177179Z + echo docker-tag=pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:25:25.4178022Z + echo docker-image=308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:25:25.4195250Z ##[group]Run set +e 2025-08-14T21:25:25.4195465Z set +e 2025-08-14T21:25:25.4195617Z set -x 2025-08-14T21:25:25.4195755Z  2025-08-14T21:25:25.4195899Z login() { 2025-08-14T21:25:25.4196190Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-08-14T21:25:25.4196517Z } 2025-08-14T21:25:25.4196655Z  2025-08-14T21:25:25.4196796Z retry () { 2025-08-14T21:25:25.4196963Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-08-14T21:25:25.4197156Z } 2025-08-14T21:25:25.4197293Z  2025-08-14T21:25:25.4197440Z retry login "${DOCKER_REGISTRY}" 2025-08-14T21:25:25.4197626Z  2025-08-14T21:25:25.4197770Z START_TIME=$(date +%s) 2025-08-14T21:25:25.4197961Z # Wait up to 120 minutes 2025-08-14T21:25:25.4198187Z while [[ $(( $(date +%s) - 7200 )) -lt $START_TIME ]]; do 2025-08-14T21:25:25.4198475Z  # Check if image already exists, if it does then skip building it 2025-08-14T21:25:25.4198766Z  if docker manifest inspect "${DOCKER_IMAGE}"; then 2025-08-14T21:25:25.4198977Z  exit 0 2025-08-14T21:25:25.4199130Z  fi 2025-08-14T21:25:25.4199273Z  2025-08-14T21:25:25.4199510Z  # NB: This flag is used by Docker build workflow to push the image to ECR, so we can 2025-08-14T21:25:25.4199891Z  # use this to differentiate between the Docker build and regular build jobs. For the 2025-08-14T21:25:25.4200263Z  # latter, it will wait for the Docker images to become available before continuing 2025-08-14T21:25:25.4200578Z  if [ "${DOCKER_PUSH:-false}" == "true" ]; then 2025-08-14T21:25:25.4200816Z  # It's a Docker build job, let's build the image 2025-08-14T21:25:25.4201024Z  break 2025-08-14T21:25:25.4201175Z  else 2025-08-14T21:25:25.4201384Z  # It's a regular build job, wait for the image to become available 2025-08-14T21:25:25.4201620Z  sleep 300 2025-08-14T21:25:25.4201777Z  fi 2025-08-14T21:25:25.4201918Z done 2025-08-14T21:25:25.4202049Z  2025-08-14T21:25:25.4202268Z # NB: This part requires a full checkout. Otherwise, the merge base will 2025-08-14T21:25:25.4202714Z # be empty. The default action would be to continue rebuild the image 2025-08-14T21:25:25.4203054Z if [[ "$BASE_REVISION" = "$(git rev-parse HEAD)" ]]; then 2025-08-14T21:25:25.4203321Z  # if we're on the base branch then use the parent commit 2025-08-14T21:25:25.4203577Z  MERGE_BASE=$(git rev-parse HEAD~) 2025-08-14T21:25:25.4203782Z else 2025-08-14T21:25:25.4203984Z  # otherwise we're on a PR, so use the most recent base commit 2025-08-14T21:25:25.4204471Z  MERGE_BASE=$(git merge-base HEAD "$BASE_REVISION") 2025-08-14T21:25:25.4204699Z fi 2025-08-14T21:25:25.4204846Z  2025-08-14T21:25:25.4205003Z if [[ -z "${MERGE_BASE}" ]]; then 2025-08-14T21:25:25.4205239Z  echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-08-14T21:25:25.4205450Z  2025-08-14T21:25:25.4205838Z  echo "Finding merge base only works with full checkout, please set fetch-depth to 0, continuing ..." 2025-08-14T21:25:25.4206260Z  exit 0 2025-08-14T21:25:25.4206414Z fi 2025-08-14T21:25:25.4206552Z  2025-08-14T21:25:25.4206756Z if ! git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}"; then 2025-08-14T21:25:25.4207156Z  echo "Directory '${DOCKER_BUILD_DIR}' not found in commit $MERGE_BASE, you should rebase onto a more recent commit" 2025-08-14T21:25:25.4207498Z  exit 1 2025-08-14T21:25:25.4207639Z fi 2025-08-14T21:25:25.4207781Z  2025-08-14T21:25:25.4208005Z PREVIOUS_DOCKER_TAG=$(git rev-parse "${MERGE_BASE}:${DOCKER_BUILD_DIR}") 2025-08-14T21:25:25.4208374Z # If no image exists but the hash is the same as the previous hash then we should error out here 2025-08-14T21:25:25.4208705Z if [[ "${PREVIOUS_DOCKER_TAG}" == "${DOCKER_TAG}" ]]; then 2025-08-14T21:25:25.4209093Z  echo "WARNING: Something has gone wrong and the previous image isn't available for the merge-base of your branch" 2025-08-14T21:25:25.4209519Z  echo " Will re-build docker image to store in local cache, TTS may be longer" 2025-08-14T21:25:25.4209773Z fi 2025-08-14T21:25:25.4209909Z  2025-08-14T21:25:25.4210080Z echo "rebuild=true" >> "${GITHUB_OUTPUT}" 2025-08-14T21:25:25.4214675Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:25:25.4214913Z env: 2025-08-14T21:25:25.4215074Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:25:25.4215252Z DOCKER_BUILD_DIR: .ci/docker 2025-08-14T21:25:25.4215477Z BASE_REVISION: 1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:25:25.4216034Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:25:25.4216739Z DOCKER_TAG: pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:25:25.4217176Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:25:25.4217421Z DOCKER_PUSH: 2025-08-14T21:25:25.4217576Z ##[endgroup] 2025-08-14T21:25:25.4238275Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:25:25.4238754Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:25:25.4241404Z + aws ecr get-login-password --region us-east-1 2025-08-14T21:25:25.4244015Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:25:25.8583135Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-08-14T21:25:25.8583722Z Configure a credential helper to remove this warning. See 2025-08-14T21:25:25.8583986Z Login Succeeded 2025-08-14T21:25:25.8584294Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-08-14T21:25:25.8584644Z 2025-08-14T21:25:25.8610230Z ++ date +%s 2025-08-14T21:25:25.8620140Z + START_TIME=1755206725 2025-08-14T21:25:25.8622345Z ++ date +%s 2025-08-14T21:25:25.8635197Z + [[ 1755199525 -lt 1755206725 ]] 2025-08-14T21:25:25.8635905Z + docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:25:26.0724272Z { 2025-08-14T21:25:26.0724520Z "schemaVersion": 2, 2025-08-14T21:25:26.0724893Z "mediaType": "application/vnd.docker.distribution.manifest.v2+json", 2025-08-14T21:25:26.0725290Z "config": { 2025-08-14T21:25:26.0725837Z "mediaType": "application/vnd.docker.container.image.v1+json", 2025-08-14T21:25:26.0726140Z "size": 30151, 2025-08-14T21:25:26.0726447Z "digest": "sha256:0899ae453036ee7a91795ea95b1db61000579eeb74b140edab5976919ee64bbe" 2025-08-14T21:25:26.0726817Z }, 2025-08-14T21:25:26.0726961Z "layers": [ 2025-08-14T21:25:26.0727118Z { 2025-08-14T21:25:26.0727358Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0728083Z "size": 30448173, 2025-08-14T21:25:26.0728419Z "digest": "sha256:660ffc76f83b006444a5731b215acc2e35138d8be5cac8ed1ffd40f947117495" 2025-08-14T21:25:26.0728758Z }, 2025-08-14T21:25:26.0728912Z { 2025-08-14T21:25:26.0729176Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0729454Z "size": 1554, 2025-08-14T21:25:26.0729732Z "digest": "sha256:c7b4a852a45516e27a9256df90878663d770f96d271d6155d43be78cc5225eef" 2025-08-14T21:25:26.0730039Z }, 2025-08-14T21:25:26.0730177Z { 2025-08-14T21:25:26.0730406Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0730686Z "size": 313280151, 2025-08-14T21:25:26.0730990Z "digest": "sha256:e5a28988c8932eb5797557621582a064ce48651dbb5eaed379e9978535daccb9" 2025-08-14T21:25:26.0731294Z }, 2025-08-14T21:25:26.0731422Z { 2025-08-14T21:25:26.0731649Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0731918Z "size": 793, 2025-08-14T21:25:26.0732204Z "digest": "sha256:76a69b57b6837bef07dbc1b481cf28a62dfd7c7063219d9f6e0d0d63067653c7" 2025-08-14T21:25:26.0732503Z }, 2025-08-14T21:25:26.0732639Z { 2025-08-14T21:25:26.0732855Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0733122Z "size": 106, 2025-08-14T21:25:26.0733407Z "digest": "sha256:5c785dcb4cdbf1f2ceffe4d1d8e85d73225a56d0236e7ed6e36a95c836996052" 2025-08-14T21:25:26.0733719Z }, 2025-08-14T21:25:26.0733849Z { 2025-08-14T21:25:26.0734073Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0734343Z "size": 704, 2025-08-14T21:25:26.0734616Z "digest": "sha256:836ab08052e8eb2bae68e69ae086fd23a5f04a8491c320718ab47f84f03aebb1" 2025-08-14T21:25:26.0734923Z }, 2025-08-14T21:25:26.0735061Z { 2025-08-14T21:25:26.0735274Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0735551Z "size": 1217, 2025-08-14T21:25:26.0735834Z "digest": "sha256:53b11c77468cbefca210560f7d8be8e58f9eeb415e096ab0c3fb0277f0b41caf" 2025-08-14T21:25:26.0736152Z }, 2025-08-14T21:25:26.0736284Z { 2025-08-14T21:25:26.0736508Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0736780Z "size": 485, 2025-08-14T21:25:26.0737046Z "digest": "sha256:e97311a6a967664cbe10c5027a1ec60c514caa9a1160167d8363088fd1f9fe09" 2025-08-14T21:25:26.0737341Z }, 2025-08-14T21:25:26.0737479Z { 2025-08-14T21:25:26.0737885Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0738173Z "size": 110343699, 2025-08-14T21:25:26.0738470Z "digest": "sha256:2c414689d31dc46a22fe02d4f43699f528cc1c02fb505824768383fa0bbf1c74" 2025-08-14T21:25:26.0738771Z }, 2025-08-14T21:25:26.0738913Z { 2025-08-14T21:25:26.0739137Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0739414Z "size": 4817, 2025-08-14T21:25:26.0739700Z "digest": "sha256:6d89b5f065d59e4abcaa9b5ff3bf0afded2394d493d2df0f7babf7154f7548e0" 2025-08-14T21:25:26.0740146Z }, 2025-08-14T21:25:26.0740288Z { 2025-08-14T21:25:26.0740499Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0740770Z "size": 1709, 2025-08-14T21:25:26.0741047Z "digest": "sha256:5a5cc76ada432cccf7d18e0eb79379afb95deaaa7afec482406267924d291ae4" 2025-08-14T21:25:26.0741358Z }, 2025-08-14T21:25:26.0741498Z { 2025-08-14T21:25:26.0741716Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0741980Z "size": 724, 2025-08-14T21:25:26.0742379Z "digest": "sha256:fc6b37d40530f2c5339430321eab67ae1e2e87e997587c7bc8c41504464208f9" 2025-08-14T21:25:26.0742729Z }, 2025-08-14T21:25:26.0742871Z { 2025-08-14T21:25:26.0743082Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0743356Z "size": 542, 2025-08-14T21:25:26.0743622Z "digest": "sha256:2e16579078600b91216fd14aca1e0ce0f9d1801b230689dd309980e8d2783935" 2025-08-14T21:25:26.0744005Z }, 2025-08-14T21:25:26.0744145Z { 2025-08-14T21:25:26.0744366Z + exit 0 2025-08-14T21:25:26.0744640Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0744915Z "size": 3397512507, 2025-08-14T21:25:26.0745198Z "digest": "sha256:7b92d7a4b8c766d7b7873aa33088e171fb44a8e968645e4b31dfe6de2968aead" 2025-08-14T21:25:26.0745503Z }, 2025-08-14T21:25:26.0745635Z { 2025-08-14T21:25:26.0745849Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0746125Z "size": 32, 2025-08-14T21:25:26.0746394Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-14T21:25:26.0746702Z }, 2025-08-14T21:25:26.0746842Z { 2025-08-14T21:25:26.0747051Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0747329Z "size": 380, 2025-08-14T21:25:26.0747597Z "digest": "sha256:d6226eb61f823984003d5ac28f4d66fec9b27baf5d54a9513286483f5912cd88" 2025-08-14T21:25:26.0747898Z }, 2025-08-14T21:25:26.0748030Z { 2025-08-14T21:25:26.0748245Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0748543Z "size": 234681, 2025-08-14T21:25:26.0748817Z "digest": "sha256:83c70f4266a6ee5f8f44a88d4cb951382f6c960323b8250046bddc080e62268b" 2025-08-14T21:25:26.0749171Z }, 2025-08-14T21:25:26.0749308Z { 2025-08-14T21:25:26.0749523Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0749790Z "size": 231, 2025-08-14T21:25:26.0750052Z "digest": "sha256:60c725d21861c24c417efe3a5474414ba04f0f49c78c6d6451478ab9e45469ec" 2025-08-14T21:25:26.0750353Z }, 2025-08-14T21:25:26.0750491Z { 2025-08-14T21:25:26.0750696Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0750966Z "size": 4464546, 2025-08-14T21:25:26.0751246Z "digest": "sha256:a504e76e66a49926b4ea837b7a7ff3c842a27b2caaa4d80cf5057a1e55293666" 2025-08-14T21:25:26.0751545Z }, 2025-08-14T21:25:26.0751679Z { 2025-08-14T21:25:26.0751886Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0752142Z "size": 1864, 2025-08-14T21:25:26.0752422Z "digest": "sha256:fc1c200a4f77face2af0146f9b03ad04f31fe06fec216473ffd2ebd538cde056" 2025-08-14T21:25:26.0752727Z }, 2025-08-14T21:25:26.0752856Z { 2025-08-14T21:25:26.0753071Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0753337Z "size": 475, 2025-08-14T21:25:26.0753595Z "digest": "sha256:43273c22704f81f162741d2039015f745273eee1d1fdec47be35c9b2a90dcc5b" 2025-08-14T21:25:26.0753874Z }, 2025-08-14T21:25:26.0754004Z { 2025-08-14T21:25:26.0754210Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0754451Z "size": 178, 2025-08-14T21:25:26.0754707Z "digest": "sha256:89df389d042adbd7621a94d36b6e3db60ff6c559efb95c6fcc11b8afd42f0599" 2025-08-14T21:25:26.0754995Z }, 2025-08-14T21:25:26.0755118Z { 2025-08-14T21:25:26.0755388Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0755644Z "size": 586, 2025-08-14T21:25:26.0755889Z "digest": "sha256:684349f50d9456597026ee5c1bd890c51d1e498614f367adf03329c5227add79" 2025-08-14T21:25:26.0756168Z }, 2025-08-14T21:25:26.0756299Z { 2025-08-14T21:25:26.0756497Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0756745Z "size": 218, 2025-08-14T21:25:26.0757003Z "digest": "sha256:21d0eae87fb3ac753b3f0e91ae638360d23922d4cd119410a5a1b97bbe0ca435" 2025-08-14T21:25:26.0757290Z }, 2025-08-14T21:25:26.0757415Z { 2025-08-14T21:25:26.0757624Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0757874Z "size": 802, 2025-08-14T21:25:26.0758125Z "digest": "sha256:c9c2b424b8e08d943dc259a3796d66eede3a1e93a6460df5db132c0036d3d6af" 2025-08-14T21:25:26.0758408Z }, 2025-08-14T21:25:26.0758538Z { 2025-08-14T21:25:26.0758737Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0759035Z "size": 32, 2025-08-14T21:25:26.0759290Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-14T21:25:26.0759577Z }, 2025-08-14T21:25:26.0759713Z { 2025-08-14T21:25:26.0759923Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0760174Z "size": 104, 2025-08-14T21:25:26.0760442Z "digest": "sha256:98dda28f339592e3ca6d589d551e69b8314f2b7fc2a1544eacc1b3c2d3378521" 2025-08-14T21:25:26.0760736Z }, 2025-08-14T21:25:26.0760873Z { 2025-08-14T21:25:26.0761080Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0761341Z "size": 1496, 2025-08-14T21:25:26.0761612Z "digest": "sha256:acf5babd87f23aa905883eb434073e9a00ff41679134f2f4827dd86949f5a9d9" 2025-08-14T21:25:26.0761903Z }, 2025-08-14T21:25:26.0762041Z { 2025-08-14T21:25:26.0762255Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0762512Z "size": 453555614, 2025-08-14T21:25:26.0762794Z "digest": "sha256:7c5050d8408d3c4f9f5e8f2cb215245473bfc2f1510fe5ee01c2a6c505068b5a" 2025-08-14T21:25:26.0763090Z }, 2025-08-14T21:25:26.0763218Z { 2025-08-14T21:25:26.0763429Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0763687Z "size": 163, 2025-08-14T21:25:26.0763949Z "digest": "sha256:7ddd14e2b548b9ae6e216a081bb20116434aacbbe571c99b40e60fb2fde22a2a" 2025-08-14T21:25:26.0764249Z }, 2025-08-14T21:25:26.0764386Z { 2025-08-14T21:25:26.0764596Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0764847Z "size": 347, 2025-08-14T21:25:26.0765113Z "digest": "sha256:4ba8e7a736c8199931fd7ff9931a5f17b7b931d0383a3e158f1b12b191a1d250" 2025-08-14T21:25:26.0765413Z }, 2025-08-14T21:25:26.0765639Z { 2025-08-14T21:25:26.0765874Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0766147Z "size": 32, 2025-08-14T21:25:26.0766434Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-14T21:25:26.0766751Z }, 2025-08-14T21:25:26.0766895Z { 2025-08-14T21:25:26.0767115Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0767394Z "size": 106, 2025-08-14T21:25:26.0767671Z "digest": "sha256:907c320fee2f90da0cf5028c90a0ef49a137518baf79b483dcf7f22d5a0a497d" 2025-08-14T21:25:26.0767987Z }, 2025-08-14T21:25:26.0768121Z { 2025-08-14T21:25:26.0768345Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0768620Z "size": 425, 2025-08-14T21:25:26.0768901Z "digest": "sha256:18c4ed1ec491095788e352ae018afd84de0f251fbcfb8f74d5d893e1e9ab196d" 2025-08-14T21:25:26.0769215Z }, 2025-08-14T21:25:26.0769357Z { 2025-08-14T21:25:26.0769573Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0769848Z "size": 19308711, 2025-08-14T21:25:26.0770145Z "digest": "sha256:d7618c2df6cdb4bbf3d9870ba2d089094ac46c429b573d9adb94411fac54cfca" 2025-08-14T21:25:26.0770520Z }, 2025-08-14T21:25:26.0770668Z { 2025-08-14T21:25:26.0770896Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0771159Z "size": 108, 2025-08-14T21:25:26.0771440Z "digest": "sha256:b7bdd9a6f789ba483a46c92e5d373638850f33e88b1baa4bbe67e1c6a09cb7d0" 2025-08-14T21:25:26.0771750Z }, 2025-08-14T21:25:26.0771889Z { 2025-08-14T21:25:26.0772103Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0772377Z "size": 691, 2025-08-14T21:25:26.0772659Z "digest": "sha256:6738ba83282e002d92bff3d2b4951e3c1a67f5ec2c1bad2fd780c2f5d444748f" 2025-08-14T21:25:26.0772964Z }, 2025-08-14T21:25:26.0773105Z { 2025-08-14T21:25:26.0773329Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0773591Z "size": 724, 2025-08-14T21:25:26.0773864Z "digest": "sha256:fc6b37d40530f2c5339430321eab67ae1e2e87e997587c7bc8c41504464208f9" 2025-08-14T21:25:26.0774241Z }, 2025-08-14T21:25:26.0774378Z { 2025-08-14T21:25:26.0774601Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0774870Z "size": 116, 2025-08-14T21:25:26.0775132Z "digest": "sha256:dfb0f24886393e1d394f1f433dc9346026679dafd7a60c3a93de17d94078c1ca" 2025-08-14T21:25:26.0775435Z }, 2025-08-14T21:25:26.0775576Z { 2025-08-14T21:25:26.0775800Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0776062Z "size": 136, 2025-08-14T21:25:26.0776343Z "digest": "sha256:dc833b0762f2e144670a660f6b7ce62cec71a5fdd24df4e67b5c6173d5834451" 2025-08-14T21:25:26.0776625Z }, 2025-08-14T21:25:26.0776750Z { 2025-08-14T21:25:26.0776954Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0777200Z "size": 139, 2025-08-14T21:25:26.0777446Z "digest": "sha256:8827df8ca2da347e0032d1bff3b0312437f711c5d0b5f2164f8a60c3368a9827" 2025-08-14T21:25:26.0777726Z }, 2025-08-14T21:25:26.0777858Z { 2025-08-14T21:25:26.0778055Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0778308Z "size": 17672683360, 2025-08-14T21:25:26.0778579Z "digest": "sha256:fac8f3bd0f85eaffb43df539683dc3d861c370e583623253559fd7a1f5b00229" 2025-08-14T21:25:26.0778863Z }, 2025-08-14T21:25:26.0778988Z { 2025-08-14T21:25:26.0779190Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0779435Z "size": 214, 2025-08-14T21:25:26.0779680Z "digest": "sha256:d7cf7f140df32761610e1d58686db7f7c66a85affa4bb4b9d3c245e232443a8f" 2025-08-14T21:25:26.0779963Z }, 2025-08-14T21:25:26.0780092Z { 2025-08-14T21:25:26.0780286Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0780536Z "size": 272992162, 2025-08-14T21:25:26.0780806Z "digest": "sha256:733eedc8da8d8e7bd5a85a58d3d7818f14ed9a4fdf2dbd587038bb7725fbb9f7" 2025-08-14T21:25:26.0781087Z }, 2025-08-14T21:25:26.0781219Z { 2025-08-14T21:25:26.0781430Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0781676Z "size": 6435582332, 2025-08-14T21:25:26.0781941Z "digest": "sha256:5b092eb06909a2ea8906849acac588a10864da349670d65c0bfea342187edba2" 2025-08-14T21:25:26.0782217Z }, 2025-08-14T21:25:26.0782347Z { 2025-08-14T21:25:26.0782539Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0782783Z "size": 129, 2025-08-14T21:25:26.0783023Z "digest": "sha256:bc596103109216e154006085503386753b0b114b5900bf44758cdff324df5504" 2025-08-14T21:25:26.0783282Z }, 2025-08-14T21:25:26.0783415Z { 2025-08-14T21:25:26.0783615Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0783854Z "size": 776, 2025-08-14T21:25:26.0784108Z "digest": "sha256:0531cc34c12ab9127f1858c4cf365bb3a02bc31e8d6df5eabba2e1b6ef026ccf" 2025-08-14T21:25:26.0784393Z }, 2025-08-14T21:25:26.0784516Z { 2025-08-14T21:25:26.0784714Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0785016Z "size": 724, 2025-08-14T21:25:26.0785267Z "digest": "sha256:fc6b37d40530f2c5339430321eab67ae1e2e87e997587c7bc8c41504464208f9" 2025-08-14T21:25:26.0785543Z }, 2025-08-14T21:25:26.0785670Z { 2025-08-14T21:25:26.0785871Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0786112Z "size": 141, 2025-08-14T21:25:26.0786369Z "digest": "sha256:38c303d3b62eb463762816db04062a480014a6f3c9754386f3e83ba331ab4d1d" 2025-08-14T21:25:26.0786656Z }, 2025-08-14T21:25:26.0786786Z { 2025-08-14T21:25:26.0787002Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0787263Z "size": 32, 2025-08-14T21:25:26.0787525Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-14T21:25:26.0787822Z }, 2025-08-14T21:25:26.0787958Z { 2025-08-14T21:25:26.0788165Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0788473Z "size": 160, 2025-08-14T21:25:26.0788745Z "digest": "sha256:e06d15594a2a76995baebbce7032946ff9f94e281246fbc3f8ab19d8bcc38b81" 2025-08-14T21:25:26.0789043Z }, 2025-08-14T21:25:26.0789174Z { 2025-08-14T21:25:26.0789388Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0789649Z "size": 1010, 2025-08-14T21:25:26.0789920Z "digest": "sha256:0e55deb5cb38fd36b600183f7d86eaca0dabc04d2ff4d49ec2266ee3329edc4a" 2025-08-14T21:25:26.0790225Z }, 2025-08-14T21:25:26.0790363Z { 2025-08-14T21:25:26.0790567Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0790829Z "size": 724, 2025-08-14T21:25:26.0791093Z "digest": "sha256:fc6b37d40530f2c5339430321eab67ae1e2e87e997587c7bc8c41504464208f9" 2025-08-14T21:25:26.0791379Z }, 2025-08-14T21:25:26.0791516Z { 2025-08-14T21:25:26.0791729Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0791982Z "size": 134, 2025-08-14T21:25:26.0792267Z "digest": "sha256:4a53d66dce071bb7416414aa1adbc3e4a59003300c0d42038612fabdeb5a1b01" 2025-08-14T21:25:26.0792569Z }, 2025-08-14T21:25:26.0792706Z { 2025-08-14T21:25:26.0792923Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0793184Z "size": 32, 2025-08-14T21:25:26.0793449Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-14T21:25:26.0793742Z }, 2025-08-14T21:25:26.0793878Z { 2025-08-14T21:25:26.0794089Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0794340Z "size": 159, 2025-08-14T21:25:26.0794604Z "digest": "sha256:1519daa051b8b80e04125f2f2215dc412dcdbb9502711925e97aeccbda069eaf" 2025-08-14T21:25:26.0794899Z }, 2025-08-14T21:25:26.0795028Z { 2025-08-14T21:25:26.0795241Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0795500Z "size": 1371, 2025-08-14T21:25:26.0795772Z "digest": "sha256:381ed91d2119f078fbba19102a65befc4cb242f8cf47a11fb6f76ea424690692" 2025-08-14T21:25:26.0796073Z }, 2025-08-14T21:25:26.0796211Z { 2025-08-14T21:25:26.0796425Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0796680Z "size": 32, 2025-08-14T21:25:26.0796945Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-14T21:25:26.0797241Z }, 2025-08-14T21:25:26.0797371Z { 2025-08-14T21:25:26.0797583Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0797839Z "size": 137, 2025-08-14T21:25:26.0798102Z "digest": "sha256:c6b0a01a96dd479640297d4b012031ffc1bd9fc0daf61d86058f9b675c0a0705" 2025-08-14T21:25:26.0798396Z }, 2025-08-14T21:25:26.0798534Z { 2025-08-14T21:25:26.0798739Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0798999Z "size": 380, 2025-08-14T21:25:26.0799270Z "digest": "sha256:62df6413daeefebde04dcc401134734952e4ea37fc85ff23c89cb9b4fbd45155" 2025-08-14T21:25:26.0799621Z }, 2025-08-14T21:25:26.0799754Z { 2025-08-14T21:25:26.0799968Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0800235Z "size": 32, 2025-08-14T21:25:26.0800501Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-14T21:25:26.0800814Z }, 2025-08-14T21:25:26.0800942Z { 2025-08-14T21:25:26.0801134Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0801376Z "size": 104, 2025-08-14T21:25:26.0801630Z "digest": "sha256:7a18bc2a6881b76a6f591c98dafb47e44d903f7a905f7eba0fc3aedb5c90fff7" 2025-08-14T21:25:26.0801907Z }, 2025-08-14T21:25:26.0802036Z { 2025-08-14T21:25:26.0802241Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0802487Z "size": 407, 2025-08-14T21:25:26.0802742Z "digest": "sha256:93359cd58a8cece344fd4291b27647e57761c9399bb54bb0c18149c12af5f66a" 2025-08-14T21:25:26.0803062Z }, 2025-08-14T21:25:26.0803191Z { 2025-08-14T21:25:26.0803403Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0803666Z "size": 32, 2025-08-14T21:25:26.0803936Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-14T21:25:26.0804237Z }, 2025-08-14T21:25:26.0804374Z { 2025-08-14T21:25:26.0804588Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0804857Z "size": 109, 2025-08-14T21:25:26.0805129Z "digest": "sha256:c35ba0a1f353d6894c914a4bfbea9a2c9b8ac1b526af64d34cbe9a12bd83c78e" 2025-08-14T21:25:26.0805445Z }, 2025-08-14T21:25:26.0805670Z { 2025-08-14T21:25:26.0805894Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0806157Z "size": 1896, 2025-08-14T21:25:26.0806425Z "digest": "sha256:dcf1e01c98d6a6f72674d79a4e8e4047b54796576cd06ad682c225a92820a8f5" 2025-08-14T21:25:26.0806743Z }, 2025-08-14T21:25:26.0806885Z { 2025-08-14T21:25:26.0807124Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0807401Z "size": 242635753, 2025-08-14T21:25:26.0807695Z "digest": "sha256:bad0564f61fdf377e3ae31f6fec0ec28b6922da0b9db28408b55b8e97ff1ea51" 2025-08-14T21:25:26.0808003Z }, 2025-08-14T21:25:26.0808125Z { 2025-08-14T21:25:26.0808329Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0808589Z "size": 106, 2025-08-14T21:25:26.0808830Z "digest": "sha256:539ded9057364aade7abe23ab908d2caf53966a186734aa58ae84a56bee659eb" 2025-08-14T21:25:26.0809105Z }, 2025-08-14T21:25:26.0809233Z { 2025-08-14T21:25:26.0809423Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0809665Z "size": 163, 2025-08-14T21:25:26.0809904Z "digest": "sha256:28d482062637d32514edfc447913e98745d7c13d2f277531e64ffcf090ae6d92" 2025-08-14T21:25:26.0810173Z }, 2025-08-14T21:25:26.0810291Z { 2025-08-14T21:25:26.0810490Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0810734Z "size": 7943, 2025-08-14T21:25:26.0810978Z "digest": "sha256:3245316ff51b50b27da4ef7279733c92f76cc652b3fce3877c0e3d510430e8b3" 2025-08-14T21:25:26.0811251Z }, 2025-08-14T21:25:26.0811377Z { 2025-08-14T21:25:26.0811570Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0811810Z "size": 8073, 2025-08-14T21:25:26.0812062Z "digest": "sha256:b53167d1a6df0e4b67d637d073150dff1fb87a823864c0c98d77c15e56babc24" 2025-08-14T21:25:26.0812331Z }, 2025-08-14T21:25:26.0812457Z { 2025-08-14T21:25:26.0812661Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0812901Z "size": 303, 2025-08-14T21:25:26.0813152Z "digest": "sha256:7f5277f691672469f431fd90a8c2bb702c6c68333f6be2cff868f00e416c5a1a" 2025-08-14T21:25:26.0813430Z }, 2025-08-14T21:25:26.0813561Z { 2025-08-14T21:25:26.0813768Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0814062Z "size": 32, 2025-08-14T21:25:26.0814316Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-14T21:25:26.0814587Z }, 2025-08-14T21:25:26.0814711Z { 2025-08-14T21:25:26.0814910Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0815147Z "size": 108, 2025-08-14T21:25:26.0815398Z "digest": "sha256:23dff10cdaa5b1e9c7250f0c58a6279f104b35408281e951bfe9983f97e3d9ed" 2025-08-14T21:25:26.0815682Z }, 2025-08-14T21:25:26.0815806Z { 2025-08-14T21:25:26.0816013Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0816270Z "size": 54145699, 2025-08-14T21:25:26.0816538Z "digest": "sha256:9fb73296da6ac15f37f36663bd10afc98abb8a01fb40bff4848de7247d28e018" 2025-08-14T21:25:26.0816827Z }, 2025-08-14T21:25:26.0816958Z { 2025-08-14T21:25:26.0817161Z "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip", 2025-08-14T21:25:26.0817474Z "size": 32, 2025-08-14T21:25:26.0817755Z "digest": "sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1" 2025-08-14T21:25:26.0818066Z } 2025-08-14T21:25:26.0818199Z ] 2025-08-14T21:25:26.0818343Z } 2025-08-14T21:25:26.0838653Z ##[group]Run set -eux 2025-08-14T21:25:26.0838873Z set -eux 2025-08-14T21:25:26.0839407Z aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token | jq --raw-output '.SecretString' | jq -r .docker_hub_readonly_token | docker login --username pytorchbot --password-stdin 2025-08-14T21:25:26.0845376Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:25:26.0845729Z env: 2025-08-14T21:25:26.0845932Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:25:26.0846126Z ##[endgroup] 2025-08-14T21:25:26.0875164Z + aws secretsmanager get-secret-value --secret-id docker_hub_readonly_token 2025-08-14T21:25:26.0879970Z + jq --raw-output .SecretString 2025-08-14T21:25:26.0880251Z + jq -r .docker_hub_readonly_token 2025-08-14T21:25:26.0880574Z + docker login --username pytorchbot --password-stdin 2025-08-14T21:25:26.5722310Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-08-14T21:25:26.5722720Z Login Succeeded 2025-08-14T21:25:26.5723279Z Configure a credential helper to remove this warning. See 2025-08-14T21:25:26.5723687Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-08-14T21:25:26.5723939Z 2025-08-14T21:25:26.5790900Z ##[group]Run tag=${ECR_DOCKER_IMAGE##*:} 2025-08-14T21:25:26.5791163Z tag=${ECR_DOCKER_IMAGE##*:} 2025-08-14T21:25:26.5791411Z echo "docker pull ghcr.io/pytorch/ci-image:${tag/:/-}" 2025-08-14T21:25:26.5795917Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:25:26.5796156Z env: 2025-08-14T21:25:26.5796306Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:25:26.5796831Z ECR_DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:25:26.5797373Z ##[endgroup] 2025-08-14T21:25:26.5822793Z docker pull ghcr.io/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:25:26.5855694Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main 2025-08-14T21:25:26.5855985Z with: 2025-08-14T21:25:26.5856511Z docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:25:26.5857117Z docker-registry: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:25:26.5857380Z env: 2025-08-14T21:25:26.5857544Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:25:26.5857721Z ##[endgroup] 2025-08-14T21:25:26.5867874Z ##[group]Run set -x 2025-08-14T21:25:26.5868086Z set -x 2025-08-14T21:25:26.5868254Z set +e 2025-08-14T21:25:26.5868417Z  2025-08-14T21:25:26.5868581Z login() { 2025-08-14T21:25:26.5868897Z  aws ecr get-login-password --region us-east-1 | docker login -u AWS --password-stdin "$1" 2025-08-14T21:25:26.5869255Z } 2025-08-14T21:25:26.5869419Z  2025-08-14T21:25:26.5869628Z retry () { 2025-08-14T21:25:26.5869842Z  $* || (sleep 1 && $*) || (sleep 2 && $*) 2025-08-14T21:25:26.5870083Z } 2025-08-14T21:25:26.5870251Z  2025-08-14T21:25:26.5870438Z retry login "${DOCKER_REGISTRY}" 2025-08-14T21:25:26.5870666Z  2025-08-14T21:25:26.5871030Z IMAGE_SIZE=$(docker manifest inspect "${DOCKER_IMAGE}" | jq '[.layers[].size, .config.size] | add / 1024 / 1024') 2025-08-14T21:25:26.5871501Z echo "Compressed size of image in MB: ${IMAGE_SIZE}" 2025-08-14T21:25:26.5871773Z  2025-08-14T21:25:26.5871941Z set -e 2025-08-14T21:25:26.5872210Z # ignore output since only exit code is used for conditional 2025-08-14T21:25:26.5872714Z # only pull docker image if it's not available locally 2025-08-14T21:25:26.5873131Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then 2025-08-14T21:25:26.5873519Z  retry docker pull "${DOCKER_IMAGE}" 2025-08-14T21:25:26.5873781Z fi 2025-08-14T21:25:26.5878458Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:25:26.5878727Z env: 2025-08-14T21:25:26.5878902Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:25:26.5879503Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:25:26.5880187Z DOCKER_REGISTRY: 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:25:26.5880447Z ##[endgroup] 2025-08-14T21:25:26.5902142Z + set +e 2025-08-14T21:25:26.5902491Z + retry login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:25:26.5902850Z + login 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:25:26.5908294Z + docker login -u AWS --password-stdin 308535385114.dkr.ecr.us-east-1.amazonaws.com 2025-08-14T21:25:26.5911818Z + aws ecr get-login-password --region us-east-1 2025-08-14T21:25:27.0198112Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json. 2025-08-14T21:25:27.0198509Z Configure a credential helper to remove this warning. See 2025-08-14T21:25:27.0198877Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store 2025-08-14T21:25:27.0199106Z 2025-08-14T21:25:27.0199355Z Login Succeeded 2025-08-14T21:25:27.0225637Z ++ docker manifest inspect 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:25:27.0226264Z ++ jq '[.layers[].size, .config.size] | add / 1024 / 1024' 2025-08-14T21:25:27.2452936Z + IMAGE_SIZE=27663.483686447144 2025-08-14T21:25:27.2453363Z Compressed size of image in MB: 27663.483686447144 2025-08-14T21:25:27.2458721Z + echo 'Compressed size of image in MB: 27663.483686447144' 2025-08-14T21:25:27.2462854Z + set -e 2025-08-14T21:25:27.2467371Z + docker inspect --type=image 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:25:27.2581747Z + retry docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:25:27.2582895Z + docker pull 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:25:27.4955346Z pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe: Pulling from pytorch/ci-image 2025-08-14T21:25:27.4955944Z 660ffc76f83b: Pulling fs layer 2025-08-14T21:25:27.4956160Z c7b4a852a455: Pulling fs layer 2025-08-14T21:25:27.4956391Z e5a28988c893: Pulling fs layer 2025-08-14T21:25:27.4956584Z 76a69b57b683: Pulling fs layer 2025-08-14T21:25:27.4956779Z 5c785dcb4cdb: Pulling fs layer 2025-08-14T21:25:27.4956967Z 836ab08052e8: Pulling fs layer 2025-08-14T21:25:27.4957172Z 53b11c77468c: Pulling fs layer 2025-08-14T21:25:27.4957366Z e97311a6a967: Pulling fs layer 2025-08-14T21:25:27.4957552Z 2c414689d31d: Pulling fs layer 2025-08-14T21:25:27.4957743Z 6d89b5f065d5: Pulling fs layer 2025-08-14T21:25:27.4957940Z 5a5cc76ada43: Pulling fs layer 2025-08-14T21:25:27.4958131Z fc6b37d40530: Pulling fs layer 2025-08-14T21:25:27.4958327Z 2e1657907860: Pulling fs layer 2025-08-14T21:25:27.4958515Z 7b92d7a4b8c7: Pulling fs layer 2025-08-14T21:25:27.4958696Z 4f4fb700ef54: Pulling fs layer 2025-08-14T21:25:27.4958892Z d6226eb61f82: Pulling fs layer 2025-08-14T21:25:27.4959124Z 83c70f4266a6: Pulling fs layer 2025-08-14T21:25:27.4959308Z 60c725d21861: Pulling fs layer 2025-08-14T21:25:27.4959486Z a504e76e66a4: Pulling fs layer 2025-08-14T21:25:27.4959951Z fc1c200a4f77: Pulling fs layer 2025-08-14T21:25:27.4960143Z 43273c22704f: Pulling fs layer 2025-08-14T21:25:27.4960324Z 89df389d042a: Pulling fs layer 2025-08-14T21:25:27.4960523Z 684349f50d94: Pulling fs layer 2025-08-14T21:25:27.4960721Z 21d0eae87fb3: Pulling fs layer 2025-08-14T21:25:27.4960906Z c9c2b424b8e0: Pulling fs layer 2025-08-14T21:25:27.4961105Z 98dda28f3395: Pulling fs layer 2025-08-14T21:25:27.4961300Z acf5babd87f2: Pulling fs layer 2025-08-14T21:25:27.4961486Z 7c5050d8408d: Pulling fs layer 2025-08-14T21:25:27.4961675Z 7ddd14e2b548: Pulling fs layer 2025-08-14T21:25:27.4961870Z 4ba8e7a736c8: Pulling fs layer 2025-08-14T21:25:27.4962049Z 907c320fee2f: Pulling fs layer 2025-08-14T21:25:27.4962237Z 18c4ed1ec491: Pulling fs layer 2025-08-14T21:25:27.4962427Z d7618c2df6cd: Pulling fs layer 2025-08-14T21:25:27.4962618Z b7bdd9a6f789: Pulling fs layer 2025-08-14T21:25:27.4962807Z 6738ba83282e: Pulling fs layer 2025-08-14T21:25:27.4962995Z dfb0f2488639: Pulling fs layer 2025-08-14T21:25:27.4963186Z dc833b0762f2: Pulling fs layer 2025-08-14T21:25:27.4963460Z 8827df8ca2da: Pulling fs layer 2025-08-14T21:25:27.4963740Z fac8f3bd0f85: Pulling fs layer 2025-08-14T21:25:27.4963943Z d7cf7f140df3: Pulling fs layer 2025-08-14T21:25:27.4964136Z 733eedc8da8d: Pulling fs layer 2025-08-14T21:25:27.4964333Z 5b092eb06909: Pulling fs layer 2025-08-14T21:25:27.4964520Z bc5961031092: Pulling fs layer 2025-08-14T21:25:27.4964709Z 0531cc34c12a: Pulling fs layer 2025-08-14T21:25:27.4964889Z 38c303d3b62e: Pulling fs layer 2025-08-14T21:25:27.4965074Z e06d15594a2a: Pulling fs layer 2025-08-14T21:25:27.4965265Z 0e55deb5cb38: Pulling fs layer 2025-08-14T21:25:27.4965444Z 4a53d66dce07: Pulling fs layer 2025-08-14T21:25:27.4965898Z 1519daa051b8: Pulling fs layer 2025-08-14T21:25:27.4966099Z 381ed91d2119: Pulling fs layer 2025-08-14T21:25:27.4966285Z c6b0a01a96dd: Pulling fs layer 2025-08-14T21:25:27.4966484Z 62df6413daee: Pulling fs layer 2025-08-14T21:25:27.4966690Z 7a18bc2a6881: Pulling fs layer 2025-08-14T21:25:27.4966881Z 93359cd58a8c: Pulling fs layer 2025-08-14T21:25:27.4967088Z c35ba0a1f353: Pulling fs layer 2025-08-14T21:25:27.4967289Z dcf1e01c98d6: Pulling fs layer 2025-08-14T21:25:27.4967482Z bad0564f61fd: Pulling fs layer 2025-08-14T21:25:27.4967801Z 539ded905736: Pulling fs layer 2025-08-14T21:25:27.4968000Z 28d482062637: Pulling fs layer 2025-08-14T21:25:27.4968186Z 3245316ff51b: Pulling fs layer 2025-08-14T21:25:27.4968367Z b53167d1a6df: Pulling fs layer 2025-08-14T21:25:27.4968563Z 7f5277f69167: Pulling fs layer 2025-08-14T21:25:27.4999561Z 23dff10cdaa5: Pulling fs layer 2025-08-14T21:25:27.5000037Z 9fb73296da6a: Pulling fs layer 2025-08-14T21:25:27.5000345Z 7ddd14e2b548: Waiting 2025-08-14T21:25:27.5000586Z 4ba8e7a736c8: Waiting 2025-08-14T21:25:27.5000769Z 4a53d66dce07: Waiting 2025-08-14T21:25:27.5000962Z dcf1e01c98d6: Waiting 2025-08-14T21:25:27.5001163Z bad0564f61fd: Waiting 2025-08-14T21:25:27.5001342Z 907c320fee2f: Waiting 2025-08-14T21:25:27.5001507Z 1519daa051b8: Waiting 2025-08-14T21:25:27.5001699Z 539ded905736: Waiting 2025-08-14T21:25:27.5001873Z 18c4ed1ec491: Waiting 2025-08-14T21:25:27.5002036Z 28d482062637: Waiting 2025-08-14T21:25:27.5002205Z 381ed91d2119: Waiting 2025-08-14T21:25:27.5002378Z d7618c2df6cd: Waiting 2025-08-14T21:25:27.5002549Z 3245316ff51b: Waiting 2025-08-14T21:25:27.5002724Z b53167d1a6df: Waiting 2025-08-14T21:25:27.5002895Z 5b092eb06909: Waiting 2025-08-14T21:25:27.5003122Z 7f5277f69167: Waiting 2025-08-14T21:25:27.5003316Z bc5961031092: Waiting 2025-08-14T21:25:27.5003497Z 0531cc34c12a: Waiting 2025-08-14T21:25:27.5003668Z 23dff10cdaa5: Waiting 2025-08-14T21:25:27.5003862Z 38c303d3b62e: Waiting 2025-08-14T21:25:27.5004039Z 9fb73296da6a: Waiting 2025-08-14T21:25:27.5004203Z b7bdd9a6f789: Waiting 2025-08-14T21:25:27.5004374Z 6738ba83282e: Waiting 2025-08-14T21:25:27.5004539Z e06d15594a2a: Waiting 2025-08-14T21:25:27.5004707Z 0e55deb5cb38: Waiting 2025-08-14T21:25:27.5004889Z c6b0a01a96dd: Waiting 2025-08-14T21:25:27.5005064Z fac8f3bd0f85: Waiting 2025-08-14T21:25:27.5005409Z d7cf7f140df3: Waiting 2025-08-14T21:25:27.5005781Z 62df6413daee: Waiting 2025-08-14T21:25:27.5006005Z dfb0f2488639: Waiting 2025-08-14T21:25:27.5006176Z dc833b0762f2: Waiting 2025-08-14T21:25:27.5006351Z 733eedc8da8d: Waiting 2025-08-14T21:25:27.5006540Z 7a18bc2a6881: Waiting 2025-08-14T21:25:27.5006712Z 8827df8ca2da: Waiting 2025-08-14T21:25:27.5006871Z 93359cd58a8c: Waiting 2025-08-14T21:25:27.5007034Z d6226eb61f82: Waiting 2025-08-14T21:25:27.5007196Z 684349f50d94: Waiting 2025-08-14T21:25:27.5007350Z 83c70f4266a6: Waiting 2025-08-14T21:25:27.5007516Z 76a69b57b683: Waiting 2025-08-14T21:25:27.5007681Z 21d0eae87fb3: Waiting 2025-08-14T21:25:27.5007835Z 60c725d21861: Waiting 2025-08-14T21:25:27.5007999Z 5c785dcb4cdb: Waiting 2025-08-14T21:25:27.5008165Z c9c2b424b8e0: Waiting 2025-08-14T21:25:27.5008320Z a504e76e66a4: Waiting 2025-08-14T21:25:27.5008483Z 836ab08052e8: Waiting 2025-08-14T21:25:27.5008647Z fc1c200a4f77: Waiting 2025-08-14T21:25:27.5008802Z 98dda28f3395: Waiting 2025-08-14T21:25:27.5008973Z 43273c22704f: Waiting 2025-08-14T21:25:27.5009136Z 53b11c77468c: Waiting 2025-08-14T21:25:27.5009299Z 7c5050d8408d: Waiting 2025-08-14T21:25:27.5030286Z 89df389d042a: Waiting 2025-08-14T21:25:27.5030488Z c35ba0a1f353: Waiting 2025-08-14T21:25:27.5030656Z e97311a6a967: Waiting 2025-08-14T21:25:27.5030809Z 2c414689d31d: Waiting 2025-08-14T21:25:27.5030960Z 6d89b5f065d5: Waiting 2025-08-14T21:25:27.5031110Z 5a5cc76ada43: Waiting 2025-08-14T21:25:27.5031260Z acf5babd87f2: Waiting 2025-08-14T21:25:27.5031411Z 7b92d7a4b8c7: Waiting 2025-08-14T21:25:27.5031558Z fc6b37d40530: Waiting 2025-08-14T21:25:27.5031701Z 4f4fb700ef54: Waiting 2025-08-14T21:25:27.5031844Z 2e1657907860: Waiting 2025-08-14T21:25:27.5953643Z c7b4a852a455: Verifying Checksum 2025-08-14T21:25:27.5953971Z c7b4a852a455: Download complete 2025-08-14T21:25:27.6767691Z 76a69b57b683: Verifying Checksum 2025-08-14T21:25:27.6768110Z 76a69b57b683: Download complete 2025-08-14T21:25:27.7555508Z 5c785dcb4cdb: Verifying Checksum 2025-08-14T21:25:27.7555839Z 5c785dcb4cdb: Download complete 2025-08-14T21:25:27.8377230Z 836ab08052e8: Verifying Checksum 2025-08-14T21:25:27.8377514Z 836ab08052e8: Download complete 2025-08-14T21:25:27.8560865Z 660ffc76f83b: Verifying Checksum 2025-08-14T21:25:27.8561652Z 660ffc76f83b: Download complete 2025-08-14T21:25:27.9134306Z 53b11c77468c: Verifying Checksum 2025-08-14T21:25:27.9134620Z 53b11c77468c: Download complete 2025-08-14T21:25:27.9334989Z e97311a6a967: Verifying Checksum 2025-08-14T21:25:27.9335302Z e97311a6a967: Download complete 2025-08-14T21:25:28.0094149Z 6d89b5f065d5: Download complete 2025-08-14T21:25:28.1171898Z 5a5cc76ada43: Download complete 2025-08-14T21:25:28.1867045Z fc6b37d40530: Download complete 2025-08-14T21:25:28.2644718Z 2e1657907860: Verifying Checksum 2025-08-14T21:25:28.2645028Z 2e1657907860: Download complete 2025-08-14T21:25:28.9929168Z 660ffc76f83b: Pull complete 2025-08-14T21:25:29.0140040Z c7b4a852a455: Pull complete 2025-08-14T21:25:29.0831184Z 2c414689d31d: Verifying Checksum 2025-08-14T21:25:29.0831524Z 2c414689d31d: Download complete 2025-08-14T21:25:29.0904359Z 4f4fb700ef54: Download complete 2025-08-14T21:25:29.2159624Z d6226eb61f82: Verifying Checksum 2025-08-14T21:25:29.2159910Z d6226eb61f82: Download complete 2025-08-14T21:25:29.3115277Z 83c70f4266a6: Verifying Checksum 2025-08-14T21:25:29.3115600Z 83c70f4266a6: Download complete 2025-08-14T21:25:29.4035061Z 60c725d21861: Verifying Checksum 2025-08-14T21:25:29.4035355Z 60c725d21861: Download complete 2025-08-14T21:25:29.4927948Z a504e76e66a4: Verifying Checksum 2025-08-14T21:25:29.4928405Z a504e76e66a4: Download complete 2025-08-14T21:25:29.5695195Z fc1c200a4f77: Verifying Checksum 2025-08-14T21:25:29.5700960Z fc1c200a4f77: Download complete 2025-08-14T21:25:29.6526802Z 43273c22704f: Verifying Checksum 2025-08-14T21:25:29.6527111Z 43273c22704f: Download complete 2025-08-14T21:25:29.8043714Z 89df389d042a: Verifying Checksum 2025-08-14T21:25:29.8044043Z 89df389d042a: Download complete 2025-08-14T21:25:29.9025484Z 684349f50d94: Verifying Checksum 2025-08-14T21:25:29.9723425Z 21d0eae87fb3: Verifying Checksum 2025-08-14T21:25:29.9723739Z 21d0eae87fb3: Download complete 2025-08-14T21:25:30.0514195Z c9c2b424b8e0: Download complete 2025-08-14T21:25:30.1331326Z 98dda28f3395: Verifying Checksum 2025-08-14T21:25:30.1331651Z 98dda28f3395: Download complete 2025-08-14T21:25:30.2187620Z acf5babd87f2: Verifying Checksum 2025-08-14T21:25:30.2187924Z acf5babd87f2: Download complete 2025-08-14T21:25:30.6966656Z e5a28988c893: Verifying Checksum 2025-08-14T21:25:30.6971463Z e5a28988c893: Download complete 2025-08-14T21:25:30.7897194Z 7ddd14e2b548: Verifying Checksum 2025-08-14T21:25:30.7897747Z 7ddd14e2b548: Download complete 2025-08-14T21:25:30.8949049Z 4ba8e7a736c8: Download complete 2025-08-14T21:25:31.0816857Z 907c320fee2f: Download complete 2025-08-14T21:25:31.1609489Z 18c4ed1ec491: Download complete 2025-08-14T21:25:31.4010329Z d7618c2df6cd: Verifying Checksum 2025-08-14T21:25:31.4010631Z d7618c2df6cd: Download complete 2025-08-14T21:25:31.5007619Z b7bdd9a6f789: Verifying Checksum 2025-08-14T21:25:31.5007968Z b7bdd9a6f789: Download complete 2025-08-14T21:25:31.5608305Z 6738ba83282e: Verifying Checksum 2025-08-14T21:25:31.5608844Z 6738ba83282e: Download complete 2025-08-14T21:25:31.6292175Z dfb0f2488639: Verifying Checksum 2025-08-14T21:25:31.7201867Z dfb0f2488639: Download complete 2025-08-14T21:25:31.7202149Z dc833b0762f2: Verifying Checksum 2025-08-14T21:25:31.7202355Z dc833b0762f2: Download complete 2025-08-14T21:25:31.7967008Z 8827df8ca2da: Verifying Checksum 2025-08-14T21:25:31.7967317Z 8827df8ca2da: Download complete 2025-08-14T21:25:34.8148453Z 7c5050d8408d: Verifying Checksum 2025-08-14T21:25:34.8150486Z 7c5050d8408d: Download complete 2025-08-14T21:25:34.9122838Z d7cf7f140df3: Download complete 2025-08-14T21:25:37.7054563Z 733eedc8da8d: Verifying Checksum 2025-08-14T21:25:37.7054928Z 733eedc8da8d: Download complete 2025-08-14T21:25:41.7344353Z e5a28988c893: Pull complete 2025-08-14T21:25:42.1684857Z 76a69b57b683: Pull complete 2025-08-14T21:25:42.4293293Z 5c785dcb4cdb: Pull complete 2025-08-14T21:25:42.8514406Z 836ab08052e8: Pull complete 2025-08-14T21:25:43.2230113Z 53b11c77468c: Pull complete 2025-08-14T21:25:43.5106097Z e97311a6a967: Pull complete 2025-08-14T21:25:47.0101280Z 2c414689d31d: Pull complete 2025-08-14T21:25:47.2738755Z 6d89b5f065d5: Pull complete 2025-08-14T21:25:47.5508789Z 5a5cc76ada43: Pull complete 2025-08-14T21:25:47.8522152Z fc6b37d40530: Pull complete 2025-08-14T21:25:48.1240294Z 2e1657907860: Pull complete 2025-08-14T21:26:02.3063454Z 7b92d7a4b8c7: Verifying Checksum 2025-08-14T21:26:02.3067221Z 7b92d7a4b8c7: Download complete 2025-08-14T21:26:02.4011578Z bc5961031092: Verifying Checksum 2025-08-14T21:26:02.4014375Z bc5961031092: Download complete 2025-08-14T21:26:02.4927389Z 0531cc34c12a: Download complete 2025-08-14T21:26:02.5999349Z 38c303d3b62e: Verifying Checksum 2025-08-14T21:26:02.5999651Z 38c303d3b62e: Download complete 2025-08-14T21:26:02.6965811Z e06d15594a2a: Verifying Checksum 2025-08-14T21:26:02.6966140Z e06d15594a2a: Download complete 2025-08-14T21:26:02.7818385Z 0e55deb5cb38: Verifying Checksum 2025-08-14T21:26:02.7818691Z 0e55deb5cb38: Download complete 2025-08-14T21:26:02.8692555Z 4a53d66dce07: Download complete 2025-08-14T21:26:02.9659679Z 1519daa051b8: Download complete 2025-08-14T21:26:03.0558042Z 381ed91d2119: Verifying Checksum 2025-08-14T21:26:03.0558331Z 381ed91d2119: Download complete 2025-08-14T21:26:03.1359922Z c6b0a01a96dd: Verifying Checksum 2025-08-14T21:26:03.1360397Z c6b0a01a96dd: Download complete 2025-08-14T21:26:03.2497895Z 62df6413daee: Verifying Checksum 2025-08-14T21:26:03.2498172Z 62df6413daee: Download complete 2025-08-14T21:26:03.3481314Z 7a18bc2a6881: Verifying Checksum 2025-08-14T21:26:03.3481623Z 7a18bc2a6881: Download complete 2025-08-14T21:26:03.4264900Z 93359cd58a8c: Verifying Checksum 2025-08-14T21:26:03.4265217Z 93359cd58a8c: Download complete 2025-08-14T21:26:03.5234063Z c35ba0a1f353: Download complete 2025-08-14T21:26:03.6126518Z dcf1e01c98d6: Verifying Checksum 2025-08-14T21:26:03.6127189Z dcf1e01c98d6: Download complete 2025-08-14T21:26:06.0934350Z bad0564f61fd: Verifying Checksum 2025-08-14T21:26:06.0934879Z bad0564f61fd: Download complete 2025-08-14T21:26:06.1885202Z 539ded905736: Download complete 2025-08-14T21:26:06.2816382Z 28d482062637: Download complete 2025-08-14T21:26:06.3516768Z 3245316ff51b: Verifying Checksum 2025-08-14T21:26:06.3517096Z 3245316ff51b: Download complete 2025-08-14T21:26:06.4700581Z b53167d1a6df: Verifying Checksum 2025-08-14T21:26:06.4700897Z b53167d1a6df: Download complete 2025-08-14T21:26:06.5599900Z 7f5277f69167: Verifying Checksum 2025-08-14T21:26:06.5600217Z 7f5277f69167: Download complete 2025-08-14T21:26:06.6413817Z 23dff10cdaa5: Download complete 2025-08-14T21:26:07.2362991Z 9fb73296da6a: Verifying Checksum 2025-08-14T21:26:07.2363440Z 9fb73296da6a: Download complete 2025-08-14T21:26:42.1235617Z 5b092eb06909: Verifying Checksum 2025-08-14T21:26:42.1235904Z 5b092eb06909: Download complete 2025-08-14T21:27:14.2899964Z 7b92d7a4b8c7: Pull complete 2025-08-14T21:27:14.3437094Z 4f4fb700ef54: Pull complete 2025-08-14T21:27:14.3900900Z d6226eb61f82: Pull complete 2025-08-14T21:27:14.4568188Z 83c70f4266a6: Pull complete 2025-08-14T21:27:14.6464993Z 60c725d21861: Pull complete 2025-08-14T21:27:15.0193270Z a504e76e66a4: Pull complete 2025-08-14T21:27:15.3320131Z fc1c200a4f77: Pull complete 2025-08-14T21:27:15.6641490Z 43273c22704f: Pull complete 2025-08-14T21:27:16.1121785Z 89df389d042a: Pull complete 2025-08-14T21:27:16.4843972Z 684349f50d94: Pull complete 2025-08-14T21:27:16.9860840Z 21d0eae87fb3: Pull complete 2025-08-14T21:27:17.2965264Z c9c2b424b8e0: Pull complete 2025-08-14T21:27:17.8552653Z 98dda28f3395: Pull complete 2025-08-14T21:27:17.9286873Z acf5babd87f2: Pull complete 2025-08-14T21:27:28.7557639Z 7c5050d8408d: Pull complete 2025-08-14T21:27:29.1973815Z 7ddd14e2b548: Pull complete 2025-08-14T21:27:29.6606685Z 4ba8e7a736c8: Pull complete 2025-08-14T21:27:30.5505229Z 907c320fee2f: Pull complete 2025-08-14T21:27:30.9683307Z 18c4ed1ec491: Pull complete 2025-08-14T21:27:31.6523646Z d7618c2df6cd: Pull complete 2025-08-14T21:27:31.8462375Z b7bdd9a6f789: Pull complete 2025-08-14T21:27:32.0562179Z 6738ba83282e: Pull complete 2025-08-14T21:27:32.8580643Z dfb0f2488639: Pull complete 2025-08-14T21:27:33.2337258Z dc833b0762f2: Pull complete 2025-08-14T21:27:33.6428117Z 8827df8ca2da: Pull complete 2025-08-14T21:28:28.5643833Z fac8f3bd0f85: Download complete 2025-08-14T21:32:23.1834565Z fac8f3bd0f85: Pull complete 2025-08-14T21:32:23.5373220Z d7cf7f140df3: Pull complete 2025-08-14T21:32:26.2594094Z 733eedc8da8d: Pull complete 2025-08-14T21:34:48.5224349Z 5b092eb06909: Pull complete 2025-08-14T21:34:48.5527437Z bc5961031092: Pull complete 2025-08-14T21:34:48.5834090Z 0531cc34c12a: Pull complete 2025-08-14T21:34:48.6364693Z 38c303d3b62e: Pull complete 2025-08-14T21:34:48.6846797Z e06d15594a2a: Pull complete 2025-08-14T21:34:48.7081472Z 0e55deb5cb38: Pull complete 2025-08-14T21:34:48.7567426Z 4a53d66dce07: Pull complete 2025-08-14T21:34:48.8045659Z 1519daa051b8: Pull complete 2025-08-14T21:34:48.8329513Z 381ed91d2119: Pull complete 2025-08-14T21:34:48.8836096Z c6b0a01a96dd: Pull complete 2025-08-14T21:34:48.9087072Z 62df6413daee: Pull complete 2025-08-14T21:34:48.9603454Z 7a18bc2a6881: Pull complete 2025-08-14T21:34:48.9849071Z 93359cd58a8c: Pull complete 2025-08-14T21:34:49.0310896Z c35ba0a1f353: Pull complete 2025-08-14T21:34:49.0563931Z dcf1e01c98d6: Pull complete 2025-08-14T21:34:57.9728918Z bad0564f61fd: Pull complete 2025-08-14T21:34:58.2165095Z 539ded905736: Pull complete 2025-08-14T21:34:58.6693107Z 28d482062637: Pull complete 2025-08-14T21:34:59.0192290Z 3245316ff51b: Pull complete 2025-08-14T21:34:59.3761945Z b53167d1a6df: Pull complete 2025-08-14T21:34:59.9081222Z 7f5277f69167: Pull complete 2025-08-14T21:35:00.7884464Z 23dff10cdaa5: Pull complete 2025-08-14T21:35:03.7048765Z 9fb73296da6a: Pull complete 2025-08-14T21:35:04.3613424Z Digest: sha256:4236794baba289041d240d08fd393bbd57497c3012e5e0ccd9fd98f61ebf35c6 2025-08-14T21:35:04.4518655Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:35:04.5005234Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:35:04.5059327Z ##[group]Run echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-08-14T21:35:04.5059957Z echo "IN_CONTAINER_RUNNER=$(if [ -f /.inarc ] || [ -f /.incontainer ]; then echo true ; else echo false; fi)" >> "$GITHUB_OUTPUT" 2025-08-14T21:35:04.5068451Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:35:04.5068685Z env: 2025-08-14T21:35:04.5068852Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:35:04.5069033Z ##[endgroup] 2025-08-14T21:35:04.5147080Z Prepare all required actions 2025-08-14T21:35:04.5209120Z ##[group]Run ./.github/actions/get-workflow-job-id 2025-08-14T21:35:04.5209370Z with: 2025-08-14T21:35:04.5209915Z github-token: *** 2025-08-14T21:35:04.5210076Z env: 2025-08-14T21:35:04.5210241Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:35:04.5210426Z ##[endgroup] 2025-08-14T21:35:04.5334971Z ##[group]Run set -eux 2025-08-14T21:35:04.5335189Z set -eux 2025-08-14T21:35:04.5335478Z python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-08-14T21:35:04.5340230Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:35:04.5340477Z env: 2025-08-14T21:35:04.5340639Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:35:04.5341021Z GITHUB_TOKEN: *** 2025-08-14T21:35:04.5341176Z ##[endgroup] 2025-08-14T21:35:04.5361711Z + python3 .github/scripts/get_workflow_job_id.py 16976338999 i-0851ccaad4f014969 2025-08-14T21:35:05.9730012Z Setting output job-id=48128301909 2025-08-14T21:35:05.9730776Z Setting output job-name=linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T21:35:05.9862097Z ##[group]Run python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-08-14T21:35:05.9862552Z python3 -m pip install psutil==5.9.8 dataclasses_json==0.6.7 nvidia-ml-py==11.525.84 2025-08-14T21:35:05.9863089Z python3 -m tools.stats.monitor --log-interval "$MONITOR_LOG_INTERVAL" --data-collect-interval "$MONITOR_DATA_COLLECT_INTERVAL" > usage_log.txt 2>&1 & 2025-08-14T21:35:05.9863564Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}" 2025-08-14T21:35:05.9868167Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:35:05.9868394Z env: 2025-08-14T21:35:05.9868545Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:35:05.9868708Z JOB_ID: 48128301909 2025-08-14T21:35:05.9869091Z JOB_NAME: linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T21:35:05.9869503Z WORKFLOW_NAME: inductor-periodic 2025-08-14T21:35:05.9869743Z WORKFLOW_RUN_ID: 16976338999 2025-08-14T21:35:05.9869914Z MONITOR_LOG_INTERVAL: 5 2025-08-14T21:35:05.9870089Z MONITOR_DATA_COLLECT_INTERVAL: 1 2025-08-14T21:35:05.9870271Z ##[endgroup] 2025-08-14T21:35:06.4804354Z Defaulting to user installation because normal site-packages is not writeable 2025-08-14T21:35:06.7532857Z Collecting psutil==5.9.8 2025-08-14T21:35:06.7688259Z Downloading psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (288 kB) 2025-08-14T21:35:06.8820006Z Collecting dataclasses_json==0.6.7 2025-08-14T21:35:06.8862509Z Downloading dataclasses_json-0.6.7-py3-none-any.whl (28 kB) 2025-08-14T21:35:06.9355536Z Collecting nvidia-ml-py==11.525.84 2025-08-14T21:35:06.9397766Z Downloading nvidia_ml_py-11.525.84-py3-none-any.whl (34 kB) 2025-08-14T21:35:07.0474510Z Collecting marshmallow<4.0.0,>=3.18.0 2025-08-14T21:35:07.0512576Z Downloading marshmallow-3.26.1-py3-none-any.whl (50 kB) 2025-08-14T21:35:07.1241568Z Collecting typing-inspect<1,>=0.4.0 2025-08-14T21:35:07.1277248Z Downloading typing_inspect-0.9.0-py3-none-any.whl (8.8 kB) 2025-08-14T21:35:07.2545152Z Collecting packaging>=17.0 2025-08-14T21:35:07.2578762Z Downloading packaging-25.0-py3-none-any.whl (66 kB) 2025-08-14T21:35:07.3922403Z Collecting typing-extensions>=3.7.4 2025-08-14T21:35:07.3966565Z Downloading typing_extensions-4.14.1-py3-none-any.whl (43 kB) 2025-08-14T21:35:07.4935979Z Collecting mypy-extensions>=0.3.0 2025-08-14T21:35:07.4977998Z Downloading mypy_extensions-1.1.0-py3-none-any.whl (5.0 kB) 2025-08-14T21:35:07.7081387Z Installing collected packages: typing-extensions, packaging, mypy-extensions, typing-inspect, marshmallow, psutil, nvidia-ml-py, dataclasses-json 2025-08-14T21:35:08.2729669Z Successfully installed dataclasses-json-0.6.7 marshmallow-3.26.1 mypy-extensions-1.1.0 nvidia-ml-py-11.525.84 packaging-25.0 psutil-5.9.8 typing-extensions-4.14.1 typing-inspect-0.9.0 2025-08-14T21:35:08.4893680Z Prepare all required actions 2025-08-14T21:35:08.4893961Z Getting action download info 2025-08-14T21:35:08.6266409Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:1da556a7aa0a088e3153970611f6c432d58e80e6) 2025-08-14T21:35:09.0304314Z Download action repository 'actions/download-artifact@v4' (SHA:d3f86a106a0bac45b974a628896c90dbdf5c8093) 2025-08-14T21:35:11.7461101Z ##[group]Run ./.github/actions/download-build-artifacts 2025-08-14T21:35:11.7461331Z with: 2025-08-14T21:35:11.7461495Z name: linux-jammy-py3.9-gcc11-build 2025-08-14T21:35:11.7461684Z s3-bucket: gha-artifacts 2025-08-14T21:35:11.7461852Z env: 2025-08-14T21:35:11.7461998Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:35:11.7462152Z ##[endgroup] 2025-08-14T21:35:11.7490814Z ##[group]Run seemethere/download-artifact-s3@v4 2025-08-14T21:35:11.7491034Z with: 2025-08-14T21:35:11.7491201Z name: linux-jammy-py3.9-gcc11-build 2025-08-14T21:35:11.7491399Z s3-bucket: gha-artifacts 2025-08-14T21:35:11.7491616Z region: us-east-1 2025-08-14T21:35:11.7491775Z env: 2025-08-14T21:35:11.7491920Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:35:11.7492083Z ##[endgroup] 2025-08-14T21:35:12.4611400Z (node:47830) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-08-14T21:35:12.4611793Z 2025-08-14T21:35:12.4611940Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-08-14T21:35:12.4612336Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-08-14T21:35:12.4612728Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-08-14T21:35:13.3640752Z Found 1 objects with prefix pytorch/pytorch/16976338999/linux-jammy-py3.9-gcc11-build/ 2025-08-14T21:35:13.3641488Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-08-14T21:35:20.4003724Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip 2025-08-14T21:35:20.4012784Z Artifact download has finished successfully 2025-08-14T21:35:20.4205074Z ##[group]Run unzip -o artifacts.zip 2025-08-14T21:35:20.4205323Z unzip -o artifacts.zip 2025-08-14T21:35:20.4211099Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:35:20.4211349Z env: 2025-08-14T21:35:20.4211515Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:35:20.4211705Z ##[endgroup] 2025-08-14T21:35:20.4287566Z Archive: artifacts.zip 2025-08-14T21:35:20.4287838Z creating: dist/ 2025-08-14T21:35:21.5661935Z inflating: dist/torch-2.9.0a0+git1fc683c-cp39-cp39-linux_x86_64.whl 2025-08-14T21:35:21.5665116Z creating: dist/vision/ 2025-08-14T21:35:21.5742880Z inflating: dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl 2025-08-14T21:35:21.5743264Z creating: dist/audio/ 2025-08-14T21:35:21.5846599Z inflating: dist/audio/torchaudio-2.8.0a0+bdb88e1-cp39-cp39-linux_x86_64.whl 2025-08-14T21:35:21.5853297Z creating: dist/ao/ 2025-08-14T21:35:21.5890875Z inflating: dist/ao/torchao-0.7.0+git51c87b6e-py3-none-any.whl 2025-08-14T21:35:21.6003289Z inflating: dist/.ninja_log 2025-08-14T21:35:21.6003604Z creating: build/custom_test_artifacts/ 2025-08-14T21:35:21.6003897Z creating: build/custom_test_artifacts/custom-op-build/ 2025-08-14T21:35:21.6004245Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/ 2025-08-14T21:35:21.6004632Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/pkgRedirects/ 2025-08-14T21:35:21.6007585Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeConfigureLog.yaml 2025-08-14T21:35:21.6008046Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/ 2025-08-14T21:35:21.6008451Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeSystem.cmake 2025-08-14T21:35:21.6008921Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/ 2025-08-14T21:35:21.6009805Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/tmp/ 2025-08-14T21:35:21.6010565Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c 2025-08-14T21:35:21.6012083Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdC/a.out 2025-08-14T21:35:21.6012846Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake 2025-08-14T21:35:21.6013303Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/ 2025-08-14T21:35:21.6013764Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/ 2025-08-14T21:35:21.6018526Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-08-14T21:35:21.6019193Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out 2025-08-14T21:35:21.6019749Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake 2025-08-14T21:35:21.6020293Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin 2025-08-14T21:35:21.6020812Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin 2025-08-14T21:35:21.6021267Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeScratch/ 2025-08-14T21:35:21.6021658Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache 2025-08-14T21:35:21.6022120Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/ 2025-08-14T21:35:21.6022576Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts 2025-08-14T21:35:21.6023084Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make 2025-08-14T21:35:21.6023580Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make 2025-08-14T21:35:21.6024027Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt 2025-08-14T21:35:21.6024465Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake 2025-08-14T21:35:21.6024908Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make 2025-08-14T21:35:21.6025344Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake 2025-08-14T21:35:21.6025800Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make 2025-08-14T21:35:21.6026241Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make 2025-08-14T21:35:21.6042409Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d 2025-08-14T21:35:21.6209958Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o 2025-08-14T21:35:21.6213718Z creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/ 2025-08-14T21:35:21.6214228Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts 2025-08-14T21:35:21.6214746Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make 2025-08-14T21:35:21.6215273Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make 2025-08-14T21:35:21.6215763Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt 2025-08-14T21:35:21.6216274Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake 2025-08-14T21:35:21.6216766Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make 2025-08-14T21:35:21.6217380Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake 2025-08-14T21:35:21.6217863Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make 2025-08-14T21:35:21.6218325Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make 2025-08-14T21:35:21.6231187Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d 2025-08-14T21:35:21.6302981Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o 2025-08-14T21:35:21.6306709Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-08-14T21:35:21.6307242Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt 2025-08-14T21:35:21.6307697Z extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks 2025-08-14T21:35:21.6308128Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2 2025-08-14T21:35:21.6308538Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake 2025-08-14T21:35:21.6308962Z inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/InstallScripts.json 2025-08-14T21:35:21.6309366Z inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt 2025-08-14T21:35:21.6309710Z inflating: build/custom_test_artifacts/custom-op-build/Makefile 2025-08-14T21:35:21.6310073Z inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake 2025-08-14T21:35:21.6465164Z inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so 2025-08-14T21:35:21.6513891Z inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops 2025-08-14T21:35:21.6514294Z creating: build/custom_test_artifacts/jit-hook-build/ 2025-08-14T21:35:21.6514623Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/ 2025-08-14T21:35:21.6515046Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/pkgRedirects/ 2025-08-14T21:35:21.6520298Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeConfigureLog.yaml 2025-08-14T21:35:21.6521462Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/ 2025-08-14T21:35:21.6521941Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeSystem.cmake 2025-08-14T21:35:21.6522418Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/ 2025-08-14T21:35:21.6522964Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/tmp/ 2025-08-14T21:35:21.6523457Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c 2025-08-14T21:35:21.6523971Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdC/a.out 2025-08-14T21:35:21.6524441Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake 2025-08-14T21:35:21.6524921Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/ 2025-08-14T21:35:21.6525903Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/ 2025-08-14T21:35:21.6526457Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-08-14T21:35:21.6526989Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out 2025-08-14T21:35:21.6527479Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake 2025-08-14T21:35:21.6528173Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin 2025-08-14T21:35:21.6528867Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin 2025-08-14T21:35:21.6529496Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeScratch/ 2025-08-14T21:35:21.6530173Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache 2025-08-14T21:35:21.6530807Z creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/ 2025-08-14T21:35:21.6531360Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts 2025-08-14T21:35:21.6531976Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make 2025-08-14T21:35:21.6532507Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make 2025-08-14T21:35:21.6532991Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt 2025-08-14T21:35:21.6533502Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake 2025-08-14T21:35:21.6534032Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make 2025-08-14T21:35:21.6534560Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake 2025-08-14T21:35:21.6535075Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make 2025-08-14T21:35:21.6535597Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make 2025-08-14T21:35:21.6553586Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d 2025-08-14T21:35:21.6612771Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o 2025-08-14T21:35:21.6613385Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-08-14T21:35:21.6613882Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt 2025-08-14T21:35:21.6614339Z extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks 2025-08-14T21:35:21.6614786Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2 2025-08-14T21:35:21.6615209Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake 2025-08-14T21:35:21.6615632Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/InstallScripts.json 2025-08-14T21:35:21.6616435Z inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt 2025-08-14T21:35:21.6616840Z inflating: build/custom_test_artifacts/jit-hook-build/Makefile 2025-08-14T21:35:21.6617256Z inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake 2025-08-14T21:35:21.6656440Z inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks 2025-08-14T21:35:21.6656839Z creating: build/custom_test_artifacts/custom-backend-build/ 2025-08-14T21:35:21.6657241Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/ 2025-08-14T21:35:21.6657671Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/pkgRedirects/ 2025-08-14T21:35:21.6663098Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeConfigureLog.yaml 2025-08-14T21:35:21.6667999Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/ 2025-08-14T21:35:21.6673485Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeSystem.cmake 2025-08-14T21:35:21.6675256Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/ 2025-08-14T21:35:21.6675839Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/tmp/ 2025-08-14T21:35:21.6682189Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/CMakeCCompilerId.c 2025-08-14T21:35:21.6687612Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdC/a.out 2025-08-14T21:35:21.6688592Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeCCompiler.cmake 2025-08-14T21:35:21.6689242Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/ 2025-08-14T21:35:21.6689853Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/tmp/ 2025-08-14T21:35:21.6690533Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/CMakeCXXCompilerId.cpp 2025-08-14T21:35:21.6691097Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CompilerIdCXX/a.out 2025-08-14T21:35:21.6691631Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeCXXCompiler.cmake 2025-08-14T21:35:21.6692183Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_C.bin 2025-08-14T21:35:21.6692769Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/4.0.0/CMakeDetermineCompilerABI_CXX.bin 2025-08-14T21:35:21.6693279Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeScratch/ 2025-08-14T21:35:21.6693726Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache 2025-08-14T21:35:21.6694183Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/ 2025-08-14T21:35:21.6694682Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts 2025-08-14T21:35:21.6695248Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make 2025-08-14T21:35:21.6695789Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make 2025-08-14T21:35:21.6696293Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt 2025-08-14T21:35:21.6696810Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake 2025-08-14T21:35:21.6697343Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make 2025-08-14T21:35:21.6697877Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake 2025-08-14T21:35:21.6698407Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make 2025-08-14T21:35:21.6698920Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make 2025-08-14T21:35:21.6699477Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d 2025-08-14T21:35:21.6793520Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o 2025-08-14T21:35:21.6796373Z creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/ 2025-08-14T21:35:21.6801400Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts 2025-08-14T21:35:21.6802479Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make 2025-08-14T21:35:21.6803034Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make 2025-08-14T21:35:21.6803563Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt 2025-08-14T21:35:21.6804099Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake 2025-08-14T21:35:21.6804639Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make 2025-08-14T21:35:21.6805177Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake 2025-08-14T21:35:21.6806222Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make 2025-08-14T21:35:21.6806830Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make 2025-08-14T21:35:21.6814113Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d 2025-08-14T21:35:21.6860774Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o 2025-08-14T21:35:21.6862427Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake 2025-08-14T21:35:21.6862967Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt 2025-08-14T21:35:21.6863452Z extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks 2025-08-14T21:35:21.6864006Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2 2025-08-14T21:35:21.6864478Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake 2025-08-14T21:35:21.6864930Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/InstallScripts.json 2025-08-14T21:35:21.6867563Z inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt 2025-08-14T21:35:21.6868108Z inflating: build/custom_test_artifacts/custom-backend-build/Makefile 2025-08-14T21:35:21.6871752Z inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake 2025-08-14T21:35:21.6959983Z inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so 2025-08-14T21:35:21.6995358Z inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend 2025-08-14T21:35:21.6995899Z creating: build/lib/ 2025-08-14T21:35:21.7073163Z inflating: build/lib/libprotobuf-lite.a 2025-08-14T21:35:21.7492082Z inflating: build/lib/libprotobuf.a 2025-08-14T21:35:21.7953573Z inflating: build/lib/libprotoc.a 2025-08-14T21:35:21.7964310Z inflating: build/lib/libpthreadpool.a 2025-08-14T21:35:21.7972719Z inflating: build/lib/libcpuinfo.a 2025-08-14T21:35:21.7981178Z inflating: build/lib/libcpuinfo_internals.a 2025-08-14T21:35:21.7981618Z inflating: build/lib/libclog.a 2025-08-14T21:35:21.7995709Z inflating: build/lib/libpytorch_qnnpack.a 2025-08-14T21:35:21.7996168Z inflating: build/lib/libnnpack_reference_layers.a 2025-08-14T21:35:21.8165968Z inflating: build/lib/libmicrokernels-prod.a 2025-08-14T21:35:21.8183629Z inflating: build/lib/libnnpack.a 2025-08-14T21:35:21.8973523Z inflating: build/lib/libmicrokernels-all.a 2025-08-14T21:35:21.9036073Z inflating: build/lib/libgtest.a 2025-08-14T21:35:21.9052329Z inflating: build/lib/libgmock.a 2025-08-14T21:35:21.9057300Z inflating: build/lib/libgmock_main.a 2025-08-14T21:35:21.9062028Z inflating: build/lib/libgtest_main.a 2025-08-14T21:35:21.9133502Z inflating: build/lib/libXNNPACK.a 2025-08-14T21:35:21.9203379Z inflating: build/lib/libbenchmark.a 2025-08-14T21:35:21.9205736Z inflating: build/lib/libbenchmark_main.a 2025-08-14T21:35:21.9206061Z inflating: build/lib/libjitprofiling.a 2025-08-14T21:35:21.9265205Z inflating: build/lib/libasmjit.a 2025-08-14T21:35:21.9272854Z inflating: build/lib/libittnotify.a 2025-08-14T21:35:22.0312093Z inflating: build/lib/libfbgemm.a 2025-08-14T21:35:22.0339413Z inflating: build/lib/libtensorpipe_uv.a 2025-08-14T21:35:22.0825334Z inflating: build/lib/libtensorpipe.a 2025-08-14T21:35:22.0930919Z inflating: build/lib/libgloo.a 2025-08-14T21:35:22.0976828Z inflating: build/lib/libonnx_proto.a 2025-08-14T21:35:22.1606434Z inflating: build/lib/libonnx.a 2025-08-14T21:35:23.0592766Z inflating: build/lib/libdnnl.a 2025-08-14T21:35:23.0605880Z inflating: build/lib/libfmt.a 2025-08-14T21:35:23.0850116Z inflating: build/lib/libkineto.a 2025-08-14T21:35:23.0951749Z inflating: build/lib/libc10.so 2025-08-14T21:35:23.0957045Z inflating: build/lib/libtorch_global_deps.so 2025-08-14T21:35:25.8348248Z inflating: build/lib/libtorch_cpu.so 2025-08-14T21:35:25.8348754Z inflating: build/lib/libtorch.so 2025-08-14T21:35:25.8411819Z inflating: build/lib/libtorchbind_test.so 2025-08-14T21:35:25.8429924Z inflating: build/lib/libjitbackend_test.so 2025-08-14T21:35:25.8450480Z inflating: build/lib/libbackend_with_compiler.so 2025-08-14T21:35:25.8471597Z inflating: build/lib/libaoti_custom_ops.so 2025-08-14T21:35:25.8475719Z inflating: build/lib/libshm.so 2025-08-14T21:35:26.0261528Z inflating: build/lib/libtorch_python.so 2025-08-14T21:35:26.0292366Z inflating: build/lib/libnnapi_backend.so 2025-08-14T21:35:26.0292790Z creating: build/bin/ 2025-08-14T21:35:26.0293080Z creating: build/bin/CMakeFiles/ 2025-08-14T21:35:26.0293405Z inflating: build/bin/cmake_install.cmake 2025-08-14T21:35:26.0293741Z inflating: build/bin/CTestTestfile.cmake 2025-08-14T21:35:26.0709008Z inflating: build/bin/protoc-3.13.0.0 2025-08-14T21:35:26.1135097Z inflating: build/bin/protoc 2025-08-14T21:35:26.1190175Z inflating: build/bin/c10_AllocatorConfig_test 2025-08-14T21:35:26.1242333Z inflating: build/bin/c10_CompileTimeFunctionPointer_test 2025-08-14T21:35:26.1297592Z inflating: build/bin/c10_DeviceGuard_test 2025-08-14T21:35:26.1353324Z inflating: build/bin/c10_Device_test 2025-08-14T21:35:26.1400682Z inflating: build/bin/c10_StreamGuard_test 2025-08-14T21:35:26.1461029Z inflating: build/bin/c10_DispatchKeySet_test 2025-08-14T21:35:26.1509329Z inflating: build/bin/c10_SymInt_test 2025-08-14T21:35:26.1563829Z inflating: build/bin/c10_Scalar_test 2025-08-14T21:35:26.1623285Z inflating: build/bin/c10_InlineDeviceGuard_test 2025-08-14T21:35:26.1678086Z inflating: build/bin/c10_InlineStreamGuard_test 2025-08-14T21:35:26.1733173Z inflating: build/bin/c10_SizesAndStrides_test 2025-08-14T21:35:26.1785814Z inflating: build/bin/c10_Bitset_test 2025-08-14T21:35:26.1856345Z inflating: build/bin/c10_cow_test 2025-08-14T21:35:26.1904058Z inflating: build/bin/c10_ArrayRef_test 2025-08-14T21:35:26.1954155Z inflating: build/bin/c10_ConstexprCrc_test 2025-08-14T21:35:26.2004367Z inflating: build/bin/c10_DeadlockDetection_test 2025-08-14T21:35:26.2059758Z inflating: build/bin/c10_Enumerate_test 2025-08-14T21:35:26.2111919Z inflating: build/bin/c10_Half_test 2025-08-14T21:35:26.2163386Z inflating: build/bin/c10_IntrusiveList_test 2025-08-14T21:35:26.2221394Z inflating: build/bin/c10_LeftRight_test 2025-08-14T21:35:26.2278277Z inflating: build/bin/c10_Metaprogramming_test 2025-08-14T21:35:26.2332216Z inflating: build/bin/c10_NetworkFlow_test 2025-08-14T21:35:26.2383642Z inflating: build/bin/c10_Synchronized_test 2025-08-14T21:35:26.2432384Z inflating: build/bin/c10_Semaphore_test 2025-08-14T21:35:26.2482488Z inflating: build/bin/c10_TypeIndex_test 2025-08-14T21:35:26.2536309Z inflating: build/bin/c10_ThreadLocal_test 2025-08-14T21:35:26.2588043Z inflating: build/bin/c10_TypeList_test 2025-08-14T21:35:26.2638898Z inflating: build/bin/c10_TypeTraits_test 2025-08-14T21:35:26.2691113Z inflating: build/bin/c10_accumulate_test 2025-08-14T21:35:26.2746253Z inflating: build/bin/c10_bfloat16_test 2025-08-14T21:35:26.2799016Z inflating: build/bin/c10_complex_test 2025-08-14T21:35:26.2857299Z inflating: build/bin/c10_complex_math_test 2025-08-14T21:35:26.2904015Z inflating: build/bin/c10_bit_cast_test 2025-08-14T21:35:26.2954434Z inflating: build/bin/c10_error_test 2025-08-14T21:35:26.3006031Z inflating: build/bin/c10_exception_test 2025-08-14T21:35:26.3057703Z inflating: build/bin/c10_flags_test 2025-08-14T21:35:26.3108463Z inflating: build/bin/c10_irange_test 2025-08-14T21:35:26.3159613Z inflating: build/bin/c10_generic_math_test 2025-08-14T21:35:26.3314159Z inflating: build/bin/c10_intrusive_ptr_test 2025-08-14T21:35:26.3367066Z inflating: build/bin/c10_lazy_test 2025-08-14T21:35:26.3426621Z inflating: build/bin/c10_logging_test 2025-08-14T21:35:26.3484976Z inflating: build/bin/c10_ordered_preserving_dict_test 2025-08-14T21:35:26.3559434Z inflating: build/bin/c10_optional_test 2025-08-14T21:35:26.3611776Z inflating: build/bin/c10_registry_test 2025-08-14T21:35:26.3760329Z inflating: build/bin/c10_small_vector_test 2025-08-14T21:35:26.3811121Z inflating: build/bin/c10_string_util_test 2025-08-14T21:35:26.3863655Z inflating: build/bin/c10_ssize_test 2025-08-14T21:35:26.3910701Z inflating: build/bin/c10_string_view_test 2025-08-14T21:35:26.3961631Z inflating: build/bin/c10_tempfile_test 2025-08-14T21:35:26.4014903Z inflating: build/bin/c10_typeid_test 2025-08-14T21:35:26.4059409Z inflating: build/bin/c10_intrusive_ptr_benchmark 2025-08-14T21:35:26.4581175Z inflating: build/bin/vec_test_all_types_DEFAULT 2025-08-14T21:35:26.5120856Z inflating: build/bin/vec_test_all_types_AVX512 2025-08-14T21:35:26.5664676Z inflating: build/bin/vec_test_all_types_AVX2 2025-08-14T21:35:26.5717135Z inflating: build/bin/static_runtime_bench 2025-08-14T21:35:26.5944497Z inflating: build/bin/static_runtime_test 2025-08-14T21:35:26.6015446Z inflating: build/bin/Dict_test 2025-08-14T21:35:26.6068722Z inflating: build/bin/Dimname_test 2025-08-14T21:35:26.6130952Z inflating: build/bin/MaybeOwned_test 2025-08-14T21:35:26.6185570Z inflating: build/bin/NamedTensor_test 2025-08-14T21:35:26.6240122Z inflating: build/bin/apply_utils_test 2025-08-14T21:35:26.6300986Z inflating: build/bin/atest 2025-08-14T21:35:26.6357672Z inflating: build/bin/basic 2025-08-14T21:35:26.6410946Z inflating: build/bin/broadcast_test 2025-08-14T21:35:26.6460994Z inflating: build/bin/cpu_allocator_test 2025-08-14T21:35:26.6517446Z inflating: build/bin/cpu_generator_test 2025-08-14T21:35:26.6569785Z inflating: build/bin/cpu_profiling_allocator_test 2025-08-14T21:35:26.6655357Z inflating: build/bin/cpu_rng_test 2025-08-14T21:35:26.6704932Z inflating: build/bin/dlconvertor_test 2025-08-14T21:35:26.6760679Z inflating: build/bin/extension_backend_test 2025-08-14T21:35:26.6814723Z inflating: build/bin/half_test 2025-08-14T21:35:26.6906098Z inflating: build/bin/ivalue_test 2025-08-14T21:35:26.6956138Z inflating: build/bin/lazy_tensor_test 2025-08-14T21:35:26.7006872Z inflating: build/bin/math_kernel_test 2025-08-14T21:35:26.7062168Z inflating: build/bin/memory_format_test 2025-08-14T21:35:26.7114124Z inflating: build/bin/memory_overlapping_test 2025-08-14T21:35:26.7167861Z inflating: build/bin/mobile_memory_cleanup 2025-08-14T21:35:26.7221288Z inflating: build/bin/native_test 2025-08-14T21:35:26.7269796Z inflating: build/bin/operator_name_test 2025-08-14T21:35:26.7320367Z inflating: build/bin/operators_test 2025-08-14T21:35:26.7370750Z inflating: build/bin/packedtensoraccessor_test 2025-08-14T21:35:26.7434642Z inflating: build/bin/pow_test 2025-08-14T21:35:26.7492061Z inflating: build/bin/quantized_test 2025-08-14T21:35:26.7540388Z inflating: build/bin/reduce_ops_test 2025-08-14T21:35:26.7592814Z inflating: build/bin/reportMemoryUsage_test 2025-08-14T21:35:26.7649114Z inflating: build/bin/scalar_tensor_test 2025-08-14T21:35:26.7708766Z inflating: build/bin/scalar_test 2025-08-14T21:35:26.7760831Z inflating: build/bin/StorageUtils_test 2025-08-14T21:35:26.7815583Z inflating: build/bin/stride_properties_test 2025-08-14T21:35:26.7894951Z inflating: build/bin/tensor_iterator_test 2025-08-14T21:35:26.7947092Z inflating: build/bin/test_parallel 2025-08-14T21:35:26.7999980Z inflating: build/bin/thread_init_test 2025-08-14T21:35:26.8056360Z inflating: build/bin/type_ptr_test 2025-08-14T21:35:26.8115929Z inflating: build/bin/type_test 2025-08-14T21:35:26.8169937Z inflating: build/bin/undefined_tensor_test 2025-08-14T21:35:26.8218637Z inflating: build/bin/verify_api_visibility 2025-08-14T21:35:26.8289050Z inflating: build/bin/legacy_vmap_test 2025-08-14T21:35:26.8337341Z inflating: build/bin/weakref_test 2025-08-14T21:35:26.8391014Z inflating: build/bin/wrapdim_test 2025-08-14T21:35:26.8444309Z inflating: build/bin/xla_tensor_test 2025-08-14T21:35:26.8503302Z inflating: build/bin/IListRef_test 2025-08-14T21:35:26.8606007Z inflating: build/bin/List_test 2025-08-14T21:35:26.8673775Z inflating: build/bin/KernelFunction_test 2025-08-14T21:35:26.8788566Z inflating: build/bin/kernel_function_legacy_test 2025-08-14T21:35:26.8882546Z inflating: build/bin/kernel_function_test 2025-08-14T21:35:26.8997457Z inflating: build/bin/kernel_lambda_legacy_test 2025-08-14T21:35:26.9096147Z inflating: build/bin/kernel_lambda_test 2025-08-14T21:35:26.9155576Z inflating: build/bin/kernel_stackbased_test 2025-08-14T21:35:26.9244818Z inflating: build/bin/make_boxed_from_unboxed_functor_test 2025-08-14T21:35:26.9296762Z inflating: build/bin/CppSignature_test 2025-08-14T21:35:26.9348497Z inflating: build/bin/backend_fallback_test 2025-08-14T21:35:26.9398333Z inflating: build/bin/op_allowlist_test 2025-08-14T21:35:26.9674714Z inflating: build/bin/op_registration_test 2025-08-14T21:35:26.9737004Z inflating: build/bin/inline_container_test 2025-08-14T21:35:27.0740444Z inflating: build/bin/test_jit 2025-08-14T21:35:27.1031959Z inflating: build/bin/test_nativert 2025-08-14T21:35:27.1082427Z inflating: build/bin/BackoffTest 2025-08-14T21:35:27.1138921Z inflating: build/bin/FileStoreTest 2025-08-14T21:35:27.1194972Z inflating: build/bin/TCPStoreTest 2025-08-14T21:35:27.1246852Z inflating: build/bin/HashStoreTest 2025-08-14T21:35:27.1311291Z inflating: build/bin/ProcessGroupGlooTest 2025-08-14T21:35:27.1316490Z inflating: build/bin/example_allreduce 2025-08-14T21:35:27.1371796Z inflating: build/bin/test_dist_autograd 2025-08-14T21:35:27.1437145Z inflating: build/bin/test_cpp_rpc 2025-08-14T21:35:27.2449715Z inflating: build/bin/test_api 2025-08-14T21:35:27.2455077Z inflating: build/bin/parallel_benchmark 2025-08-14T21:35:27.2765497Z inflating: build/bin/test_lazy 2025-08-14T21:35:27.2766019Z inflating: build/bin/torch_shm_manager 2025-08-14T21:35:27.2766957Z creating: .additional_ci_files/ 2025-08-14T21:35:27.2839561Z inflating: .additional_ci_files/test-times.json 2025-08-14T21:35:27.3102335Z inflating: .additional_ci_files/test-class-times.json 2025-08-14T21:35:27.3136705Z ##[group]Run rm artifacts.zip 2025-08-14T21:35:27.3136923Z rm artifacts.zip 2025-08-14T21:35:27.3141844Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:35:27.3142147Z env: 2025-08-14T21:35:27.3142298Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:35:27.3142466Z ##[endgroup] 2025-08-14T21:35:27.4110883Z ##[group]Run df -H 2025-08-14T21:35:27.4111066Z df -H 2025-08-14T21:35:27.4115547Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:35:27.4115777Z env: 2025-08-14T21:35:27.4115949Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:35:27.4116269Z ##[endgroup] 2025-08-14T21:35:27.4156160Z Filesystem Size Used Avail Use% Mounted on 2025-08-14T21:35:27.4162610Z devtmpfs 4.2M 0 4.2M 0% /dev 2025-08-14T21:35:27.4162867Z tmpfs 67G 0 67G 0% /dev/shm 2025-08-14T21:35:27.4163135Z tmpfs 27G 791k 27G 1% /run 2025-08-14T21:35:27.4163341Z /dev/nvme0n1p1 215G 69G 147G 32% / 2025-08-14T21:35:27.4163545Z tmpfs 67G 13k 67G 1% /tmp 2025-08-14T21:35:27.4163748Z /dev/nvme0n1p128 11M 1.4M 9.2M 13% /boot/efi 2025-08-14T21:35:27.4184478Z Prepare all required actions 2025-08-14T21:35:27.4185364Z Getting action download info 2025-08-14T21:35:27.5558214Z ##[group]Run ./.github/actions/download-td-artifacts 2025-08-14T21:35:27.5558493Z with: 2025-08-14T21:35:27.5558660Z env: 2025-08-14T21:35:27.5558818Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:35:27.5558998Z ##[endgroup] 2025-08-14T21:35:28.5694994Z ##[group]Run seemethere/download-artifact-s3@v4 2025-08-14T21:35:28.5695253Z with: 2025-08-14T21:35:28.5695404Z name: td_results 2025-08-14T21:35:28.5695570Z s3-bucket: gha-artifacts 2025-08-14T21:35:28.5695737Z region: us-east-1 2025-08-14T21:35:28.5695889Z env: 2025-08-14T21:35:28.5696041Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:35:28.5696201Z ##[endgroup] 2025-08-14T21:35:28.9039738Z (node:47850) NOTE: We are formalizing our plans to enter AWS SDK for JavaScript (v2) into maintenance mode in 2023. 2025-08-14T21:35:28.9041847Z 2025-08-14T21:35:28.9042210Z Please migrate your code to use AWS SDK for JavaScript (v3). 2025-08-14T21:35:28.9042641Z For more information, check the migration guide at https://a.co/7PzMCcy 2025-08-14T21:35:28.9042990Z (Use `node --trace-warnings ...` to show where the warning was created) 2025-08-14T21:35:28.9739930Z Found 0 objects with prefix pytorch/pytorch/16976338999/td_results/ 2025-08-14T21:35:28.9746148Z Artifact download has finished successfully 2025-08-14T21:35:28.9917843Z ##[group]Run mkdir -p .additional_ci_files 2025-08-14T21:35:28.9918143Z mkdir -p .additional_ci_files 2025-08-14T21:35:28.9918449Z mv td_results.json .additional_ci_files/td_results.json || true 2025-08-14T21:35:28.9924397Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:35:28.9924655Z env: 2025-08-14T21:35:28.9924826Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:35:28.9925008Z ##[endgroup] 2025-08-14T21:35:28.9970326Z mv: cannot stat 'td_results.json': No such file or directory 2025-08-14T21:35:29.0003892Z ##[group]Run .github/scripts/parse_ref.py 2025-08-14T21:35:29.0004162Z .github/scripts/parse_ref.py 2025-08-14T21:35:29.0008992Z shell: /usr/bin/bash -e {0} 2025-08-14T21:35:29.0009193Z env: 2025-08-14T21:35:29.0009356Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:35:29.0009544Z ##[endgroup] 2025-08-14T21:35:29.0189798Z Setting output branch=main 2025-08-14T21:35:29.0276563Z Prepare all required actions 2025-08-14T21:35:29.0276854Z Getting action download info 2025-08-14T21:35:29.1258613Z ##[group]Run ./.github/actions/filter-test-configs 2025-08-14T21:35:29.1258884Z with: 2025-08-14T21:35:29.1259345Z github-token: *** 2025-08-14T21:35:29.1261537Z test-matrix: {"include": [{"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_avx2_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}]} 2025-08-14T21:35:29.1264503Z job-name: linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T21:35:29.1264945Z env: 2025-08-14T21:35:29.1265106Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:35:29.1265298Z ##[endgroup] 2025-08-14T21:35:29.1290134Z ##[group]Run nick-fields/retry@v3.0.0 2025-08-14T21:35:29.1290347Z with: 2025-08-14T21:35:29.1290507Z shell: bash 2025-08-14T21:35:29.1290674Z timeout_minutes: 10 2025-08-14T21:35:29.1290842Z max_attempts: 5 2025-08-14T21:35:29.1291028Z retry_wait_seconds: 30 2025-08-14T21:35:29.1291509Z command: set -eux # PyYAML 6.0 doesn't work with MacOS x86 anymore # This must run on Python-3.7 (AmazonLinux2) so can't use request=3.32.2 python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-08-14T21:35:29.1292020Z polling_interval_seconds: 1 2025-08-14T21:35:29.1292218Z warning_on_retry: true 2025-08-14T21:35:29.1292408Z continue_on_error: false 2025-08-14T21:35:29.1292600Z env: 2025-08-14T21:35:29.1292755Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:35:29.1293117Z GITHUB_TOKEN: *** 2025-08-14T21:35:29.1293304Z ##[endgroup] 2025-08-14T21:35:29.2182003Z + python3 -m pip install requests==2.27.1 pyyaml==6.0.2 2025-08-14T21:35:29.3955477Z Defaulting to user installation because normal site-packages is not writeable 2025-08-14T21:35:29.4823285Z Collecting requests==2.27.1 2025-08-14T21:35:29.4966367Z Downloading requests-2.27.1-py2.py3-none-any.whl (63 kB) 2025-08-14T21:35:29.6129648Z Collecting pyyaml==6.0.2 2025-08-14T21:35:29.6183409Z Downloading PyYAML-6.0.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (737 kB) 2025-08-14T21:35:29.6726281Z Collecting certifi>=2017.4.17 2025-08-14T21:35:29.6763504Z Downloading certifi-2025.8.3-py3-none-any.whl (161 kB) 2025-08-14T21:35:29.9243998Z Collecting charset-normalizer~=2.0.0 2025-08-14T21:35:29.9279460Z Downloading charset_normalizer-2.0.12-py3-none-any.whl (39 kB) 2025-08-14T21:35:29.9344471Z Requirement already satisfied: idna<4,>=2.5 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (2.10) 2025-08-14T21:35:29.9348939Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/lib/python3.9/site-packages (from requests==2.27.1) (1.25.10) 2025-08-14T21:35:29.9943220Z Installing collected packages: charset-normalizer, certifi, requests, pyyaml 2025-08-14T21:35:30.1026270Z Successfully installed certifi-2025.8.3 charset-normalizer-2.0.12 pyyaml-6.0.2 requests-2.27.1 2025-08-14T21:35:30.1919115Z Command completed after 1 attempt(s). 2025-08-14T21:35:30.1962159Z ##[group]Run set -x 2025-08-14T21:35:30.1962383Z set -x 2025-08-14T21:35:30.1962558Z  2025-08-14T21:35:30.1962816Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-08-14T21:35:30.1963156Z # in runner workspace 2025-08-14T21:35:30.1963430Z python3 "${GITHUB_ACTION_PATH}/../../scripts/parse_ref.py" 2025-08-14T21:35:30.1969337Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:35:30.1969584Z env: 2025-08-14T21:35:30.1969757Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:35:30.1969944Z ##[endgroup] 2025-08-14T21:35:30.1991069Z + python3 /home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/filter-test-configs/../../scripts/parse_ref.py 2025-08-14T21:35:30.2138103Z Setting output branch=main 2025-08-14T21:35:30.2189104Z ##[group]Run echo "Workflow: ${GITHUB_WORKFLOW}" 2025-08-14T21:35:30.2189387Z echo "Workflow: ${GITHUB_WORKFLOW}" 2025-08-14T21:35:30.2189631Z echo "Job name: ${JOB_NAME}" 2025-08-14T21:35:30.2189833Z  2025-08-14T21:35:30.2190085Z # Use relative path here as this could be checked out anywhere, not necessarily 2025-08-14T21:35:30.2190506Z # in runner workspace 2025-08-14T21:35:30.2190784Z python3 "${GITHUB_ACTION_PATH}/../../scripts/filter_test_configs.py" \ 2025-08-14T21:35:30.2191088Z  --workflow "${GITHUB_WORKFLOW}" \ 2025-08-14T21:35:30.2191301Z  --job-name "${JOB_NAME}" \ 2025-08-14T21:35:30.2193540Z  --test-matrix "{"include": [{"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_avx2_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}]}" \ 2025-08-14T21:35:30.2195974Z  --selected-test-configs "" \ 2025-08-14T21:35:30.2196207Z  --pr-number "${PR_NUMBER}" \ 2025-08-14T21:35:30.2196423Z  --tag "${TAG}" \ 2025-08-14T21:35:30.2196628Z  --event-name "${EVENT_NAME}" \ 2025-08-14T21:35:30.2196847Z  --schedule "${SCHEDULE}" \ 2025-08-14T21:35:30.2197061Z  --branch "${HEAD_BRANCH}" 2025-08-14T21:35:30.2201956Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:35:30.2202214Z env: 2025-08-14T21:35:30.2202385Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:35:30.2202986Z GITHUB_TOKEN: *** 2025-08-14T21:35:30.2203429Z JOB_NAME: linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T21:35:30.2203863Z PR_NUMBER: 2025-08-14T21:35:30.2204052Z TAG: 2025-08-14T21:35:30.2204276Z EVENT_NAME: schedule 2025-08-14T21:35:30.2204468Z SCHEDULE: 45 0,4,8,12,16,20 * * 1-5 2025-08-14T21:35:30.2204681Z HEAD_BRANCH: main 2025-08-14T21:35:30.2204855Z ##[endgroup] 2025-08-14T21:35:30.2227655Z Workflow: inductor-periodic 2025-08-14T21:35:30.2228336Z Job name: linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T21:35:30.3864597Z Setting output keep-going=True 2025-08-14T21:35:30.3865154Z Setting output ci-verbose-test-logs=False 2025-08-14T21:35:30.3865509Z Setting output ci-test-showlocals=False 2025-08-14T21:35:30.3865890Z Setting output ci-no-test-timeout=False 2025-08-14T21:35:30.3866227Z Setting output ci-no-td=False 2025-08-14T21:35:30.3867196Z Setting output ci-td-distributed=False 2025-08-14T21:35:30.3867702Z Setting output is-unstable=False 2025-08-14T21:35:30.3868015Z Setting output reenabled-issues= 2025-08-14T21:35:30.3870381Z Setting output test-matrix={"include": [{"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_avx2_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}]} 2025-08-14T21:35:30.3873100Z Setting output is-test-matrix-empty=False 2025-08-14T21:35:30.3993265Z ##[group]Run echo "Filtered matrix:" 2025-08-14T21:35:30.3993518Z echo "Filtered matrix:" 2025-08-14T21:35:30.3995620Z echo "{"include": [{"config": "cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 1, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "dynamic_cpu_inductor_timm", "shard": 2, "num_shards": 2, "runner": "linux.8xlarge.amx"}, {"config": "cpu_inductor_freezing_avx2_huggingface", "shard": 1, "num_shards": 1, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_torchbench", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 1, "num_shards": 2, "runner": "linux.10xlarge.avx2"}, {"config": "cpu_inductor_freezing_avx2_timm", "shard": 2, "num_shards": 2, "runner": "linux.10xlarge.avx2"}]}" 2025-08-14T21:35:30.3997861Z  2025-08-14T21:35:30.3998006Z echo 2025-08-14T21:35:30.3998191Z echo "Is the current job unstable? False" 2025-08-14T21:35:30.3998393Z  2025-08-14T21:35:30.3998532Z echo 2025-08-14T21:35:30.3998701Z echo "Is keep-going label set? True" 2025-08-14T21:35:30.3998894Z  2025-08-14T21:35:30.3999031Z echo 2025-08-14T21:35:30.3999187Z echo "Reenabled issues? " 2025-08-14T21:35:30.4003902Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:35:30.4004153Z env: 2025-08-14T21:35:30.4004324Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:35:30.4004504Z ##[endgroup] 2025-08-14T21:35:30.4026186Z Filtered matrix: 2025-08-14T21:35:30.4028715Z {include: [{config: cpu_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.8xlarge.amx}, {config: cpu_inductor_timm, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_timm, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_huggingface, shard: 1, num_shards: 1, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_timm, shard: 1, num_shards: 2, runner: linux.8xlarge.amx}, {config: dynamic_cpu_inductor_timm, shard: 2, num_shards: 2, runner: linux.8xlarge.amx}, {config: cpu_inductor_freezing_avx2_huggingface, shard: 1, num_shards: 1, runner: linux.10xlarge.avx2}, {config: cpu_inductor_freezing_avx2_torchbench, shard: 1, num_shards: 2, runner: linux.10xlarge.avx2}, {config: cpu_inductor_freezing_avx2_torchbench, shard: 2, num_shards: 2, runner: linux.10xlarge.avx2}, {config: cpu_inductor_freezing_avx2_timm, shard: 1, num_shards: 2, runner: linux.10xlarge.avx2}, {config: cpu_inductor_freezing_avx2_timm, shard: 2, num_shards: 2, runner: linux.10xlarge.avx2}]} 2025-08-14T21:35:30.4030986Z 2025-08-14T21:35:30.4031074Z Is the current job unstable? False 2025-08-14T21:35:30.4031283Z 2025-08-14T21:35:30.4031374Z Is keep-going label set? True 2025-08-14T21:35:30.4031503Z 2025-08-14T21:35:30.4031577Z Reenabled issues? 2025-08-14T21:35:30.4060326Z ##[group]Run echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-08-14T21:35:30.4060753Z echo "timeout=$((JOB_TIMEOUT-30))" >> "${GITHUB_OUTPUT}" 2025-08-14T21:35:30.4064882Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:35:30.4065110Z env: 2025-08-14T21:35:30.4065261Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:35:30.4065420Z JOB_TIMEOUT: 240 2025-08-14T21:35:30.4065574Z ##[endgroup] 2025-08-14T21:35:30.4134052Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-08-14T21:35:30.4134380Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-08-14T21:35:30.4134652Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}" 2025-08-14T21:35:30.4138773Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:35:30.4139010Z env: 2025-08-14T21:35:30.4139165Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:35:30.4139331Z ##[endgroup] 2025-08-14T21:35:30.4222774Z ##[group]Run set -x 2025-08-14T21:35:30.4223012Z set -x 2025-08-14T21:35:30.4223177Z  2025-08-14T21:35:30.4223385Z if [[ $TEST_CONFIG == 'multigpu' ]]; then 2025-08-14T21:35:30.4223638Z  TEST_COMMAND=.ci/pytorch/multigpu-test.sh 2025-08-14T21:35:30.4223867Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then 2025-08-14T21:35:30.4224090Z  TEST_COMMAND=.ci/onnx/test.sh 2025-08-14T21:35:30.4224276Z else 2025-08-14T21:35:30.4224441Z  TEST_COMMAND=.ci/pytorch/test.sh 2025-08-14T21:35:30.4224621Z fi 2025-08-14T21:35:30.4224758Z  2025-08-14T21:35:30.4224928Z # Leaving 1GB for the runner and other things 2025-08-14T21:35:30.4225257Z TOTAL_AVAILABLE_MEMORY_IN_GB=$(awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo) 2025-08-14T21:35:30.4225764Z # https://docs.docker.com/engine/containers/resource_constraints/#--memory-swap-details, the 3GB swap 2025-08-14T21:35:30.4226163Z # comes from https://github.com/pytorch/test-infra/pull/6058 2025-08-14T21:35:30.4226467Z TOTAL_MEMORY_WITH_SWAP=$(("${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}" + 3)) 2025-08-14T21:35:30.4226709Z  2025-08-14T21:35:30.4226886Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-08-14T21:35:30.4227091Z  SHM_OPTS= 2025-08-14T21:35:30.4227256Z  JENKINS_USER= 2025-08-14T21:35:30.4227473Z  # ensure that docker container cleanly exits in 12 hours 2025-08-14T21:35:30.4227742Z  # if for some reason cleanup action doesn't stop container 2025-08-14T21:35:30.4227977Z  # when job is cancelled 2025-08-14T21:35:30.4228173Z  DOCKER_SHELL_CMD="sleep 12h" 2025-08-14T21:35:30.4228354Z else 2025-08-14T21:35:30.4228514Z  SHM_OPTS="--shm-size=${SHM_SIZE}" 2025-08-14T21:35:30.4228718Z  JENKINS_USER="--user jenkins" 2025-08-14T21:35:30.4228911Z  DOCKER_SHELL_CMD= 2025-08-14T21:35:30.4229069Z fi 2025-08-14T21:35:30.4229203Z  2025-08-14T21:35:30.4229413Z # detached container should get cleaned up by teardown_ec2_linux 2025-08-14T21:35:30.4229708Z # TODO: Stop building test binaries as part of the build phase 2025-08-14T21:35:30.4230047Z # Used for GPU_FLAG, SHM_OPTS, JENKINS_USER and DOCKER_SHELL_CMD since that doesn't play nice 2025-08-14T21:35:30.4230344Z # shellcheck disable=SC2086,SC2090 2025-08-14T21:35:30.4230547Z container_name=$(docker run \ 2025-08-14T21:35:30.4230732Z  ${GPU_FLAG:-} \ 2025-08-14T21:35:30.4231061Z  ${SCCACHE_SERVER_PORT_DOCKER_FLAG:-} \ 2025-08-14T21:35:30.4231321Z  -e BUILD_ENVIRONMENT \ 2025-08-14T21:35:30.4231512Z  -e PR_NUMBER \ 2025-08-14T21:35:30.4231692Z  -e GITHUB_ACTIONS \ 2025-08-14T21:35:30.4231881Z  -e GITHUB_REPOSITORY \ 2025-08-14T21:35:30.4232078Z  -e GITHUB_WORKFLOW \ 2025-08-14T21:35:30.4232255Z  -e GITHUB_JOB \ 2025-08-14T21:35:30.4232436Z  -e GITHUB_RUN_ID \ 2025-08-14T21:35:30.4232771Z  -e GITHUB_RUN_NUMBER \ 2025-08-14T21:35:30.4232958Z  -e GITHUB_RUN_ATTEMPT \ 2025-08-14T21:35:30.4233154Z  -e JOB_ID \ 2025-08-14T21:35:30.4233327Z  -e JOB_NAME \ 2025-08-14T21:35:30.4233497Z  -e BASE_SHA \ 2025-08-14T21:35:30.4233672Z  -e BRANCH \ 2025-08-14T21:35:30.4233840Z  -e SHA1 \ 2025-08-14T21:35:30.4234007Z  -e AWS_DEFAULT_REGION \ 2025-08-14T21:35:30.4234201Z  -e IN_WHEEL_TEST \ 2025-08-14T21:35:30.4234384Z  -e SHARD_NUMBER \ 2025-08-14T21:35:30.4234565Z  -e TEST_CONFIG \ 2025-08-14T21:35:30.4234741Z  -e NUM_TEST_SHARDS \ 2025-08-14T21:35:30.4234932Z  -e REENABLED_ISSUES \ 2025-08-14T21:35:30.4235135Z  -e CONTINUE_THROUGH_ERROR \ 2025-08-14T21:35:30.4235402Z  -e VERBOSE_TEST_LOGS \ 2025-08-14T21:35:30.4235609Z  -e TEST_SHOWLOCALS \ 2025-08-14T21:35:30.4235809Z  -e NO_TEST_TIMEOUT \ 2025-08-14T21:35:30.4236003Z  -e NO_TD \ 2025-08-14T21:35:30.4236177Z  -e TD_DISTRIBUTED \ 2025-08-14T21:35:30.4236364Z  -e PR_LABELS \ 2025-08-14T21:35:30.4236563Z  -e MAX_JOBS="$(nproc --ignore=2)" \ 2025-08-14T21:35:30.4236773Z  -e SCCACHE_BUCKET \ 2025-08-14T21:35:30.4236958Z  -e SCCACHE_REGION \ 2025-08-14T21:35:30.4237139Z  -e XLA_CUDA \ 2025-08-14T21:35:30.4237325Z  -e XLA_CLANG_CACHE_S3_BUCKET_NAME \ 2025-08-14T21:35:30.4237555Z  -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \ 2025-08-14T21:35:30.4237952Z  -e PYTORCH_TEST_RERUN_DISABLED_TESTS \ 2025-08-14T21:35:30.4238186Z  -e SKIP_SCCACHE_INITIALIZATION=1 \ 2025-08-14T21:35:30.4238407Z  -e HUGGING_FACE_HUB_TOKEN \ 2025-08-14T21:35:30.4238626Z  -e SCRIBE_GRAPHQL_ACCESS_TOKEN \ 2025-08-14T21:35:30.4238834Z  -e DASHBOARD_TAG \ 2025-08-14T21:35:30.4239018Z  -e ARTIFACTS_FILE_SUFFIX \ 2025-08-14T21:35:30.4239257Z  --memory="${TOTAL_AVAILABLE_MEMORY_IN_GB%.*}g" \ 2025-08-14T21:35:30.4239523Z  --memory-swap="${TOTAL_MEMORY_WITH_SWAP}g" \ 2025-08-14T21:35:30.4239778Z  --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \ 2025-08-14T21:35:30.4240030Z  --security-opt seccomp=unconfined \ 2025-08-14T21:35:30.4240248Z  --cap-add=SYS_PTRACE \ 2025-08-14T21:35:30.4240441Z  --ipc=host \ 2025-08-14T21:35:30.4240612Z  ${SHM_OPTS} \ 2025-08-14T21:35:30.4240781Z  --tty \ 2025-08-14T21:35:30.4240944Z  --detach \ 2025-08-14T21:35:30.4241118Z  --name="${container_name}" \ 2025-08-14T21:35:30.4241397Z  ${JENKINS_USER} \ 2025-08-14T21:35:30.4241635Z  -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \ 2025-08-14T21:35:30.4241882Z  -w /var/lib/jenkins/workspace \ 2025-08-14T21:35:30.4242090Z  "${DOCKER_IMAGE}" \ 2025-08-14T21:35:30.4242282Z  ${DOCKER_SHELL_CMD} 2025-08-14T21:35:30.4242457Z ) 2025-08-14T21:35:30.4242669Z # Propagate download.pytorch.org IP to container 2025-08-14T21:35:30.4243148Z grep download.pytorch.org /etc/hosts | docker exec -i "${container_name}" sudo bash -c "/bin/cat >> /etc/hosts" 2025-08-14T21:35:30.4243600Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}" 2025-08-14T21:35:30.4243862Z  2025-08-14T21:35:30.4244053Z if [[ ${BUILD_ENVIRONMENT} == *"s390x"* ]]; then 2025-08-14T21:35:30.4244417Z  docker exec -t "${container_name}" sh -c "python3 -m pip install -r .ci/docker/requirements-ci.txt" 2025-08-14T21:35:30.4244725Z fi 2025-08-14T21:35:30.4244866Z  2025-08-14T21:35:30.4245173Z docker exec -t "${container_name}" sh -c "python3 -m pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}" 2025-08-14T21:35:30.4249754Z shell: /usr/bin/bash -e {0} 2025-08-14T21:35:30.4250078Z env: 2025-08-14T21:35:30.4250232Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:35:30.4250444Z BUILD_ENVIRONMENT: linux-jammy-py3.9-gcc11-build 2025-08-14T21:35:30.4250653Z PR_NUMBER: 2025-08-14T21:35:30.4250815Z GITHUB_REPOSITORY: pytorch/pytorch 2025-08-14T21:35:30.4251021Z GITHUB_WORKFLOW: inductor-periodic 2025-08-14T21:35:30.4251199Z GITHUB_JOB: test 2025-08-14T21:35:30.4251403Z GITHUB_RUN_ID: 16976338999 2025-08-14T21:35:30.4251575Z GITHUB_RUN_NUMBER: 66307 2025-08-14T21:35:30.4251742Z GITHUB_RUN_ATTEMPT: 1 2025-08-14T21:35:30.4251899Z JOB_ID: 48128301909 2025-08-14T21:35:30.4252291Z JOB_NAME: linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T21:35:30.4252690Z BRANCH: main 2025-08-14T21:35:30.4252857Z SHA1: 1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:35:30.4253172Z BASE_SHA: 1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:35:30.4253414Z TEST_CONFIG: dynamic_cpu_inductor_huggingface 2025-08-14T21:35:30.4253623Z SHARD_NUMBER: 1 2025-08-14T21:35:30.4253772Z NUM_TEST_SHARDS: 1 2025-08-14T21:35:30.4253937Z REENABLED_ISSUES: 2025-08-14T21:35:30.4254102Z CONTINUE_THROUGH_ERROR: True 2025-08-14T21:35:30.4254273Z VERBOSE_TEST_LOGS: False 2025-08-14T21:35:30.4254445Z TEST_SHOWLOCALS: False 2025-08-14T21:35:30.4254606Z NO_TEST_TIMEOUT: False 2025-08-14T21:35:30.4254780Z NO_TD: False 2025-08-14T21:35:30.4254927Z TD_DISTRIBUTED: False 2025-08-14T21:35:30.4255118Z SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2 2025-08-14T21:35:30.4255340Z SCCACHE_REGION: us-east-1 2025-08-14T21:35:30.4255505Z SHM_SIZE: 1g 2025-08-14T21:35:30.4255992Z DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:35:30.4256501Z XLA_CUDA: 2025-08-14T21:35:30.4256734Z XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla 2025-08-14T21:35:30.4257020Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0 2025-08-14T21:35:30.4257222Z PYTORCH_TEST_RERUN_DISABLED_TESTS: 0 2025-08-14T21:35:30.4257413Z DASHBOARD_TAG: 2025-08-14T21:35:30.4257740Z HUGGING_FACE_HUB_TOKEN: *** 2025-08-14T21:35:30.4258001Z SCRIBE_GRAPHQL_ACCESS_TOKEN: *** 2025-08-14T21:35:30.4258321Z ARTIFACTS_FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301909 2025-08-14T21:35:30.4258631Z ##[endgroup] 2025-08-14T21:35:30.4278882Z + [[ dynamic_cpu_inductor_huggingface == \m\u\l\t\i\g\p\u ]] 2025-08-14T21:35:30.4279401Z + [[ linux-jammy-py3.9-gcc11-build == *onnx* ]] 2025-08-14T21:35:30.4279815Z + TEST_COMMAND=.ci/pytorch/test.sh 2025-08-14T21:35:30.4283902Z ++ awk '/MemTotal/ { printf "%.3f \n", $2/1024/1024 - 1 }' /proc/meminfo 2025-08-14T21:35:30.4306518Z + TOTAL_AVAILABLE_MEMORY_IN_GB='122.780 ' 2025-08-14T21:35:30.4306960Z + TOTAL_MEMORY_WITH_SWAP=125 2025-08-14T21:35:30.4307325Z + [[ linux-jammy-py3.9-gcc11-build == *\s\3\9\0\x* ]] 2025-08-14T21:35:30.4307693Z + SHM_OPTS=--shm-size=1g 2025-08-14T21:35:30.4307972Z + JENKINS_USER='--user jenkins' 2025-08-14T21:35:30.4308241Z + DOCKER_SHELL_CMD= 2025-08-14T21:35:30.4312739Z +++ nproc --ignore=2 2025-08-14T21:35:30.4338639Z ++ docker run -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e GITHUB_REPOSITORY -e GITHUB_WORKFLOW -e GITHUB_JOB -e GITHUB_RUN_ID -e GITHUB_RUN_NUMBER -e GITHUB_RUN_ATTEMPT -e JOB_ID -e JOB_NAME -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e REENABLED_ISSUES -e CONTINUE_THROUGH_ERROR -e VERBOSE_TEST_LOGS -e TEST_SHOWLOCALS -e NO_TEST_TIMEOUT -e NO_TD -e TD_DISTRIBUTED -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_REGION -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS -e SKIP_SCCACHE_INITIALIZATION=1 -e HUGGING_FACE_HUB_TOKEN -e SCRIBE_GRAPHQL_ACCESS_TOKEN -e DASHBOARD_TAG -e ARTIFACTS_FILE_SUFFIX --memory=122g --memory-swap=125g --env-file=/tmp/github_env_16976338999 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=1g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:35:40.7228298Z + container_name=bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:35:40.7232692Z + grep download.pytorch.org /etc/hosts 2025-08-14T21:35:40.7235119Z + docker exec -i bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b sudo bash -c '/bin/cat >> /etc/hosts' 2025-08-14T21:35:40.8612917Z + echo DOCKER_CONTAINER_ID=bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:35:40.8613661Z + [[ linux-jammy-py3.9-gcc11-build == *\s\3\9\0\x* ]] 2025-08-14T21:35:40.8616799Z ++ echo dist/torch-2.9.0a0+git1fc683c-cp39-cp39-linux_x86_64.whl 2025-08-14T21:35:40.8619666Z + docker exec -t bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b sh -c 'python3 -m pip install dist/torch-2.9.0a0+git1fc683c-cp39-cp39-linux_x86_64.whl[opt-einsum] && .ci/pytorch/test.sh' 2025-08-14T21:35:41.2034495Z Processing ./dist/torch-2.9.0a0+git1fc683c-cp39-cp39-linux_x86_64.whl (from torch==2.9.0a0+git1fc683c) 2025-08-14T21:35:41.4121609Z Requirement already satisfied: filelock in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git1fc683c->torch==2.9.0a0+git1fc683c) (3.18.0) 2025-08-14T21:35:41.4122517Z Requirement already satisfied: typing-extensions>=4.10.0 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git1fc683c->torch==2.9.0a0+git1fc683c) (4.14.1) 2025-08-14T21:35:41.4126889Z Requirement already satisfied: sympy>=1.13.3 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git1fc683c->torch==2.9.0a0+git1fc683c) (1.13.3) 2025-08-14T21:35:41.4131211Z Requirement already satisfied: networkx>=2.5.1 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git1fc683c->torch==2.9.0a0+git1fc683c) (2.8.8) 2025-08-14T21:35:41.4135012Z Requirement already satisfied: jinja2 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git1fc683c->torch==2.9.0a0+git1fc683c) (3.1.6) 2025-08-14T21:35:41.4135855Z Requirement already satisfied: fsspec>=0.8.5 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git1fc683c->torch==2.9.0a0+git1fc683c) (2025.3.0) 2025-08-14T21:35:41.4150047Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from torch==2.9.0a0+git1fc683c->torch==2.9.0a0+git1fc683c) (3.3.0) 2025-08-14T21:35:41.4433741Z Requirement already satisfied: numpy>=1.7 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from opt-einsum>=3.3->torch==2.9.0a0+git1fc683c->torch==2.9.0a0+git1fc683c) (1.22.4) 2025-08-14T21:35:41.4445993Z Requirement already satisfied: mpmath<1.4,>=1.1.0 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from sympy>=1.13.3->torch==2.9.0a0+git1fc683c->torch==2.9.0a0+git1fc683c) (1.3.0) 2025-08-14T21:35:41.4495963Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/envs/py_3.9/lib/python3.9/site-packages (from jinja2->torch==2.9.0a0+git1fc683c->torch==2.9.0a0+git1fc683c) (3.0.2) 2025-08-14T21:35:42.1665101Z Installing collected packages: torch 2025-08-14T21:35:49.1206150Z ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts. 2025-08-14T21:35:49.1206750Z dall-e 0.1 requires torchvision, which is not installed. 2025-08-14T21:35:49.1210814Z effdet 0.4.1 requires torchvision, which is not installed. 2025-08-14T21:35:49.1215189Z pytorch-labs-segment-anything-fast 0.2 requires torchao, which is not installed. 2025-08-14T21:35:49.1215658Z pytorch-labs-segment-anything-fast 0.2 requires torchvision>=0.17.0.dev20231026, which is not installed. 2025-08-14T21:35:49.1216690Z timm 1.0.14 requires torchvision, which is not installed. 2025-08-14T21:35:49.1217005Z Successfully installed torch-2.9.0a0+git1fc683c 2025-08-14T21:35:49.2094813Z + export TERM=vt100 2025-08-14T21:35:49.2097430Z + TERM=vt100 2025-08-14T21:35:49.2097733Z ++ dirname .ci/pytorch/test.sh 2025-08-14T21:35:49.2106778Z + source .ci/pytorch/common.sh 2025-08-14T21:35:49.2109080Z +++ dirname .ci/pytorch/common.sh 2025-08-14T21:35:49.2119287Z ++ source .ci/pytorch/common_utils.sh 2025-08-14T21:35:49.2119591Z +++ declare -f -t trap_add 2025-08-14T21:35:49.2119786Z ++ set -ex -o pipefail 2025-08-14T21:35:49.2119986Z ++ [[ linux-jammy-py3.9-gcc11-build == *rocm* ]] 2025-08-14T21:35:49.2120220Z ++ BUILD_TEST_LIBTORCH=0 2025-08-14T21:35:49.2123972Z ++ dirname .ci/pytorch/test.sh 2025-08-14T21:35:49.2125896Z + source .ci/pytorch/common-build.sh 2025-08-14T21:35:49.2128985Z ++ [[ linux-jammy-py3.9-gcc11-build != *win-* ]] 2025-08-14T21:35:49.2136460Z ++++ dirname .ci/pytorch/common-build.sh 2025-08-14T21:35:49.2144079Z +++ cd .ci/pytorch 2025-08-14T21:35:49.2148545Z +++ pwd -P 2025-08-14T21:35:49.2153659Z ++ script_dir=/var/lib/jenkins/workspace/.ci/pytorch 2025-08-14T21:35:49.2155700Z ++ [[ linux-jammy-py3.9-gcc11-build == *-pch* ]] 2025-08-14T21:35:49.2156117Z ++ which sccache 2025-08-14T21:35:49.2170968Z ++ [[ -z ossci-compiler-cache-circleci-v2 ]] 2025-08-14T21:35:49.2173036Z ++ sccache --stop-server 2025-08-14T21:35:49.2192141Z ++ true 2025-08-14T21:35:49.2196695Z ++ rm -f /var/lib/jenkins/sccache_error.log 2025-08-14T21:35:49.2204765Z ++ trap_add sccache_epilogue EXIT 2025-08-14T21:35:49.2205046Z ++ trap_add_cmd=sccache_epilogue 2025-08-14T21:35:49.2205250Z ++ shift 2025-08-14T21:35:49.2205560Z ++ for trap_add_name in "$@" 2025-08-14T21:35:49.2210959Z ++++ trap -p EXIT 2025-08-14T21:35:49.2216711Z +++ eval 'extract_trap_cmd ' 2025-08-14T21:35:49.2220975Z ++++ extract_trap_cmd 2025-08-14T21:35:49.2224115Z ++++ printf '%s\n' '' 2025-08-14T21:35:49.2224328Z +++ printf '%s\n' sccache_epilogue 2025-08-14T21:35:49.2224534Z ++ trap -- ' 2025-08-14T21:35:49.2224684Z sccache_epilogue' EXIT 2025-08-14T21:35:49.2224844Z ++ [[ -n 1 ]] 2025-08-14T21:35:49.2225113Z ++ echo 'Skipping sccache server initialization, setting environment variables' 2025-08-14T21:35:49.2225468Z Skipping sccache server initialization, setting environment variables 2025-08-14T21:35:49.2225728Z ++ export SCCACHE_IDLE_TIMEOUT=0 2025-08-14T21:35:49.2225915Z ++ SCCACHE_IDLE_TIMEOUT=0 2025-08-14T21:35:49.2226132Z ++ export SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-08-14T21:35:49.2226392Z ++ SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-08-14T21:35:49.2226671Z ++ export RUST_LOG=sccache::server=error 2025-08-14T21:35:49.2226875Z ++ RUST_LOG=sccache::server=error 2025-08-14T21:35:49.2227055Z ++ sccache --zero-stats 2025-08-14T21:35:49.3737157Z Statistics zeroed. 2025-08-14T21:35:49.3749371Z ++ which ccache 2025-08-14T21:35:49.3782034Z + [[ linux-jammy-py3.9-gcc11-build != *rocm* ]] 2025-08-14T21:35:49.3782512Z + [[ linux-jammy-py3.9-gcc11-build != *s390x* ]] 2025-08-14T21:35:49.3782883Z + [[ -d /var/lib/jenkins/workspace ]] 2025-08-14T21:35:49.3785895Z ++ stat -c %u /var/lib/jenkins/workspace 2025-08-14T21:35:49.3799178Z + WORKSPACE_ORIGINAL_OWNER_ID=1000 2025-08-14T21:35:49.3799460Z + trap_add cleanup_workspace EXIT 2025-08-14T21:35:49.3799672Z + trap_add_cmd=cleanup_workspace 2025-08-14T21:35:49.3799856Z + shift 2025-08-14T21:35:49.3800005Z + for trap_add_name in "$@" 2025-08-14T21:35:49.3801933Z +++ trap -p EXIT 2025-08-14T21:35:49.3802179Z ++ eval 'extract_trap_cmd trap -- '\'' 2025-08-14T21:35:49.3802427Z sccache_epilogue'\'' EXIT' 2025-08-14T21:35:49.3802622Z +++ extract_trap_cmd trap -- ' 2025-08-14T21:35:49.3802812Z sccache_epilogue' EXIT 2025-08-14T21:35:49.3802978Z +++ printf '%s\n' ' 2025-08-14T21:35:49.3803147Z sccache_epilogue' 2025-08-14T21:35:49.3803357Z ++ printf '%s\n' cleanup_workspace 2025-08-14T21:35:49.3805185Z + trap -- ' 2025-08-14T21:35:49.3805732Z sccache_epilogue 2025-08-14T21:35:49.3805913Z cleanup_workspace' EXIT 2025-08-14T21:35:49.3806141Z + sudo chown -R jenkins /var/lib/jenkins/workspace 2025-08-14T21:35:49.7848876Z + git config --global --add safe.directory /var/lib/jenkins/workspace 2025-08-14T21:35:49.7866863Z + echo 'Environment variables:' 2025-08-14T21:35:49.7868051Z Environment variables: 2025-08-14T21:35:49.7868233Z + env 2025-08-14T21:35:49.7876093Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-08-14T21:35:49.7880308Z CONTINUE_THROUGH_ERROR=True 2025-08-14T21:35:49.7883814Z BUILD_ENVIRONMENT=linux-jammy-py3.9-gcc11-build 2025-08-14T21:35:49.7885750Z HOSTNAME=bbffe2680397 2025-08-14T21:35:49.7886135Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_f268a462-bc4d-48b6-9f56-069a24ae65c2 2025-08-14T21:35:49.7886509Z GITHUB_ACTION=__run_2 2025-08-14T21:35:49.7886971Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-08-14T21:35:49.7887209Z GITHUB_RUN_NUMBER=66307 2025-08-14T21:35:49.7887414Z TEST_CONFIG=dynamic_cpu_inductor_huggingface 2025-08-14T21:35:49.7887626Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-08-14T21:35:49.7887827Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-08-14T21:35:49.7888021Z SCCACHE_IDLE_TIMEOUT=0 2025-08-14T21:35:49.7888425Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-08-14T21:35:49.7888615Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-08-14T21:35:49.7888797Z GITHUB_REF_TYPE=branch 2025-08-14T21:35:49.7888963Z TORCH_CUDA_ARCH_LIST=Maxwell 2025-08-14T21:35:49.7889158Z BASE_SHA=1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:35:49.7889349Z XLA_CUDA= 2025-08-14T21:35:49.7889506Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-08-14T21:35:49.7889751Z HUGGING_FACE_HUB_TOKEN=*** 2025-08-14T21:35:49.7890172Z *** 2025-08-14T21:35:49.7890317Z GITHUB_REPOSITORY_ID=65600975 2025-08-14T21:35:49.7890496Z GITHUB_ACTIONS=true 2025-08-14T21:35:49.7890700Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-08-14T21:35:49.7890933Z SHA1=1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:35:49.7891165Z GITHUB_SHA=1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:35:49.7891510Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/heads/main 2025-08-14T21:35:49.7891823Z UCC_HOME=/usr 2025-08-14T21:35:49.7891974Z VERBOSE_TEST_LOGS=False 2025-08-14T21:35:49.7892143Z GITHUB_REF=refs/heads/main 2025-08-14T21:35:49.7892299Z SHARD_NUMBER=1 2025-08-14T21:35:49.7892452Z GITHUB_REF_PROTECTED=true 2025-08-14T21:35:49.7892618Z HOME=/var/lib/jenkins 2025-08-14T21:35:49.7892792Z GITHUB_API_URL=https://api.github.com 2025-08-14T21:35:49.7892997Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-08-14T21:35:49.7893174Z UCX_COMMIT= 2025-08-14T21:35:49.7893313Z USE_SYSTEM_NCCL=1 2025-08-14T21:35:49.7893453Z NUM_TEST_SHARDS=1 2025-08-14T21:35:49.7893597Z UCX_HOME=/usr 2025-08-14T21:35:49.7893924Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_f268a462-bc4d-48b6-9f56-069a24ae65c2 2025-08-14T21:35:49.7894496Z JOB_NAME=linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T21:35:49.7895053Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_f268a462-bc4d-48b6-9f56-069a24ae65c2 2025-08-14T21:35:49.7895493Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-08-14T21:35:49.7895778Z GITHUB_EVENT_NAME=schedule 2025-08-14T21:35:49.7895937Z DASHBOARD_TAG= 2025-08-14T21:35:49.7896091Z GITHUB_RUN_ID=16976338999 2025-08-14T21:35:49.7896256Z INSTALLED_OPENBLAS= 2025-08-14T21:35:49.7896589Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_f268a462-bc4d-48b6-9f56-069a24ae65c2 2025-08-14T21:35:49.7896967Z GITHUB_ACTOR=pytorchmergebot 2025-08-14T21:35:49.7897136Z PR_NUMBER= 2025-08-14T21:35:49.7897268Z DESIRED_CUDA= 2025-08-14T21:35:49.7897416Z GITHUB_RUN_ATTEMPT=1 2025-08-14T21:35:49.7897581Z ANACONDA_PYTHON_VERSION=3.9 2025-08-14T21:35:49.7897780Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-08-14T21:35:49.7898105Z TERM=vt100 2025-08-14T21:35:49.7898248Z INSTALLED_VISION=yes 2025-08-14T21:35:49.7898392Z BRANCH=main 2025-08-14T21:35:49.7898538Z SCCACHE_REGION=us-east-1 2025-08-14T21:35:49.7898717Z OPENSSL_ROOT_DIR=/opt/openssl 2025-08-14T21:35:49.7898891Z CUDA_PATH=/usr/local/cuda 2025-08-14T21:35:49.7899195Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-08-14T21:35:49.7899527Z GITHUB_SERVER_URL=https://github.com 2025-08-14T21:35:49.7899715Z UCC_COMMIT= 2025-08-14T21:35:49.7899848Z REENABLED_ISSUES= 2025-08-14T21:35:49.7899995Z DOCS=yes 2025-08-14T21:35:49.7900128Z SHLVL=1 2025-08-14T21:35:49.7900256Z MAX_JOBS=30 2025-08-14T21:35:49.7900399Z GITHUB_ACTOR_ID=97764156 2025-08-14T21:35:49.7900608Z GITHUB_WORKFLOW_SHA=1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:35:49.7900877Z GITHUB_REF_NAME=main 2025-08-14T21:35:49.7901116Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-08-14T21:35:49.7901372Z GITHUB_JOB=test 2025-08-14T21:35:49.7901517Z NO_TEST_TIMEOUT=False 2025-08-14T21:35:49.7901675Z TD_DISTRIBUTED=False 2025-08-14T21:35:49.7901843Z GITHUB_REPOSITORY=pytorch/pytorch 2025-08-14T21:35:49.7902021Z GITHUB_RETENTION_DAYS=90 2025-08-14T21:35:49.7902188Z OPENSSL_DIR=/opt/openssl 2025-08-14T21:35:49.7902355Z GITHUB_ACTION_REPOSITORY= 2025-08-14T21:35:49.7902790Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-08-14T21:35:49.7903216Z GITHUB_BASE_REF= 2025-08-14T21:35:49.7903366Z INSTALLED_ACL= 2025-08-14T21:35:49.7903638Z ARTIFACTS_FILE_SUFFIX=test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301909 2025-08-14T21:35:49.7903927Z CI=true 2025-08-14T21:35:49.7904071Z GITHUB_REPOSITORY_OWNER=pytorch 2025-08-14T21:35:49.7904301Z RUST_LOG=sccache::server=error 2025-08-14T21:35:49.7904464Z JOB_ID=48128301909 2025-08-14T21:35:49.7904609Z GITHUB_HEAD_REF= 2025-08-14T21:35:49.7904759Z GITHUB_ACTION_REF= 2025-08-14T21:35:49.7904934Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-08-14T21:35:49.7905144Z TEST_SHOWLOCALS=False 2025-08-14T21:35:49.7905313Z GITHUB_WORKFLOW=inductor-periodic 2025-08-14T21:35:49.7905495Z DEBIAN_FRONTEND=noninteractive 2025-08-14T21:35:49.7905844Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_f268a462-bc4d-48b6-9f56-069a24ae65c2 2025-08-14T21:35:49.7906189Z NO_TD=False 2025-08-14T21:35:49.7906339Z SKIP_SCCACHE_INITIALIZATION=1 2025-08-14T21:35:49.7906532Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-08-14T21:35:49.7906717Z _=/usr/bin/env 2025-08-14T21:35:49.7906911Z ++ python -c 'import site; print(site.getsitepackages()[0])' 2025-08-14T21:35:49.8126287Z + TORCH_INSTALL_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch 2025-08-14T21:35:49.8130952Z + TORCH_BIN_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/bin 2025-08-14T21:35:49.8131410Z + TORCH_LIB_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/lib 2025-08-14T21:35:49.8131813Z + TORCH_TEST_DIR=/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/test 2025-08-14T21:35:49.8132132Z + BUILD_DIR=build 2025-08-14T21:35:49.8132336Z + BUILD_RENAMED_DIR=build_renamed 2025-08-14T21:35:49.8132564Z + BUILD_BIN_DIR=build/bin 2025-08-14T21:35:49.8132756Z + SHARD_NUMBER=1 2025-08-14T21:35:49.8132956Z + NUM_TEST_SHARDS=1 2025-08-14T21:35:49.8133188Z + export TORCH_SERIALIZATION_DEBUG=1 2025-08-14T21:35:49.8133418Z + TORCH_SERIALIZATION_DEBUG=1 2025-08-14T21:35:49.8133651Z + export VALGRIND=ON 2025-08-14T21:35:49.8133856Z + VALGRIND=ON 2025-08-14T21:35:49.8134107Z + [[ linux-jammy-py3.9-gcc11-build == *clang9* ]] 2025-08-14T21:35:49.8134378Z + [[ linux-jammy-py3.9-gcc11-build == *xpu* ]] 2025-08-14T21:35:49.8134657Z + [[ linux-jammy-py3.9-gcc11-build == *s390x* ]] 2025-08-14T21:35:49.8134897Z + [[ 0 == \1 ]] 2025-08-14T21:35:49.8135048Z + [[ True == \1 ]] 2025-08-14T21:35:49.8135235Z + [[ linux-jammy-py3.9-gcc11-build != *bazel* ]] 2025-08-14T21:35:49.8135787Z ++ realpath build/custom_test_artifacts 2025-08-14T21:35:49.8137984Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts 2025-08-14T21:35:49.8138481Z + [[ -n '' ]] 2025-08-14T21:35:49.8138759Z + echo 'Environment variables' 2025-08-14T21:35:49.8139157Z Environment variables 2025-08-14T21:35:49.8139339Z + env 2025-08-14T21:35:49.8170335Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-08-14T21:35:49.8172008Z CONTINUE_THROUGH_ERROR=True 2025-08-14T21:35:49.8172388Z BUILD_ENVIRONMENT=linux-jammy-py3.9-gcc11-build 2025-08-14T21:35:49.8175917Z HOSTNAME=bbffe2680397 2025-08-14T21:35:49.8181441Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_f268a462-bc4d-48b6-9f56-069a24ae65c2 2025-08-14T21:35:49.8187057Z GITHUB_ACTION=__run_2 2025-08-14T21:35:49.8189245Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0 2025-08-14T21:35:49.8189701Z GITHUB_RUN_NUMBER=66307 2025-08-14T21:35:49.8190085Z TEST_CONFIG=dynamic_cpu_inductor_huggingface 2025-08-14T21:35:49.8190474Z GITHUB_REPOSITORY_OWNER_ID=21003710 2025-08-14T21:35:49.8190813Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all 2025-08-14T21:35:49.8191177Z SCCACHE_IDLE_TIMEOUT=0 2025-08-14T21:35:49.8192014Z SCRIBE_GRAPHQL_ACCESS_TOKEN=*** 2025-08-14T21:35:49.8192330Z GITHUB_TRIGGERING_ACTOR=pytorchmergebot 2025-08-14T21:35:49.8192537Z GITHUB_REF_TYPE=branch 2025-08-14T21:35:49.8192707Z TORCH_CUDA_ARCH_LIST=Maxwell 2025-08-14T21:35:49.8192916Z BASE_SHA=1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:35:49.8193125Z XLA_CUDA= 2025-08-14T21:35:49.8193282Z NCCL_LIB_DIR=/usr/local/cuda/lib64/ 2025-08-14T21:35:49.8193529Z HUGGING_FACE_HUB_TOKEN=*** 2025-08-14T21:35:49.8193755Z *** 2025-08-14T21:35:49.8193903Z GITHUB_REPOSITORY_ID=65600975 2025-08-14T21:35:49.8194077Z GITHUB_ACTIONS=true 2025-08-14T21:35:49.8194287Z SCCACHE_ERROR_LOG=/var/lib/jenkins/sccache_error.log 2025-08-14T21:35:49.8194525Z SHA1=1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:35:49.8194754Z GITHUB_SHA=1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:35:49.8195078Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/inductor-periodic.yml@refs/heads/main 2025-08-14T21:35:49.8195379Z UCC_HOME=/usr 2025-08-14T21:35:49.8195534Z TORCH_SERIALIZATION_DEBUG=1 2025-08-14T21:35:49.8195702Z VERBOSE_TEST_LOGS=False 2025-08-14T21:35:49.8195866Z GITHUB_REF=refs/heads/main 2025-08-14T21:35:49.8196032Z SHARD_NUMBER=1 2025-08-14T21:35:49.8196179Z GITHUB_REF_PROTECTED=true 2025-08-14T21:35:49.8196348Z HOME=/var/lib/jenkins 2025-08-14T21:35:49.8196536Z GITHUB_API_URL=https://api.github.com 2025-08-14T21:35:49.8196738Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0 2025-08-14T21:35:49.8196920Z UCX_COMMIT= 2025-08-14T21:35:49.8197063Z USE_SYSTEM_NCCL=1 2025-08-14T21:35:49.8197207Z NUM_TEST_SHARDS=1 2025-08-14T21:35:49.8197354Z UCX_HOME=/usr 2025-08-14T21:35:49.8197695Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_f268a462-bc4d-48b6-9f56-069a24ae65c2 2025-08-14T21:35:49.8198287Z JOB_NAME=linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T21:35:49.8198847Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_f268a462-bc4d-48b6-9f56-069a24ae65c2 2025-08-14T21:35:49.8199295Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json 2025-08-14T21:35:49.8199582Z GITHUB_EVENT_NAME=schedule 2025-08-14T21:35:49.8199746Z DASHBOARD_TAG= 2025-08-14T21:35:49.8199902Z GITHUB_RUN_ID=16976338999 2025-08-14T21:35:49.8200073Z INSTALLED_OPENBLAS= 2025-08-14T21:35:49.8200428Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_f268a462-bc4d-48b6-9f56-069a24ae65c2 2025-08-14T21:35:49.8200824Z GITHUB_ACTOR=pytorchmergebot 2025-08-14T21:35:49.8200999Z PR_NUMBER= 2025-08-14T21:35:49.8201145Z DESIRED_CUDA= 2025-08-14T21:35:49.8201291Z GITHUB_RUN_ATTEMPT=1 2025-08-14T21:35:49.8201448Z VALGRIND=ON 2025-08-14T21:35:49.8201806Z ANACONDA_PYTHON_VERSION=3.9 2025-08-14T21:35:49.8202011Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql 2025-08-14T21:35:49.8202221Z TERM=vt100 2025-08-14T21:35:49.8202364Z INSTALLED_VISION=yes 2025-08-14T21:35:49.8202511Z BRANCH=main 2025-08-14T21:35:49.8202660Z SCCACHE_REGION=us-east-1 2025-08-14T21:35:49.8202857Z OPENSSL_ROOT_DIR=/opt/openssl 2025-08-14T21:35:49.8203030Z CUDA_PATH=/usr/local/cuda 2025-08-14T21:35:49.8203339Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux 2025-08-14T21:35:49.8203688Z GITHUB_SERVER_URL=https://github.com 2025-08-14T21:35:49.8203884Z UCC_COMMIT= 2025-08-14T21:35:49.8204032Z REENABLED_ISSUES= 2025-08-14T21:35:49.8204179Z DOCS=yes 2025-08-14T21:35:49.8204317Z SHLVL=1 2025-08-14T21:35:49.8204445Z MAX_JOBS=30 2025-08-14T21:35:49.8204588Z GITHUB_ACTOR_ID=97764156 2025-08-14T21:35:49.8204861Z GITHUB_WORKFLOW_SHA=1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:35:49.8205088Z GITHUB_REF_NAME=main 2025-08-14T21:35:49.8205503Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla 2025-08-14T21:35:49.8205780Z GITHUB_JOB=test 2025-08-14T21:35:49.8205943Z NO_TEST_TIMEOUT=False 2025-08-14T21:35:49.8206120Z TD_DISTRIBUTED=False 2025-08-14T21:35:49.8206307Z GITHUB_REPOSITORY=pytorch/pytorch 2025-08-14T21:35:49.8206513Z GITHUB_RETENTION_DAYS=90 2025-08-14T21:35:49.8206691Z OPENSSL_DIR=/opt/openssl 2025-08-14T21:35:49.8206871Z GITHUB_ACTION_REPOSITORY= 2025-08-14T21:35:49.8207326Z PATH=/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-08-14T21:35:49.8207804Z GITHUB_BASE_REF= 2025-08-14T21:35:49.8207960Z INSTALLED_ACL= 2025-08-14T21:35:49.8208305Z ARTIFACTS_FILE_SUFFIX=test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301909 2025-08-14T21:35:49.8208632Z CI=true 2025-08-14T21:35:49.8208783Z GITHUB_REPOSITORY_OWNER=pytorch 2025-08-14T21:35:49.8209035Z RUST_LOG=sccache::server=error 2025-08-14T21:35:49.8209214Z JOB_ID=48128301909 2025-08-14T21:35:49.8209369Z GITHUB_HEAD_REF= 2025-08-14T21:35:49.8209526Z GITHUB_ACTION_REF= 2025-08-14T21:35:49.8209719Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2 2025-08-14T21:35:49.8209945Z TEST_SHOWLOCALS=False 2025-08-14T21:35:49.8210125Z GITHUB_WORKFLOW=inductor-periodic 2025-08-14T21:35:49.8210317Z DEBIAN_FRONTEND=noninteractive 2025-08-14T21:35:49.8210726Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_f268a462-bc4d-48b6-9f56-069a24ae65c2 2025-08-14T21:35:49.8211125Z NO_TD=False 2025-08-14T21:35:49.8211287Z SKIP_SCCACHE_INITIALIZATION=1 2025-08-14T21:35:49.8211479Z NCCL_INCLUDE_DIR=/usr/local/cuda/include/ 2025-08-14T21:35:49.8211676Z _=/usr/bin/env 2025-08-14T21:35:49.8211833Z + echo 'Testing pytorch' 2025-08-14T21:35:49.8211997Z Testing pytorch 2025-08-14T21:35:49.8212177Z + export LANG=C.UTF-8 2025-08-14T21:35:49.8212343Z + LANG=C.UTF-8 2025-08-14T21:35:49.8212485Z + PR_NUMBER= 2025-08-14T21:35:49.8212684Z + [[ dynamic_cpu_inductor_huggingface == \d\e\f\a\u\l\t ]] 2025-08-14T21:35:49.8212976Z + [[ dynamic_cpu_inductor_huggingface == \d\i\s\t\r\i\b\u\t\e\d ]] 2025-08-14T21:35:49.8213238Z + [[ dynamic_cpu_inductor_huggingface == \s\l\o\w ]] 2025-08-14T21:35:49.8213501Z + [[ linux-jammy-py3.9-gcc11-build == *slow-gradcheck* ]] 2025-08-14T21:35:49.8213759Z + [[ linux-jammy-py3.9-gcc11-build == *cuda* ]] 2025-08-14T21:35:49.8213984Z + [[ linux-jammy-py3.9-gcc11-build == *rocm* ]] 2025-08-14T21:35:49.8214219Z + [[ linux-jammy-py3.9-gcc11-build == *xpu* ]] 2025-08-14T21:35:49.8214460Z + [[ dynamic_cpu_inductor_huggingface == *crossref* ]] 2025-08-14T21:35:49.8214699Z + [[ linux-jammy-py3.9-gcc11-build == *rocm* ]] 2025-08-14T21:35:49.8214916Z + [[ linux-jammy-py3.9-gcc11-build == *xpu* ]] 2025-08-14T21:35:49.8215150Z + [[ linux-jammy-py3.9-gcc11-build != *-bazel-* ]] 2025-08-14T21:35:49.8215374Z + pip_install ninja==1.10.2 2025-08-14T21:35:49.8215609Z + pip_install_pkg='python3 -m pip install --progress-bar off' 2025-08-14T21:35:49.8215942Z + python3 -m pip install --progress-bar off ninja==1.10.2 2025-08-14T21:35:50.1896863Z Collecting ninja==1.10.2 2025-08-14T21:35:50.2033812Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl.metadata (5.0 kB) 2025-08-14T21:35:50.2213952Z Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB) 2025-08-14T21:35:50.9326879Z Installing collected packages: ninja 2025-08-14T21:35:50.9327185Z Attempting uninstall: ninja 2025-08-14T21:35:50.9327402Z Found existing installation: ninja 1.11.1.3 2025-08-14T21:35:50.9353988Z Uninstalling ninja-1.11.1.3: 2025-08-14T21:35:50.9402823Z Successfully uninstalled ninja-1.11.1.3 2025-08-14T21:35:50.9862635Z Successfully installed ninja-1.10.2 2025-08-14T21:35:51.0817236Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-08-14T21:35:51.0818246Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/opt/conda/envs/py_3.9/bin:/opt/conda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 2025-08-14T21:35:51.0818783Z + [[ linux-jammy-py3.9-gcc11-build == *aarch64* ]] 2025-08-14T21:35:51.0819034Z + [[ linux-jammy-py3.9-gcc11-build == *asan* ]] 2025-08-14T21:35:51.0819258Z + [[ linux-jammy-py3.9-gcc11-build == *-debug* ]] 2025-08-14T21:35:51.0819484Z + [[ linux-jammy-py3.9-gcc11-build != *-bazel-* ]] 2025-08-14T21:35:51.0819792Z + echo 'We are not in debug mode: linux-jammy-py3.9-gcc11-build. Expect the assertion to pass' 2025-08-14T21:35:51.0820171Z We are not in debug mode: linux-jammy-py3.9-gcc11-build. Expect the assertion to pass 2025-08-14T21:35:51.0820436Z + cd test 2025-08-14T21:35:51.0820664Z + python -c 'import torch; torch._C._crash_if_debug_asserts_fail(424242)' 2025-08-14T21:35:52.1702582Z + [[ dynamic_cpu_inductor_huggingface == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]] 2025-08-14T21:35:52.1706880Z + [[ dynamic_cpu_inductor_huggingface == \n\o\g\p\u\_\A\V\X\5\1\2 ]] 2025-08-14T21:35:52.1711212Z + [[ dynamic_cpu_inductor_huggingface == \l\e\g\a\c\y\_\n\v\i\d\i\a\_\d\r\i\v\e\r ]] 2025-08-14T21:35:52.1716727Z + DYNAMO_BENCHMARK_FLAGS=() 2025-08-14T21:35:52.1717014Z + [[ dynamic_cpu_inductor_huggingface == *pr_time_benchmarks* ]] 2025-08-14T21:35:52.1717301Z + [[ dynamic_cpu_inductor_huggingface == *dynamo_eager* ]] 2025-08-14T21:35:52.1717573Z + [[ dynamic_cpu_inductor_huggingface == *aot_eager* ]] 2025-08-14T21:35:52.1717820Z + [[ dynamic_cpu_inductor_huggingface == *aot_inductor* ]] 2025-08-14T21:35:52.1718082Z + [[ dynamic_cpu_inductor_huggingface == *max_autotune_inductor* ]] 2025-08-14T21:35:52.1718346Z + [[ dynamic_cpu_inductor_huggingface == *inductor* ]] 2025-08-14T21:35:52.1718583Z + [[ dynamic_cpu_inductor_huggingface != *perf* ]] 2025-08-14T21:35:52.1718848Z + DYNAMO_BENCHMARK_FLAGS+=(--inductor) 2025-08-14T21:35:52.1719077Z + [[ dynamic_cpu_inductor_huggingface == *dynamic* ]] 2025-08-14T21:35:52.1719369Z + DYNAMO_BENCHMARK_FLAGS+=(--dynamic-shapes --dynamic-batch-only) 2025-08-14T21:35:52.1719643Z + [[ dynamic_cpu_inductor_huggingface == *cpu* ]] 2025-08-14T21:35:52.1719875Z + DYNAMO_BENCHMARK_FLAGS+=(--device cpu) 2025-08-14T21:35:52.1936914Z + [[ linux-jammy-py3.9-gcc11-build == *libtorch* ]] 2025-08-14T21:35:52.1937286Z + [[ linux-jammy-py3.9-gcc11-build == *-bazel-* ]] 2025-08-14T21:35:52.1938774Z + cd test 2025-08-14T21:35:52.1944064Z + python -c 'import torch; print(torch.__config__.show())' 2025-08-14T21:35:53.1061584Z PyTorch built with: 2025-08-14T21:35:53.1065975Z - GCC 11.4 2025-08-14T21:35:53.1066325Z - C++ Version: 201703 2025-08-14T21:35:53.1066894Z - Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-08-14T21:35:53.1067467Z - Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-08-14T21:35:53.1067747Z - OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-08-14T21:35:53.1068313Z - LAPACK is enabled (usually provided by MKL) 2025-08-14T21:35:53.1068514Z - NNPACK is enabled 2025-08-14T21:35:53.1068681Z - CPU capability usage: AVX512 2025-08-14T21:35:53.1071329Z - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, COMMIT_SHA=1fc683cf17c8c673044538d10266c00f92987be2, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOCUPTI -DLIBKINETO_NOROCTRACER -DLIBKINETO_NOXPUPTI=ON -DUSE_FBGEMM -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -DC10_NODEPRECATED -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=range-loop-construct -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-unknown-pragmas -Wno-unused-parameter -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wsuggest-override -Wno-psabi -Wno-error=old-style-cast -faligned-new -Werror -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, TORCH_VERSION=2.9.0, USE_CUDA=OFF, USE_CUDNN=OFF, USE_CUSPARSELT=OFF, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_GLOO=ON, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=OFF, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, USE_ROCM_KERNEL_ASSERT=OFF, USE_XCCL=OFF, USE_XPU=OFF, 2025-08-14T21:35:53.1074032Z 2025-08-14T21:35:53.3003131Z + cd test 2025-08-14T21:35:53.3003454Z + python -c 'import torch; print(torch.__config__.parallel_info())' 2025-08-14T21:35:54.2263101Z ATen/Parallel: 2025-08-14T21:35:54.2264849Z at::get_num_threads() : 16 2025-08-14T21:35:54.2265099Z at::get_num_interop_threads() : 16 2025-08-14T21:35:54.2265345Z OpenMP 201511 (a.k.a. OpenMP 4.5) 2025-08-14T21:35:54.2265554Z omp_get_max_threads() : 16 2025-08-14T21:35:54.2265962Z Intel(R) oneAPI Math Kernel Library Version 2024.2-Product Build 20240605 for Intel(R) 64 architecture applications 2025-08-14T21:35:54.2266326Z mkl_get_max_threads() : 16 2025-08-14T21:35:54.2266595Z Intel(R) MKL-DNN v3.7.1 (Git Hash 8d263e693366ef8db40acc569cc7d8edf644556d) 2025-08-14T21:35:54.2266859Z std::thread::hardware_concurrency() : 32 2025-08-14T21:35:54.2267059Z Environment variables: 2025-08-14T21:35:54.2267243Z OMP_NUM_THREADS : [not set] 2025-08-14T21:35:54.2267408Z MKL_NUM_THREADS : [not set] 2025-08-14T21:35:54.2267583Z ATen parallel backend: OpenMP 2025-08-14T21:35:54.2267694Z 2025-08-14T21:35:54.4185993Z + [[ dynamic_cpu_inductor_huggingface == *numpy_2* ]] 2025-08-14T21:35:54.4186346Z + [[ linux-jammy-py3.9-gcc11-build == *aarch64* ]] 2025-08-14T21:35:54.4186598Z + [[ dynamic_cpu_inductor_huggingface == *backward* ]] 2025-08-14T21:35:54.4186841Z + [[ dynamic_cpu_inductor_huggingface == *xla* ]] 2025-08-14T21:35:54.4187104Z + [[ dynamic_cpu_inductor_huggingface == *executorch* ]] 2025-08-14T21:35:54.4187398Z + [[ dynamic_cpu_inductor_huggingface == \j\i\t\_\l\e\g\a\c\y ]] 2025-08-14T21:35:54.4187665Z + [[ linux-jammy-py3.9-gcc11-build == *libtorch* ]] 2025-08-14T21:35:54.4187928Z + [[ dynamic_cpu_inductor_huggingface == distributed ]] 2025-08-14T21:35:54.4188178Z + [[ dynamic_cpu_inductor_huggingface == *operator_benchmark* ]] 2025-08-14T21:35:54.4188463Z + [[ dynamic_cpu_inductor_huggingface == *inductor_distributed* ]] 2025-08-14T21:35:54.4188743Z + [[ dynamic_cpu_inductor_huggingface == *inductor-halide* ]] 2025-08-14T21:35:54.4189025Z + [[ dynamic_cpu_inductor_huggingface == *inductor-triton-cpu* ]] 2025-08-14T21:35:54.4189317Z + [[ dynamic_cpu_inductor_huggingface == *inductor-micro-benchmark* ]] 2025-08-14T21:35:54.4189595Z + [[ dynamic_cpu_inductor_huggingface == *huggingface* ]] 2025-08-14T21:35:54.4189816Z + install_torchvision 2025-08-14T21:35:54.4189975Z + local orig_preload 2025-08-14T21:35:54.4190132Z + local commit 2025-08-14T21:35:54.4190293Z ++ get_pinned_commit vision 2025-08-14T21:35:54.4190473Z ++ cat .github/ci_commit_pins/vision.txt 2025-08-14T21:35:54.4800329Z + commit=966da7e46f65d6d49df3e31214470a4fe5cc8e66 2025-08-14T21:35:54.4800670Z + orig_preload= 2025-08-14T21:35:54.4801203Z + '[' -n '' ']' 2025-08-14T21:35:54.4801382Z + [[ linux-jammy-py3.9-gcc11-build == *cuda* ]] 2025-08-14T21:35:54.4801822Z + pip_build_and_install git+https://github.com/pytorch/vision.git@966da7e46f65d6d49df3e31214470a4fe5cc8e66 dist/vision 2025-08-14T21:35:54.4802334Z + local build_target=git+https://github.com/pytorch/vision.git@966da7e46f65d6d49df3e31214470a4fe5cc8e66 2025-08-14T21:35:54.4802701Z + local wheel_dir=dist/vision 2025-08-14T21:35:54.4802892Z + local found_whl=0 2025-08-14T21:35:54.4803071Z + for file in "${wheel_dir}"/*.whl 2025-08-14T21:35:54.4803370Z + [[ -f dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl ]] 2025-08-14T21:35:54.4803669Z + found_whl=1 2025-08-14T21:35:54.4803813Z + break 2025-08-14T21:35:54.4803948Z + '[' 1 == 0 ']' 2025-08-14T21:35:54.4804103Z + for file in "${wheel_dir}"/*.whl 2025-08-14T21:35:54.4804496Z + pip_install_whl dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl 2025-08-14T21:35:54.4804893Z + args=('dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl') 2025-08-14T21:35:54.4805227Z + local args 2025-08-14T21:35:54.4805728Z + [[ dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl == *\ * ]] 2025-08-14T21:35:54.4806063Z + for path in "${args[@]}" 2025-08-14T21:35:54.4806393Z + echo 'Installing dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl' 2025-08-14T21:35:54.4806788Z Installing dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl 2025-08-14T21:35:54.4807294Z + python3 -mpip install --no-index --no-deps dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl 2025-08-14T21:35:54.7588750Z Processing ./dist/vision/torchvision-0.22.0a0+966da7e-cp39-cp39-linux_x86_64.whl 2025-08-14T21:35:54.7660386Z Installing collected packages: torchvision 2025-08-14T21:35:55.3363924Z Successfully installed torchvision-0.22.0a0+966da7e 2025-08-14T21:35:55.3795235Z + '[' -n '' ']' 2025-08-14T21:35:55.3796879Z + id=0 2025-08-14T21:35:55.3797139Z + test_dynamo_benchmark huggingface 0 2025-08-14T21:35:55.3797454Z ++ pwd 2025-08-14T21:35:55.3797718Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2025-08-14T21:35:55.3798050Z + local suite=huggingface 2025-08-14T21:35:55.3798245Z + shift 2025-08-14T21:35:55.3798427Z + local shard_id=0 2025-08-14T21:35:55.3798600Z + shift 2025-08-14T21:35:55.3798796Z + [[ dynamic_cpu_inductor_huggingface == *perf_compare* ]] 2025-08-14T21:35:55.3799080Z + [[ dynamic_cpu_inductor_huggingface == *perf* ]] 2025-08-14T21:35:55.3799334Z + [[ dynamic_cpu_inductor_huggingface == *cpu* ]] 2025-08-14T21:35:55.3799549Z + local dt=float32 2025-08-14T21:35:55.3799740Z + [[ dynamic_cpu_inductor_huggingface == *amp* ]] 2025-08-14T21:35:55.3800025Z + [[ dynamic_cpu_inductor_huggingface == *freezing* ]] 2025-08-14T21:35:55.3800348Z + test_single_dynamo_benchmark inference huggingface 0 --inference --float32 2025-08-14T21:35:55.3800658Z ++ pwd 2025-08-14T21:35:55.3800879Z + TEST_REPORTS_DIR=/var/lib/jenkins/workspace/test/test-reports 2025-08-14T21:35:55.3801201Z + mkdir -p /var/lib/jenkins/workspace/test/test-reports 2025-08-14T21:35:55.3819453Z + local name=inference 2025-08-14T21:35:55.3820182Z + shift 2025-08-14T21:35:55.3820462Z + local suite=huggingface 2025-08-14T21:35:55.3820702Z + shift 2025-08-14T21:35:55.3820861Z + local shard_id=0 2025-08-14T21:35:55.3821042Z + shift 2025-08-14T21:35:55.3821213Z + partition_flags=() 2025-08-14T21:35:55.3821422Z + local partition_flags 2025-08-14T21:35:55.3821618Z + [[ -n 1 ]] 2025-08-14T21:35:55.3821790Z + [[ -n 0 ]] 2025-08-14T21:35:55.3822102Z + partition_flags=(--total-partitions "$NUM_TEST_SHARDS" --partition-id "$shard_id") 2025-08-14T21:35:55.3822487Z + [[ dynamic_cpu_inductor_huggingface == *perf_compare* ]] 2025-08-14T21:35:55.3822781Z + [[ dynamic_cpu_inductor_huggingface == *perf* ]] 2025-08-14T21:35:55.3823092Z + [[ dynamic_cpu_inductor_huggingface == *_avx2* ]] 2025-08-14T21:35:55.3823358Z + [[ dynamic_cpu_inductor_huggingface == *_avx512* ]] 2025-08-14T21:35:55.3824274Z + python benchmarks/dynamo/huggingface.py --ci --accuracy --timing --explain --print-compilation-time --inductor --dynamic-shapes --dynamic-batch-only --device cpu --inference --float32 --total-partitions 1 --partition-id 0 --output /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv 2025-08-14T21:35:58.6965753Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:35:58.6966856Z from pkg_resources import resource_filename 2025-08-14T21:35:59.2328192Z 2025-08-14T21:35:59.2367825Z config.json: 0% 0.00/694 [00:00bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4029815Z 2025-08-14T21:38:07.4029932Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4030467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4030982Z layer_outputs = layer_module( 2025-08-14T21:38:07.4031351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4031729Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4032171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4032607Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4033045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4033469Z self_outputs = self.self( 2025-08-14T21:38:07.4033888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4034355Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4034880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4035494Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4035759Z 2025-08-14T21:38:07.4035867Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4036408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4036917Z layer_outputs = layer_module( 2025-08-14T21:38:07.4037271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4037778Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4038253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4038711Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4039149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4039665Z self_outputs = self.self( 2025-08-14T21:38:07.4040078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4040533Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4041059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4041666Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4041921Z 2025-08-14T21:38:07.4042039Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4042648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4043162Z layer_outputs = layer_module( 2025-08-14T21:38:07.4043526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4043906Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4044332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4044765Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4045196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4045676Z self_outputs = self.self( 2025-08-14T21:38:07.4046118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4046596Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4047147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4047754Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4048015Z 2025-08-14T21:38:07.4048102Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4048324Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4048544Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4048753Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4049001Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4049512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4049986Z layer_outputs = layer_module( 2025-08-14T21:38:07.4050332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4050714Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4051158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4051564Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4051999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4052436Z self_outputs = self.self( 2025-08-14T21:38:07.4052825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:38:07.4053273Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4053775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4054358Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:38:07.4054877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:38:07.4055409Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:38:07.4055621Z 2025-08-14T21:38:07.4055701Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4055939Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4056442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4056927Z layer_outputs = layer_module( 2025-08-14T21:38:07.4057317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4057683Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4058093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4058513Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4058929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4059336Z self_outputs = self.self( 2025-08-14T21:38:07.4059757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:38:07.4060193Z attn_scores += diagonal_mask 2025-08-14T21:38:07.4060320Z 2025-08-14T21:38:07.4060435Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4060966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4061479Z layer_outputs = layer_module( 2025-08-14T21:38:07.4061843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4062233Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4062676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4063121Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4063563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4063981Z self_outputs = self.self( 2025-08-14T21:38:07.4064380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:38:07.4064804Z attn_probs = nn.functional.softmax( 2025-08-14T21:38:07.4064948Z 2025-08-14T21:38:07.4065062Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4065601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4066080Z layer_outputs = layer_module( 2025-08-14T21:38:07.4066423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4066783Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4067189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4067600Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4068015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4068416Z self_outputs = self.self( 2025-08-14T21:38:07.4068848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4069310Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4069837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4070425Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:38:07.4070878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4071251Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4071412Z 2025-08-14T21:38:07.4071566Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4072115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4072601Z layer_outputs = layer_module( 2025-08-14T21:38:07.4072971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4073356Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4073788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4074233Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4074673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4075111Z self_outputs = self.self( 2025-08-14T21:38:07.4075535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4076020Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4076575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4077140Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:38:07.4077675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:38:07.4078165Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:38:07.4078523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4078878Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4079046Z 2025-08-14T21:38:07.4079158Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4079688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4080202Z layer_outputs = layer_module( 2025-08-14T21:38:07.4080557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4080938Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4081375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4081801Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4082212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4082620Z self_outputs = self.self( 2025-08-14T21:38:07.4083035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4083548Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4084102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4084675Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4084883Z 2025-08-14T21:38:07.4084994Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4085611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4086137Z layer_outputs = layer_module( 2025-08-14T21:38:07.4086546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4086930Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4087342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4087744Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4088145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4088533Z self_outputs = self.self( 2025-08-14T21:38:07.4088918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4089352Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4089855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4090386Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4090589Z 2025-08-14T21:38:07.4090692Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4091184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4091645Z layer_outputs = layer_module( 2025-08-14T21:38:07.4091969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4092316Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4092716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4093118Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4093514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4093906Z self_outputs = self.self( 2025-08-14T21:38:07.4094293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:38:07.4094793Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:38:07.4095027Z 2025-08-14T21:38:07.4095126Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4095620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4096085Z layer_outputs = layer_module( 2025-08-14T21:38:07.4096411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4096757Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4097165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4097607Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4098014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:38:07.4098455Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:38:07.4098895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:38:07.4099315Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4099450Z 2025-08-14T21:38:07.4099548Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4100045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4100556Z layer_outputs = layer_module( 2025-08-14T21:38:07.4100887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4101238Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4101637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4102047Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4102426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4102809Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4103211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4103667Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4104081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:38:07.4104484Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4104613Z 2025-08-14T21:38:07.4104717Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4105187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4105641Z layer_outputs = layer_module( 2025-08-14T21:38:07.4105960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4106294Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4106672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4107079Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4107451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4107832Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4108225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4108666Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4109103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:38:07.4109544Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:38:07.4109913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:38:07.4110245Z return self.act(input) 2025-08-14T21:38:07.4110352Z 2025-08-14T21:38:07.4110460Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4110950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4111473Z layer_outputs = layer_module( 2025-08-14T21:38:07.4111809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4112164Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4112557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4112977Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4113352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4113714Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4114141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:38:07.4114588Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:38:07.4115025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:38:07.4115416Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4115553Z 2025-08-14T21:38:07.4115650Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4116129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4116589Z layer_outputs = layer_module( 2025-08-14T21:38:07.4116906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4117244Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4117631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4118025Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4118405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4118790Z self_outputs = self.self( 2025-08-14T21:38:07.4119164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-14T21:38:07.4119554Z query_vectors = self.query(hidden_states) 2025-08-14T21:38:07.4119691Z 2025-08-14T21:38:07.4119789Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4120268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4120724Z layer_outputs = layer_module( 2025-08-14T21:38:07.4121042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4121384Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4121791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4122192Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4122588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4122982Z self_outputs = self.self( 2025-08-14T21:38:07.4123363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4123781Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4124263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4124872Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4125106Z 2025-08-14T21:38:07.4125218Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4125837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4126352Z layer_outputs = layer_module( 2025-08-14T21:38:07.4126714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4127100Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4127580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4128001Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4128412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4128818Z self_outputs = self.self( 2025-08-14T21:38:07.4129200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-14T21:38:07.4129610Z key_vectors = self.key(hidden_states) 2025-08-14T21:38:07.4129740Z 2025-08-14T21:38:07.4129846Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4130332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4130799Z layer_outputs = layer_module( 2025-08-14T21:38:07.4131139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4131492Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4131896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4132303Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4132715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4133127Z self_outputs = self.self( 2025-08-14T21:38:07.4133526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4133953Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4134435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4135001Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4135238Z 2025-08-14T21:38:07.4135339Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4135826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4136291Z layer_outputs = layer_module( 2025-08-14T21:38:07.4136615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4136970Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4137376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4137926Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4138341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4138816Z self_outputs = self.self( 2025-08-14T21:38:07.4139191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4139604Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4140061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4140608Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4140835Z 2025-08-14T21:38:07.4140943Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4141474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4141930Z layer_outputs = layer_module( 2025-08-14T21:38:07.4142261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4142607Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4142996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4143394Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4143786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4144177Z self_outputs = self.self( 2025-08-14T21:38:07.4144548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4144966Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4145432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4145984Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4146211Z 2025-08-14T21:38:07.4146289Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4146492Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4146690Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4146877Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4147098Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4147588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4148045Z layer_outputs = layer_module( 2025-08-14T21:38:07.4148371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4148715Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4149115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4149506Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4149903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4150294Z self_outputs = self.self( 2025-08-14T21:38:07.4150671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:38:07.4151095Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4151585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4152103Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:38:07.4152639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:38:07.4153142Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:38:07.4153339Z 2025-08-14T21:38:07.4153414Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4153641Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4154125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4154574Z layer_outputs = layer_module( 2025-08-14T21:38:07.4154902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4155284Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4155671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4156068Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4156457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4156843Z self_outputs = self.self( 2025-08-14T21:38:07.4157211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:38:07.4157604Z attn_scores += diagonal_mask 2025-08-14T21:38:07.4157723Z 2025-08-14T21:38:07.4157829Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4158314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4158763Z layer_outputs = layer_module( 2025-08-14T21:38:07.4159099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4159454Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4159847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4160250Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4160661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4161060Z self_outputs = self.self( 2025-08-14T21:38:07.4161438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:38:07.4161854Z attn_probs = nn.functional.softmax( 2025-08-14T21:38:07.4161986Z 2025-08-14T21:38:07.4162094Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4162599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4163075Z layer_outputs = layer_module( 2025-08-14T21:38:07.4163437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4163822Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4164241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4164650Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4165060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4165535Z self_outputs = self.self( 2025-08-14T21:38:07.4165955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-14T21:38:07.4166477Z value_vectors = self.value(hidden_states) 2025-08-14T21:38:07.4166632Z 2025-08-14T21:38:07.4166742Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4167266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4167724Z layer_outputs = layer_module( 2025-08-14T21:38:07.4168065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4168417Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4168822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4169250Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4169656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4170067Z self_outputs = self.self( 2025-08-14T21:38:07.4170448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4170899Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4171415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4171989Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:38:07.4172392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4172740Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4172902Z 2025-08-14T21:38:07.4173002Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4173506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4173974Z layer_outputs = layer_module( 2025-08-14T21:38:07.4174309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4174662Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4175062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4175471Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4175882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4176283Z self_outputs = self.self( 2025-08-14T21:38:07.4176659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4177112Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4177622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4178155Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:38:07.4178639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:38:07.4179102Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:38:07.4179434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4179777Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4179921Z 2025-08-14T21:38:07.4180060Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4180559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4181030Z layer_outputs = layer_module( 2025-08-14T21:38:07.4181355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4181702Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4182103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4182507Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4182963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4183362Z self_outputs = self.self( 2025-08-14T21:38:07.4183745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4184189Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4184687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4185235Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4185427Z 2025-08-14T21:38:07.4185530Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4186005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4186451Z layer_outputs = layer_module( 2025-08-14T21:38:07.4186776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4187121Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4187504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4187899Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4188290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4188676Z self_outputs = self.self( 2025-08-14T21:38:07.4189042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4189474Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4189967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4190491Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4190680Z 2025-08-14T21:38:07.4190776Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4191254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4191705Z layer_outputs = layer_module( 2025-08-14T21:38:07.4192036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4192379Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4192782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4193188Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4193603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4194034Z self_outputs = self.self( 2025-08-14T21:38:07.4194409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:38:07.4194908Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:38:07.4195131Z 2025-08-14T21:38:07.4195229Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4195715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4196171Z layer_outputs = layer_module( 2025-08-14T21:38:07.4196528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4196864Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4197258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4197651Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4198042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:38:07.4198462Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:38:07.4198885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:38:07.4199284Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4199411Z 2025-08-14T21:38:07.4199506Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4199988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4200442Z layer_outputs = layer_module( 2025-08-14T21:38:07.4200766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4201095Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4201481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4201882Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4202261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4202623Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4203017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4203444Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4203861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:38:07.4204255Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4204388Z 2025-08-14T21:38:07.4204485Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4204959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4205479Z layer_outputs = layer_module( 2025-08-14T21:38:07.4205865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4206268Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4206725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4207178Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4207620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4207992Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4208390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4208881Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4209362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:38:07.4209854Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:38:07.4210302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:38:07.4210679Z return self.act(input) 2025-08-14T21:38:07.4210809Z 2025-08-14T21:38:07.4210922Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4211474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4211924Z layer_outputs = layer_module( 2025-08-14T21:38:07.4212250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4212590Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4212985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4213379Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4213762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4214134Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4214520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:38:07.4214963Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:38:07.4215398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:38:07.4215800Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4215929Z 2025-08-14T21:38:07.4216026Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4216508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4216964Z layer_outputs = layer_module( 2025-08-14T21:38:07.4217291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4217627Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4218021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4218416Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4218811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4219208Z self_outputs = self.self( 2025-08-14T21:38:07.4219598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-14T21:38:07.4220012Z query_vectors = self.query(hidden_states) 2025-08-14T21:38:07.4220145Z 2025-08-14T21:38:07.4220245Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4220744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4221266Z layer_outputs = layer_module( 2025-08-14T21:38:07.4221603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4221945Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4222355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4222759Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4223152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4223544Z self_outputs = self.self( 2025-08-14T21:38:07.4223963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4224382Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4224853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4225403Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4225636Z 2025-08-14T21:38:07.4225734Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4226216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4226660Z layer_outputs = layer_module( 2025-08-14T21:38:07.4226987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4227327Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4227719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4228108Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4228497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4228888Z self_outputs = self.self( 2025-08-14T21:38:07.4229251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-14T21:38:07.4229642Z key_vectors = self.key(hidden_states) 2025-08-14T21:38:07.4229774Z 2025-08-14T21:38:07.4229871Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4230366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4230839Z layer_outputs = layer_module( 2025-08-14T21:38:07.4231180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4231542Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4231965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4232442Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4232856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4233267Z self_outputs = self.self( 2025-08-14T21:38:07.4233653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4234093Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4234589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4235210Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4235450Z 2025-08-14T21:38:07.4235551Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4236058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4236533Z layer_outputs = layer_module( 2025-08-14T21:38:07.4236874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4237273Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4237812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4238314Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4238736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4239138Z self_outputs = self.self( 2025-08-14T21:38:07.4239528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4239967Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4240462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4241038Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4241283Z 2025-08-14T21:38:07.4241388Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4241899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4242380Z layer_outputs = layer_module( 2025-08-14T21:38:07.4242712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4243073Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4243481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4243885Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4244297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4244705Z self_outputs = self.self( 2025-08-14T21:38:07.4245099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4245597Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4246137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4246750Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4246988Z 2025-08-14T21:38:07.4247079Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4247294Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4247498Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4247701Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4247919Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4248422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4248907Z layer_outputs = layer_module( 2025-08-14T21:38:07.4249248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4249644Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4250040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4250440Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4250833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4251229Z self_outputs = self.self( 2025-08-14T21:38:07.4251613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:38:07.4252042Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4252551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4253085Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:38:07.4253596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:38:07.4254110Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:38:07.4254309Z 2025-08-14T21:38:07.4254386Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4254619Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4255116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4255578Z layer_outputs = layer_module( 2025-08-14T21:38:07.4255906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4256255Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4256657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4257055Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4257458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4257854Z self_outputs = self.self( 2025-08-14T21:38:07.4258238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:38:07.4258630Z attn_scores += diagonal_mask 2025-08-14T21:38:07.4258869Z 2025-08-14T21:38:07.4258994Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4259569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4260340Z layer_outputs = layer_module( 2025-08-14T21:38:07.4260756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4261171Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4261681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4262123Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4262582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4263069Z self_outputs = self.self( 2025-08-14T21:38:07.4263487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:38:07.4263941Z attn_probs = nn.functional.softmax( 2025-08-14T21:38:07.4264183Z 2025-08-14T21:38:07.4264301Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4264848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4265337Z layer_outputs = layer_module( 2025-08-14T21:38:07.4265752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4266157Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4266622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4267061Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4267630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4268089Z self_outputs = self.self( 2025-08-14T21:38:07.4268544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-14T21:38:07.4268991Z value_vectors = self.value(hidden_states) 2025-08-14T21:38:07.4269165Z 2025-08-14T21:38:07.4269299Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4269851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4270377Z layer_outputs = layer_module( 2025-08-14T21:38:07.4270750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4271180Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4289622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4290284Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4290736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4291158Z self_outputs = self.self( 2025-08-14T21:38:07.4291572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4292051Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4292569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4293134Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:38:07.4293558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4293910Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4294064Z 2025-08-14T21:38:07.4294180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4294681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4295163Z layer_outputs = layer_module( 2025-08-14T21:38:07.4295509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4295860Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4296271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4296686Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4297096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4297636Z self_outputs = self.self( 2025-08-14T21:38:07.4298027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4298476Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4298991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4299515Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:38:07.4300012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:38:07.4300468Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:38:07.4300856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4301198Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4301351Z 2025-08-14T21:38:07.4301456Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4301958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4302427Z layer_outputs = layer_module( 2025-08-14T21:38:07.4302771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4303125Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4303525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4303931Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4304340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4304736Z self_outputs = self.self( 2025-08-14T21:38:07.4305125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4305621Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4306114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4306632Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4306836Z 2025-08-14T21:38:07.4306936Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4307420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4307873Z layer_outputs = layer_module( 2025-08-14T21:38:07.4308195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4308538Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4308930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4309327Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4309711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4310096Z self_outputs = self.self( 2025-08-14T21:38:07.4310473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4310906Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4311411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4311998Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4312201Z 2025-08-14T21:38:07.4312313Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4312813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4313290Z layer_outputs = layer_module( 2025-08-14T21:38:07.4313633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4313994Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4314451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4314859Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4315254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4315646Z self_outputs = self.self( 2025-08-14T21:38:07.4316016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:38:07.4316513Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:38:07.4316740Z 2025-08-14T21:38:07.4316848Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4317330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4317804Z layer_outputs = layer_module( 2025-08-14T21:38:07.4318140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4318494Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4318894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4319312Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4319724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:38:07.4320172Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:38:07.4320659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:38:07.4321118Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4321265Z 2025-08-14T21:38:07.4321384Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4321922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4322432Z layer_outputs = layer_module( 2025-08-14T21:38:07.4322795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4323185Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4323615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4324065Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4324494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4324912Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4325347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4326002Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4326504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:38:07.4326971Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4327111Z 2025-08-14T21:38:07.4327219Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4327741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4328228Z layer_outputs = layer_module( 2025-08-14T21:38:07.4328571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4328988Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4329400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4329826Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4330217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4330611Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4331022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4331475Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4331910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:38:07.4332362Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:38:07.4332746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:38:07.4333088Z return self.act(input) 2025-08-14T21:38:07.4333206Z 2025-08-14T21:38:07.4333311Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4333823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4334308Z layer_outputs = layer_module( 2025-08-14T21:38:07.4334647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4335007Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4335420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4335848Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4336240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4336637Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4337050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:38:07.4337514Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:38:07.4338135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:38:07.4338572Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4338708Z 2025-08-14T21:38:07.4338817Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4339303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4339776Z layer_outputs = layer_module( 2025-08-14T21:38:07.4340114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4340591Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4341001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4341411Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4341816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4342225Z self_outputs = self.self( 2025-08-14T21:38:07.4342607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-14T21:38:07.4343016Z query_vectors = self.query(hidden_states) 2025-08-14T21:38:07.4343194Z 2025-08-14T21:38:07.4343306Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4343803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4344282Z layer_outputs = layer_module( 2025-08-14T21:38:07.4344619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4344975Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4345371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4345778Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4346184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4346588Z self_outputs = self.self( 2025-08-14T21:38:07.4346970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4347404Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4347889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4348454Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4348690Z 2025-08-14T21:38:07.4348810Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4349311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4349780Z layer_outputs = layer_module( 2025-08-14T21:38:07.4350112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4350462Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4350875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4351287Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4351682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4352085Z self_outputs = self.self( 2025-08-14T21:38:07.4352471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-14T21:38:07.4352885Z key_vectors = self.key(hidden_states) 2025-08-14T21:38:07.4353015Z 2025-08-14T21:38:07.4353117Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4353618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4354132Z layer_outputs = layer_module( 2025-08-14T21:38:07.4354462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4354813Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4355219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4355629Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4356012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4356401Z self_outputs = self.self( 2025-08-14T21:38:07.4356773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4357220Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4357689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4358235Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4358463Z 2025-08-14T21:38:07.4358566Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4359045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4359496Z layer_outputs = layer_module( 2025-08-14T21:38:07.4359818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4360154Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4360539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4360934Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4361332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4361712Z self_outputs = self.self( 2025-08-14T21:38:07.4362087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4362510Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4362986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4363530Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4363763Z 2025-08-14T21:38:07.4363860Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4364342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4364796Z layer_outputs = layer_module( 2025-08-14T21:38:07.4365128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4365552Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4365965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4366366Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4366782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4367202Z self_outputs = self.self( 2025-08-14T21:38:07.4367604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4368080Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4368562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4369125Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4369350Z 2025-08-14T21:38:07.4369438Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4369635Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4369834Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4370027Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4370238Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4370752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4371214Z layer_outputs = layer_module( 2025-08-14T21:38:07.4371539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4371886Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4372282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4372676Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4373067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4373461Z self_outputs = self.self( 2025-08-14T21:38:07.4373850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:38:07.4374277Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4374754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4375271Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:38:07.4375772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:38:07.4376283Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:38:07.4376480Z 2025-08-14T21:38:07.4376559Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4376792Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4377284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4377737Z layer_outputs = layer_module( 2025-08-14T21:38:07.4378074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4378424Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4378822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4379214Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4379488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4379557Z self_outputs = self.self( 2025-08-14T21:38:07.4379827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:38:07.4379899Z attn_scores += diagonal_mask 2025-08-14T21:38:07.4379905Z 2025-08-14T21:38:07.4380007Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4380372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4380439Z layer_outputs = layer_module( 2025-08-14T21:38:07.4380646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4380727Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4380992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4381069Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4381328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4381435Z self_outputs = self.self( 2025-08-14T21:38:07.4381709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:38:07.4381786Z attn_probs = nn.functional.softmax( 2025-08-14T21:38:07.4381790Z 2025-08-14T21:38:07.4381894Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4382230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4382295Z layer_outputs = layer_module( 2025-08-14T21:38:07.4382509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4382581Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4382851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4382924Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4383187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4383262Z self_outputs = self.self( 2025-08-14T21:38:07.4383522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-14T21:38:07.4383602Z value_vectors = self.value(hidden_states) 2025-08-14T21:38:07.4383605Z 2025-08-14T21:38:07.4383707Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4384038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4384111Z layer_outputs = layer_module( 2025-08-14T21:38:07.4384320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4384394Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4384664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4384736Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4385005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4385071Z self_outputs = self.self( 2025-08-14T21:38:07.4385331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4385449Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4385780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4385948Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:38:07.4387318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4387414Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4387418Z 2025-08-14T21:38:07.4387524Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4387861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4387931Z layer_outputs = layer_module( 2025-08-14T21:38:07.4388154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4388230Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4388543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4388617Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4388885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4388962Z self_outputs = self.self( 2025-08-14T21:38:07.4389226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4389344Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4389677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4389806Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:38:07.4390114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:38:07.4390201Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:38:07.4390388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4390492Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4390496Z 2025-08-14T21:38:07.4390597Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4390944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4391015Z layer_outputs = layer_module( 2025-08-14T21:38:07.4391227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4391310Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4391581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4391662Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4391937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4392006Z self_outputs = self.self( 2025-08-14T21:38:07.4392291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4392398Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4392739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4392887Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4392891Z 2025-08-14T21:38:07.4392989Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4393326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4393441Z layer_outputs = layer_module( 2025-08-14T21:38:07.4393653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4393725Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4393977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4394055Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4394307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4394371Z self_outputs = self.self( 2025-08-14T21:38:07.4394664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4394771Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4395104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4395241Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4395244Z 2025-08-14T21:38:07.4395732Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4396064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4396131Z layer_outputs = layer_module( 2025-08-14T21:38:07.4396341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4396416Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4396672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4396753Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4397011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4397084Z self_outputs = self.self( 2025-08-14T21:38:07.4397338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:38:07.4397508Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:38:07.4397512Z 2025-08-14T21:38:07.4397612Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4397936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4398008Z layer_outputs = layer_module( 2025-08-14T21:38:07.4398208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4398279Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4398541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4398609Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4398866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:38:07.4398975Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:38:07.4399235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:38:07.4399319Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4399356Z 2025-08-14T21:38:07.4399450Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4399767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4399842Z layer_outputs = layer_module( 2025-08-14T21:38:07.4400040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4400116Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4400375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4400456Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4400738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4400813Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4401090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4401194Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4401462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:38:07.4401548Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4401552Z 2025-08-14T21:38:07.4401650Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4401986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4402064Z layer_outputs = layer_module( 2025-08-14T21:38:07.4402278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4402362Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4402694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4402771Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4403022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4403095Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4403367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4403467Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4403730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:38:07.4403843Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:38:07.4404044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:38:07.4404112Z return self.act(input) 2025-08-14T21:38:07.4404123Z 2025-08-14T21:38:07.4404218Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4404549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4404622Z layer_outputs = layer_module( 2025-08-14T21:38:07.4404827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4404899Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4405166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4405243Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4405611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4405698Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4405994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:38:07.4406130Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:38:07.4406427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:38:07.4406524Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4406528Z 2025-08-14T21:38:07.4406636Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4407034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4407120Z layer_outputs = layer_module( 2025-08-14T21:38:07.4407332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4407410Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4407686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4407760Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4408040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4408114Z self_outputs = self.self( 2025-08-14T21:38:07.4408409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-14T21:38:07.4408503Z query_vectors = self.query(hidden_states) 2025-08-14T21:38:07.4408510Z 2025-08-14T21:38:07.4408616Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4408984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4409058Z layer_outputs = layer_module( 2025-08-14T21:38:07.4409285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4409375Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4409665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4409752Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4410058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4410130Z self_outputs = self.self( 2025-08-14T21:38:07.4410428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4410536Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4410889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4411090Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4411093Z 2025-08-14T21:38:07.4411198Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4411568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4411646Z layer_outputs = layer_module( 2025-08-14T21:38:07.4411874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4412002Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4412308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4412398Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4412712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4412787Z self_outputs = self.self( 2025-08-14T21:38:07.4413096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-14T21:38:07.4413181Z key_vectors = self.key(hidden_states) 2025-08-14T21:38:07.4413185Z 2025-08-14T21:38:07.4413332Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4413696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4413765Z layer_outputs = layer_module( 2025-08-14T21:38:07.4413977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4414049Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4414311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4414390Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4414654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4414726Z self_outputs = self.self( 2025-08-14T21:38:07.4414993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4415091Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4415416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4415586Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4415590Z 2025-08-14T21:38:07.4415693Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4416027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4416093Z layer_outputs = layer_module( 2025-08-14T21:38:07.4416306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4416377Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4416651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4416722Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4416986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4417058Z self_outputs = self.self( 2025-08-14T21:38:07.4417319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4417412Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4417739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4417913Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4417949Z 2025-08-14T21:38:07.4418054Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4418390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4418456Z layer_outputs = layer_module( 2025-08-14T21:38:07.4418669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4418761Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4419024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4419094Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4419395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4419461Z self_outputs = self.self( 2025-08-14T21:38:07.4419735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4419830Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4420147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4420323Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4420326Z 2025-08-14T21:38:07.4420404Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4420477Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4420556Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4420628Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4420734Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4421067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4421136Z layer_outputs = layer_module( 2025-08-14T21:38:07.4421349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4421423Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4421680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4421760Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4422020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4422092Z self_outputs = self.self( 2025-08-14T21:38:07.4422353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:38:07.4422460Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4422781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4422914Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:38:07.4423223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:38:07.4423362Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:38:07.4423366Z 2025-08-14T21:38:07.4423440Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4423542Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4423870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4423975Z layer_outputs = layer_module( 2025-08-14T21:38:07.4424185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4424258Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4424532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4424604Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4424869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4424943Z self_outputs = self.self( 2025-08-14T21:38:07.4425237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:38:07.4425313Z attn_scores += diagonal_mask 2025-08-14T21:38:07.4425319Z 2025-08-14T21:38:07.4425415Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4425743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4425818Z layer_outputs = layer_module( 2025-08-14T21:38:07.4426024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4426105Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4426366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4426438Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4426710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4426778Z self_outputs = self.self( 2025-08-14T21:38:07.4427043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:38:07.4427117Z attn_probs = nn.functional.softmax( 2025-08-14T21:38:07.4427120Z 2025-08-14T21:38:07.4427216Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4427550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4427617Z layer_outputs = layer_module( 2025-08-14T21:38:07.4427822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4427902Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4428167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4428247Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4428508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4428572Z self_outputs = self.self( 2025-08-14T21:38:07.4428839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-14T21:38:07.4428917Z value_vectors = self.value(hidden_states) 2025-08-14T21:38:07.4428920Z 2025-08-14T21:38:07.4429023Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4429346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4429413Z layer_outputs = layer_module( 2025-08-14T21:38:07.4429627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4429740Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4430001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4430078Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4430336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4430409Z self_outputs = self.self( 2025-08-14T21:38:07.4430667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4430775Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4431140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4431310Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:38:07.4431502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4431599Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4431603Z 2025-08-14T21:38:07.4431703Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4432051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4432130Z layer_outputs = layer_module( 2025-08-14T21:38:07.4432345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4432421Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4432684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4432766Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4433029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4433095Z self_outputs = self.self( 2025-08-14T21:38:07.4433363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4433471Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4433810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4433941Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:38:07.4434240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:38:07.4434335Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:38:07.4434514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4434614Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4434617Z 2025-08-14T21:38:07.4434713Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4435043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4435127Z layer_outputs = layer_module( 2025-08-14T21:38:07.4435328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4435409Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4435667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4435770Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4436040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4436104Z self_outputs = self.self( 2025-08-14T21:38:07.4436364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4436477Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4436809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4436986Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4436991Z 2025-08-14T21:38:07.4437087Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4437425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4437501Z layer_outputs = layer_module( 2025-08-14T21:38:07.4437853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4437941Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4438207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4438280Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4438560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4438626Z self_outputs = self.self( 2025-08-14T21:38:07.4438897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4439008Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4439339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4439487Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4439491Z 2025-08-14T21:38:07.4439589Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4439933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4440003Z layer_outputs = layer_module( 2025-08-14T21:38:07.4440216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4440304Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4440573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4440646Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4440926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4440993Z self_outputs = self.self( 2025-08-14T21:38:07.4441268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:38:07.4441446Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:38:07.4441450Z 2025-08-14T21:38:07.4441551Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4441902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4442053Z layer_outputs = layer_module( 2025-08-14T21:38:07.4442275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4442351Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4442625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4442707Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4442981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:38:07.4443096Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:38:07.4443419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:38:07.4443506Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4443510Z 2025-08-14T21:38:07.4443619Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4443967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4444037Z layer_outputs = layer_module( 2025-08-14T21:38:07.4444258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4444335Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4444619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4444704Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4444960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4445046Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4445325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4445482Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4445776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:38:07.4445864Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4445868Z 2025-08-14T21:38:07.4445988Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4446368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4446451Z layer_outputs = layer_module( 2025-08-14T21:38:07.4446691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4446775Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4447087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4447166Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4447416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4447498Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4447777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4447890Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4448165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:38:07.4448324Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:38:07.4448566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:38:07.4448644Z return self.act(input) 2025-08-14T21:38:07.4448648Z 2025-08-14T21:38:07.4448765Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4449143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4449220Z layer_outputs = layer_module( 2025-08-14T21:38:07.4449461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4449589Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4449895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4449994Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4450274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4450366Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4450671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:38:07.4450803Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:38:07.4451125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:38:07.4451212Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4451219Z 2025-08-14T21:38:07.4451339Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4451717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4451793Z layer_outputs = layer_module( 2025-08-14T21:38:07.4452035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4452118Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4452424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4452506Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4452818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4452904Z self_outputs = self.self( 2025-08-14T21:38:07.4453203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-14T21:38:07.4453293Z query_vectors = self.query(hidden_states) 2025-08-14T21:38:07.4453305Z 2025-08-14T21:38:07.4453415Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4453797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4453879Z layer_outputs = layer_module( 2025-08-14T21:38:07.4454114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4454185Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4454446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4454517Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4454778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4454874Z self_outputs = self.self( 2025-08-14T21:38:07.4455131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4455230Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4455546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4455722Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4455725Z 2025-08-14T21:38:07.4455818Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4456188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4456264Z layer_outputs = layer_module( 2025-08-14T21:38:07.4456466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4456536Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4456797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4456866Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4457127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4457192Z self_outputs = self.self( 2025-08-14T21:38:07.4457447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-14T21:38:07.4457530Z key_vectors = self.key(hidden_states) 2025-08-14T21:38:07.4457533Z 2025-08-14T21:38:07.4457632Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4457967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4458034Z layer_outputs = layer_module( 2025-08-14T21:38:07.4458239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4458320Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4458626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4458700Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4458957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4459020Z self_outputs = self.self( 2025-08-14T21:38:07.4459289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4459384Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4459701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4459877Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4459880Z 2025-08-14T21:38:07.4459975Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4460308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4460375Z layer_outputs = layer_module( 2025-08-14T21:38:07.4460582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4460692Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4460950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4461028Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4461293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4461357Z self_outputs = self.self( 2025-08-14T21:38:07.4461627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4461720Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4462074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4462259Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4462263Z 2025-08-14T21:38:07.4462358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4462692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4462758Z layer_outputs = layer_module( 2025-08-14T21:38:07.4462967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4463039Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4463298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4463377Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4463644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4463713Z self_outputs = self.self( 2025-08-14T21:38:07.4463987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4464078Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4464405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4464572Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4464576Z 2025-08-14T21:38:07.4464651Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4464735Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4464808Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4464880Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4464985Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4465318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4465402Z layer_outputs = layer_module( 2025-08-14T21:38:07.4465607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4465679Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4465948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4466017Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4466284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4466347Z self_outputs = self.self( 2025-08-14T21:38:07.4466635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:38:07.4466744Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4467061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4467199Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:38:07.4467504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:38:07.4467644Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:38:07.4467648Z 2025-08-14T21:38:07.4467758Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4467856Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4468187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4468260Z layer_outputs = layer_module( 2025-08-14T21:38:07.4468466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4468544Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4468805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4468875Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4469152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4469218Z self_outputs = self.self( 2025-08-14T21:38:07.4469476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:38:07.4469545Z attn_scores += diagonal_mask 2025-08-14T21:38:07.4469549Z 2025-08-14T21:38:07.4469640Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4469970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4470035Z layer_outputs = layer_module( 2025-08-14T21:38:07.4470246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4470319Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4470577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4470661Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4470925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4470993Z self_outputs = self.self( 2025-08-14T21:38:07.4471267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:38:07.4471343Z attn_probs = nn.functional.softmax( 2025-08-14T21:38:07.4471346Z 2025-08-14T21:38:07.4471448Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4471775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4471841Z layer_outputs = layer_module( 2025-08-14T21:38:07.4472056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4472127Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4472429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4472500Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4472758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4472830Z self_outputs = self.self( 2025-08-14T21:38:07.4473087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-14T21:38:07.4473165Z value_vectors = self.value(hidden_states) 2025-08-14T21:38:07.4473177Z 2025-08-14T21:38:07.4473271Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4473631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4473707Z layer_outputs = layer_module( 2025-08-14T21:38:07.4473927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4474000Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4474280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4474351Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4474627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4474692Z self_outputs = self.self( 2025-08-14T21:38:07.4474963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4475085Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4475423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4475598Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:38:07.4475785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4475881Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4475885Z 2025-08-14T21:38:07.4475989Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4476327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4476394Z layer_outputs = layer_module( 2025-08-14T21:38:07.4476616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4476690Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4476983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4477053Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4477313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4477383Z self_outputs = self.self( 2025-08-14T21:38:07.4477644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4477759Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4478092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4478218Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:38:07.4478559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:38:07.4478645Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:38:07.4478831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4478922Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4478925Z 2025-08-14T21:38:07.4479022Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4479357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4479426Z layer_outputs = layer_module( 2025-08-14T21:38:07.4479662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4479745Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4480008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4480087Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4480347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4480413Z self_outputs = self.self( 2025-08-14T21:38:07.4480678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4480785Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4481116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4481257Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4481263Z 2025-08-14T21:38:07.4481359Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4481699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4481768Z layer_outputs = layer_module( 2025-08-14T21:38:07.4481985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4482060Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4482325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4482406Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4482677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4482747Z self_outputs = self.self( 2025-08-14T21:38:07.4483019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4483127Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4483468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4483611Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4483615Z 2025-08-14T21:38:07.4483709Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4484052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4484120Z layer_outputs = layer_module( 2025-08-14T21:38:07.4484371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4484446Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4484713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4484793Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4485067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4485138Z self_outputs = self.self( 2025-08-14T21:38:07.4485456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:38:07.4485687Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:38:07.4485692Z 2025-08-14T21:38:07.4485813Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4486182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4486264Z layer_outputs = layer_module( 2025-08-14T21:38:07.4486493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4486575Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4486873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4486953Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4487240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:38:07.4487354Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:38:07.4487620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:38:07.4487705Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4487709Z 2025-08-14T21:38:07.4487805Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4488135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4488210Z layer_outputs = layer_module( 2025-08-14T21:38:07.4488415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4488495Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4488757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4488839Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4489089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4489161Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4489424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4489533Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4489794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:38:07.4489874Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4489878Z 2025-08-14T21:38:07.4489973Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4490311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4490416Z layer_outputs = layer_module( 2025-08-14T21:38:07.4490640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4490716Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4490979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4491055Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4491309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4491383Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4491693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4491798Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4492066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:38:07.4492180Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:38:07.4492386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:38:07.4492454Z return self.act(input) 2025-08-14T21:38:07.4492465Z 2025-08-14T21:38:07.4492562Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4492895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4492970Z layer_outputs = layer_module( 2025-08-14T21:38:07.4493183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4493259Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4493535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4493613Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4493870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4493943Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4494219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:38:07.4494344Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:38:07.4494622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:38:07.4494706Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4494712Z 2025-08-14T21:38:07.4494809Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4495144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4495219Z layer_outputs = layer_module( 2025-08-14T21:38:07.4495430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4495505Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4495780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4495855Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4496133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4496242Z self_outputs = self.self( 2025-08-14T21:38:07.4496512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-14T21:38:07.4496596Z query_vectors = self.query(hidden_states) 2025-08-14T21:38:07.4496599Z 2025-08-14T21:38:07.4496696Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4497038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4497106Z layer_outputs = layer_module( 2025-08-14T21:38:07.4497318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4497401Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4497700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4497778Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4498053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4498120Z self_outputs = self.self( 2025-08-14T21:38:07.4498394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4498491Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4498827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4499012Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4499016Z 2025-08-14T21:38:07.4499116Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4499463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4499535Z layer_outputs = layer_module( 2025-08-14T21:38:07.4499747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4499830Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4500104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4500193Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4500449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4500513Z self_outputs = self.self( 2025-08-14T21:38:07.4500777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-14T21:38:07.4500853Z key_vectors = self.key(hidden_states) 2025-08-14T21:38:07.4500857Z 2025-08-14T21:38:07.4500958Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4501290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4501357Z layer_outputs = layer_module( 2025-08-14T21:38:07.4501574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4501645Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4501900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4501980Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4502234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4502337Z self_outputs = self.self( 2025-08-14T21:38:07.4502595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4502688Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4503009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4503176Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4503179Z 2025-08-14T21:38:07.4503278Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4503628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4503698Z layer_outputs = layer_module( 2025-08-14T21:38:07.4503905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4503975Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4504238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4504308Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4504559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4504631Z self_outputs = self.self( 2025-08-14T21:38:07.4504882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4504976Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4505297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4505463Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4505467Z 2025-08-14T21:38:07.4505567Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4505885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4505950Z layer_outputs = layer_module( 2025-08-14T21:38:07.4506156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4506226Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4506486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4506557Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4506818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4506890Z self_outputs = self.self( 2025-08-14T21:38:07.4507147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4507243Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4507556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4507722Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4507726Z 2025-08-14T21:38:07.4507810Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4507885Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4507992Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4508071Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4508167Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4508501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4508566Z layer_outputs = layer_module( 2025-08-14T21:38:07.4508772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4508851Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4509118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4509216Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4509476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4509542Z self_outputs = self.self( 2025-08-14T21:38:07.4509802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:38:07.4509905Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4510215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4510360Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:38:07.4510668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:38:07.4510821Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:38:07.4510825Z 2025-08-14T21:38:07.4510903Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4511000Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4511342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4511411Z layer_outputs = layer_module( 2025-08-14T21:38:07.4511632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4511710Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4511984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4512068Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4512341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4512408Z self_outputs = self.self( 2025-08-14T21:38:07.4512676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:38:07.4512745Z attn_scores += diagonal_mask 2025-08-14T21:38:07.4512748Z 2025-08-14T21:38:07.4512851Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4513178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4513245Z layer_outputs = layer_module( 2025-08-14T21:38:07.4513456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4513529Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4513799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4513904Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4514168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4514244Z self_outputs = self.self( 2025-08-14T21:38:07.4514502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:38:07.4514578Z attn_probs = nn.functional.softmax( 2025-08-14T21:38:07.4514589Z 2025-08-14T21:38:07.4514683Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4515007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4515111Z layer_outputs = layer_module( 2025-08-14T21:38:07.4515319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4515395Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4515663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4515735Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4516005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4516069Z self_outputs = self.self( 2025-08-14T21:38:07.4516327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-14T21:38:07.4516413Z value_vectors = self.value(hidden_states) 2025-08-14T21:38:07.4516417Z 2025-08-14T21:38:07.4516513Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4516845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4516913Z layer_outputs = layer_module( 2025-08-14T21:38:07.4517119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4517202Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4517461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4517533Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4517800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4517864Z self_outputs = self.self( 2025-08-14T21:38:07.4518130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4518242Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4518572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4518745Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:38:07.4518932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4519034Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4519037Z 2025-08-14T21:38:07.4519135Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4519477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4519556Z layer_outputs = layer_module( 2025-08-14T21:38:07.4519770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4519932Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4520213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4520290Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4520577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4520645Z self_outputs = self.self( 2025-08-14T21:38:07.4520923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4521047Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4521432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4521578Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:38:07.4521898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:38:07.4521990Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:38:07.4522185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4522283Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4522287Z 2025-08-14T21:38:07.4522396Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4522746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4522818Z layer_outputs = layer_module( 2025-08-14T21:38:07.4523050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4523128Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4523411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4523488Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4523764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4523838Z self_outputs = self.self( 2025-08-14T21:38:07.4524111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4524228Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4524585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4524734Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4524738Z 2025-08-14T21:38:07.4524845Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4525193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4525264Z layer_outputs = layer_module( 2025-08-14T21:38:07.4525702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4525791Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4526103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4526183Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4526527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4526610Z self_outputs = self.self( 2025-08-14T21:38:07.4526899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4527027Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4527390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4527539Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4527543Z 2025-08-14T21:38:07.4527652Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4528037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4528123Z layer_outputs = layer_module( 2025-08-14T21:38:07.4528345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4528423Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4528713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4528789Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4529080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4529154Z self_outputs = self.self( 2025-08-14T21:38:07.4529427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:38:07.4529617Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:38:07.4529623Z 2025-08-14T21:38:07.4529722Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4530058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4530132Z layer_outputs = layer_module( 2025-08-14T21:38:07.4530343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4530428Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4530697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4530772Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4531049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:38:07.4531158Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:38:07.4531436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:38:07.4531516Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4531520Z 2025-08-14T21:38:07.4531617Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4531960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4532028Z layer_outputs = layer_module( 2025-08-14T21:38:07.4532243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4532327Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4532597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4532718Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4532965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4533039Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4533323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4533424Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4533689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:38:07.4533797Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4533801Z 2025-08-14T21:38:07.4533900Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4534244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4534312Z layer_outputs = layer_module( 2025-08-14T21:38:07.4534528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4534601Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4534867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4534952Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4535202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4535279Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4535555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4535661Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4535934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:38:07.4536042Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:38:07.4536246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:38:07.4536322Z return self.act(input) 2025-08-14T21:38:07.4536326Z 2025-08-14T21:38:07.4536423Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4536769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4536839Z layer_outputs = layer_module( 2025-08-14T21:38:07.4537051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4537134Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4537400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4537478Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4537904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4537983Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4538268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:38:07.4538390Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:38:07.4538663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:38:07.4538821Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4538825Z 2025-08-14T21:38:07.4538924Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4539266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4539335Z layer_outputs = layer_module( 2025-08-14T21:38:07.4539544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4539627Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4539892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4540030Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4540305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4540377Z self_outputs = self.self( 2025-08-14T21:38:07.4540650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-14T21:38:07.4540729Z query_vectors = self.query(hidden_states) 2025-08-14T21:38:07.4540733Z 2025-08-14T21:38:07.4540830Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4541174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4541242Z layer_outputs = layer_module( 2025-08-14T21:38:07.4541459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4541533Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4541804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4541886Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4542154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4542226Z self_outputs = self.self( 2025-08-14T21:38:07.4542492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4542588Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4542919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4543096Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4543102Z 2025-08-14T21:38:07.4543206Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4543541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4543611Z layer_outputs = layer_module( 2025-08-14T21:38:07.4543827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4543901Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4544170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4544251Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4544518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4544593Z self_outputs = self.self( 2025-08-14T21:38:07.4544892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-14T21:38:07.4544969Z key_vectors = self.key(hidden_states) 2025-08-14T21:38:07.4544972Z 2025-08-14T21:38:07.4545079Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4545416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4545493Z layer_outputs = layer_module( 2025-08-14T21:38:07.4545703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4545778Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4546087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4546165Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4546446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4546512Z self_outputs = self.self( 2025-08-14T21:38:07.4546779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4546883Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4547214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4547391Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4547402Z 2025-08-14T21:38:07.4547502Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4547841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4547922Z layer_outputs = layer_module( 2025-08-14T21:38:07.4548135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4548208Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4548496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4548567Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4548848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4548912Z self_outputs = self.self( 2025-08-14T21:38:07.4549177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4549281Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4549601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4549777Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4549780Z 2025-08-14T21:38:07.4549876Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4550208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4550284Z layer_outputs = layer_module( 2025-08-14T21:38:07.4550492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4550571Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4550867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4550939Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4551206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4551270Z self_outputs = self.self( 2025-08-14T21:38:07.4551531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4551636Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4551963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4552194Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4552201Z 2025-08-14T21:38:07.4552279Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4552353Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4552435Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4552506Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4552603Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4552949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4553018Z layer_outputs = layer_module( 2025-08-14T21:38:07.4553236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4553312Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4553591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4553674Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4553936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4554007Z self_outputs = self.self( 2025-08-14T21:38:07.4554268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:38:07.4554370Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4554695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4554827Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:38:07.4555137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:38:07.4555282Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:38:07.4555285Z 2025-08-14T21:38:07.4555358Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4555461Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4555789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4555856Z layer_outputs = layer_module( 2025-08-14T21:38:07.4556074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4556148Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4556420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4556490Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4556751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4556857Z self_outputs = self.self( 2025-08-14T21:38:07.4557117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:38:07.4557193Z attn_scores += diagonal_mask 2025-08-14T21:38:07.4557196Z 2025-08-14T21:38:07.4557291Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4557618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4557695Z layer_outputs = layer_module( 2025-08-14T21:38:07.4557927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4558000Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4558271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4558341Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4558608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4558673Z self_outputs = self.self( 2025-08-14T21:38:07.4558931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:38:07.4559012Z attn_probs = nn.functional.softmax( 2025-08-14T21:38:07.4559016Z 2025-08-14T21:38:07.4559112Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4559450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4559517Z layer_outputs = layer_module( 2025-08-14T21:38:07.4559726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4559806Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4560067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4560145Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4560409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4560474Z self_outputs = self.self( 2025-08-14T21:38:07.4560747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-14T21:38:07.4560831Z value_vectors = self.value(hidden_states) 2025-08-14T21:38:07.4560834Z 2025-08-14T21:38:07.4560932Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4561293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4561359Z layer_outputs = layer_module( 2025-08-14T21:38:07.4561572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4561645Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4561909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4561989Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4562254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4562327Z self_outputs = self.self( 2025-08-14T21:38:07.4562588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4562730Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4563069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4563232Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:38:07.4563420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4563515Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4563519Z 2025-08-14T21:38:07.4563619Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4564001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4564074Z layer_outputs = layer_module( 2025-08-14T21:38:07.4564284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4564367Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4564638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4564718Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4564986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4565052Z self_outputs = self.self( 2025-08-14T21:38:07.4565327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4565493Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4565856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4565987Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:38:07.4566296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:38:07.4566394Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:38:07.4566582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4566686Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4566690Z 2025-08-14T21:38:07.4566789Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4567138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4567220Z layer_outputs = layer_module( 2025-08-14T21:38:07.4567441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4567518Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4567807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4567884Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4568172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4568240Z self_outputs = self.self( 2025-08-14T21:38:07.4568517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4568637Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4569025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4569176Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4569179Z 2025-08-14T21:38:07.4569277Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4569614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4569691Z layer_outputs = layer_module( 2025-08-14T21:38:07.4569903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4570015Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4570286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4570361Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4570636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4570704Z self_outputs = self.self( 2025-08-14T21:38:07.4570969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4571085Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4571421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4571574Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4571577Z 2025-08-14T21:38:07.4571675Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4572012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4572087Z layer_outputs = layer_module( 2025-08-14T21:38:07.4572298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4572378Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4572653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4572726Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4573002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4573072Z self_outputs = self.self( 2025-08-14T21:38:07.4573344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:38:07.4573526Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:38:07.4573530Z 2025-08-14T21:38:07.4573628Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4573970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4574037Z layer_outputs = layer_module( 2025-08-14T21:38:07.4574253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4574326Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4574596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4574675Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4574975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:38:07.4575082Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:38:07.4575358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:38:07.4575438Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4575441Z 2025-08-14T21:38:07.4575546Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4575882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4575951Z layer_outputs = layer_module( 2025-08-14T21:38:07.4576199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4576280Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4576556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4576637Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4576886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4576967Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4577241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4577346Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4577622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:38:07.4577700Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4577706Z 2025-08-14T21:38:07.4577812Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4578147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4578216Z layer_outputs = layer_module( 2025-08-14T21:38:07.4578438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4578513Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4578786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4578866Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4579120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4579203Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4579472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4579583Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4579856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:38:07.4579962Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:38:07.4580168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:38:07.4580235Z return self.act(input) 2025-08-14T21:38:07.4580238Z 2025-08-14T21:38:07.4580332Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4580667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4580768Z layer_outputs = layer_module( 2025-08-14T21:38:07.4580978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4581050Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4581313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4581395Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4581639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4581715Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4582007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:38:07.4582124Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:38:07.4582397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:38:07.4582474Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4582478Z 2025-08-14T21:38:07.4582579Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4582909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4582977Z layer_outputs = layer_module( 2025-08-14T21:38:07.4583187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4583259Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4583523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4583604Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4583866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4583940Z self_outputs = self.self( 2025-08-14T21:38:07.4584200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-14T21:38:07.4584274Z query_vectors = self.query(hidden_states) 2025-08-14T21:38:07.4584278Z 2025-08-14T21:38:07.4584379Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4584707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4584783Z layer_outputs = layer_module( 2025-08-14T21:38:07.4584988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4585065Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4585333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4585404Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4585666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4585738Z self_outputs = self.self( 2025-08-14T21:38:07.4585998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4586100Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4586417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4586629Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4586633Z 2025-08-14T21:38:07.4586734Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4587069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4587142Z layer_outputs = layer_module( 2025-08-14T21:38:07.4587351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4587423Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4587696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4587797Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4588070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4588138Z self_outputs = self.self( 2025-08-14T21:38:07.4588406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-14T21:38:07.4588487Z key_vectors = self.key(hidden_states) 2025-08-14T21:38:07.4588491Z 2025-08-14T21:38:07.4588586Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4588934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4589000Z layer_outputs = layer_module( 2025-08-14T21:38:07.4589211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4589292Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4589560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4589633Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4589910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4589975Z self_outputs = self.self( 2025-08-14T21:38:07.4590246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4590342Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4590670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4590853Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4590856Z 2025-08-14T21:38:07.4590955Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4591301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4591368Z layer_outputs = layer_module( 2025-08-14T21:38:07.4591577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4591658Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4591927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4592005Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4592276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4592341Z self_outputs = self.self( 2025-08-14T21:38:07.4592639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4592732Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4593050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4593227Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4593230Z 2025-08-14T21:38:07.4593326Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4593658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4593757Z layer_outputs = layer_module( 2025-08-14T21:38:07.4593966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4594049Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4594312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4594390Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4594652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4594717Z self_outputs = self.self( 2025-08-14T21:38:07.4594986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4595080Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4595405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4595576Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4595579Z 2025-08-14T21:38:07.4595654Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4595735Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4595809Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4595882Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4595985Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4596316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4596389Z layer_outputs = layer_module( 2025-08-14T21:38:07.4596596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4596673Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4596948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4597022Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4597287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4597358Z self_outputs = self.self( 2025-08-14T21:38:07.4597620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:38:07.4597729Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4598045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4598181Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:38:07.4598492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:38:07.4598667Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:38:07.4598674Z 2025-08-14T21:38:07.4598756Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4598850Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4599185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4599261Z layer_outputs = layer_module( 2025-08-14T21:38:07.4599473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4599554Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4599854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4599932Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4600211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4600281Z self_outputs = self.self( 2025-08-14T21:38:07.4600549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:38:07.4600628Z attn_scores += diagonal_mask 2025-08-14T21:38:07.4600631Z 2025-08-14T21:38:07.4600728Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4601082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4601154Z layer_outputs = layer_module( 2025-08-14T21:38:07.4601372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4601458Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4601735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4601815Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4602093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4602161Z self_outputs = self.self( 2025-08-14T21:38:07.4602447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:38:07.4602526Z attn_probs = nn.functional.softmax( 2025-08-14T21:38:07.4602529Z 2025-08-14T21:38:07.4602637Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4602988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4603062Z layer_outputs = layer_module( 2025-08-14T21:38:07.4603285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4603361Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4603640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4603722Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4604000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4604075Z self_outputs = self.self( 2025-08-14T21:38:07.4604354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-14T21:38:07.4604471Z value_vectors = self.value(hidden_states) 2025-08-14T21:38:07.4604474Z 2025-08-14T21:38:07.4604578Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4604926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4605000Z layer_outputs = layer_module( 2025-08-14T21:38:07.4605217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4605294Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4605644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4605723Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4606038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4606124Z self_outputs = self.self( 2025-08-14T21:38:07.4606412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4606542Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4606920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4607100Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:38:07.4607314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4607409Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4607413Z 2025-08-14T21:38:07.4607521Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4607865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4607934Z layer_outputs = layer_module( 2025-08-14T21:38:07.4608146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4608218Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4608489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4608558Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4608822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4608895Z self_outputs = self.self( 2025-08-14T21:38:07.4609157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4609268Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4609605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4609732Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:38:07.4610036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:38:07.4610121Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:38:07.4610299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4610402Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4610408Z 2025-08-14T21:38:07.4610505Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4610899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4610971Z layer_outputs = layer_module( 2025-08-14T21:38:07.4611189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4611276Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4611556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4611640Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4611917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4612016Z self_outputs = self.self( 2025-08-14T21:38:07.4612305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4612419Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4612755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4612913Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4612917Z 2025-08-14T21:38:07.4613018Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4613366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4613439Z layer_outputs = layer_module( 2025-08-14T21:38:07.4613661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4613747Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4614022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4614104Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4614381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4614450Z self_outputs = self.self( 2025-08-14T21:38:07.4614738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4614847Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4615189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4615333Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4615339Z 2025-08-14T21:38:07.4615439Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4615778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4615848Z layer_outputs = layer_module( 2025-08-14T21:38:07.4616063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4616139Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4616406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4616488Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4616758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4616856Z self_outputs = self.self( 2025-08-14T21:38:07.4617127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:38:07.4617300Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:38:07.4617303Z 2025-08-14T21:38:07.4617403Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4617734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4617799Z layer_outputs = layer_module( 2025-08-14T21:38:07.4618011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4618124Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4618394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4618468Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4618727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:38:07.4618838Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:38:07.4619098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:38:07.4619183Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4619187Z 2025-08-14T21:38:07.4619280Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4619607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4619681Z layer_outputs = layer_module( 2025-08-14T21:38:07.4619885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4619954Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4620219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4620296Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4620546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4620620Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4620881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4620993Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4621254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:38:07.4621339Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4621342Z 2025-08-14T21:38:07.4621436Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4621759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4621831Z layer_outputs = layer_module( 2025-08-14T21:38:07.4622036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4622114Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4622376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4622452Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4622732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4622804Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4623067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4623174Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4623433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:38:07.4623545Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:38:07.4623743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:38:07.4623811Z return self.act(input) 2025-08-14T21:38:07.4623842Z 2025-08-14T21:38:07.4623947Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4624282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4624357Z layer_outputs = layer_module( 2025-08-14T21:38:07.4624562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4624635Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4624908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4624983Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4625227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4625308Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4625577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:38:07.4625700Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:38:07.4625967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:38:07.4626044Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4626047Z 2025-08-14T21:38:07.4626149Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4626476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4626550Z layer_outputs = layer_module( 2025-08-14T21:38:07.4626761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4626834Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4627109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4627181Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4627452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4627522Z self_outputs = self.self( 2025-08-14T21:38:07.4627788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-14T21:38:07.4627875Z query_vectors = self.query(hidden_states) 2025-08-14T21:38:07.4627878Z 2025-08-14T21:38:07.4627974Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4628311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4628391Z layer_outputs = layer_module( 2025-08-14T21:38:07.4628642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4628723Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4629002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4629073Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4629341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4629406Z self_outputs = self.self( 2025-08-14T21:38:07.4629672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4629803Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4630124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4630314Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4630318Z 2025-08-14T21:38:07.4630414Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4630760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4630827Z layer_outputs = layer_module( 2025-08-14T21:38:07.4631036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4631116Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4631394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4631469Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4631738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4631803Z self_outputs = self.self( 2025-08-14T21:38:07.4632074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-14T21:38:07.4632146Z key_vectors = self.key(hidden_states) 2025-08-14T21:38:07.4632150Z 2025-08-14T21:38:07.4632241Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4632566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4632630Z layer_outputs = layer_module( 2025-08-14T21:38:07.4632836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4632910Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4633165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4633242Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4633496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4633568Z self_outputs = self.self( 2025-08-14T21:38:07.4633821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4633913Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4634231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4634396Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4634431Z 2025-08-14T21:38:07.4634526Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4634850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4634914Z layer_outputs = layer_module( 2025-08-14T21:38:07.4635125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4635195Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4635457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4635564Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4635824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4635896Z self_outputs = self.self( 2025-08-14T21:38:07.4636149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4636240Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4636556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4636718Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4636721Z 2025-08-14T21:38:07.4636820Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4637140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4637204Z layer_outputs = layer_module( 2025-08-14T21:38:07.4637411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4637481Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4637849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4637931Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4638186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4638258Z self_outputs = self.self( 2025-08-14T21:38:07.4638515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4638613Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4638936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4639108Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4639111Z 2025-08-14T21:38:07.4639197Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4639271Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4639344Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4639421Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4639518Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4639846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4639921Z layer_outputs = layer_module( 2025-08-14T21:38:07.4640129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4640273Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4640533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4640604Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4640877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4640942Z self_outputs = self.self( 2025-08-14T21:38:07.4641217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:38:07.4641325Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4641704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4641857Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:38:07.4642181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:38:07.4642339Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:38:07.4642343Z 2025-08-14T21:38:07.4642422Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4642524Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4642883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4642953Z layer_outputs = layer_module( 2025-08-14T21:38:07.4643166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4643248Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4643518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4643598Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4643866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4643934Z self_outputs = self.self( 2025-08-14T21:38:07.4644218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:38:07.4644291Z attn_scores += diagonal_mask 2025-08-14T21:38:07.4644295Z 2025-08-14T21:38:07.4644403Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4644755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4644827Z layer_outputs = layer_module( 2025-08-14T21:38:07.4645051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4645128Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4645462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4645561Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4645857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4645936Z self_outputs = self.self( 2025-08-14T21:38:07.4646229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:38:07.4646317Z attn_probs = nn.functional.softmax( 2025-08-14T21:38:07.4646321Z 2025-08-14T21:38:07.4646435Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4646863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4646938Z layer_outputs = layer_module( 2025-08-14T21:38:07.4647149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4647223Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4647496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4647571Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4647880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4647950Z self_outputs = self.self( 2025-08-14T21:38:07.4648219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-14T21:38:07.4648310Z value_vectors = self.value(hidden_states) 2025-08-14T21:38:07.4648314Z 2025-08-14T21:38:07.4648411Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4648750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4648826Z layer_outputs = layer_module( 2025-08-14T21:38:07.4649035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4649116Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4649386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4649459Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4649737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4649803Z self_outputs = self.self( 2025-08-14T21:38:07.4650134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4650250Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4650590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4650764Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:38:07.4650954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4651056Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4651062Z 2025-08-14T21:38:07.4651160Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4651499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4651574Z layer_outputs = layer_module( 2025-08-14T21:38:07.4651783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4651857Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4652130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4652203Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4652480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4652546Z self_outputs = self.self( 2025-08-14T21:38:07.4652846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4652963Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4653304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4653440Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:38:07.4653744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:38:07.4653830Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:38:07.4654062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4654160Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4654166Z 2025-08-14T21:38:07.4654273Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4654616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4654687Z layer_outputs = layer_module( 2025-08-14T21:38:07.4654910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4654987Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4655260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4655342Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4655615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4655695Z self_outputs = self.self( 2025-08-14T21:38:07.4655970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4656083Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4656432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4656580Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4656584Z 2025-08-14T21:38:07.4656691Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4657037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4657119Z layer_outputs = layer_module( 2025-08-14T21:38:07.4657337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4657414Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4657686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4657760Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4658027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4658101Z self_outputs = self.self( 2025-08-14T21:38:07.4658364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4658474Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4658813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4658986Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4658990Z 2025-08-14T21:38:07.4659092Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4659419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4659486Z layer_outputs = layer_module( 2025-08-14T21:38:07.4659702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4659775Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4660078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4660150Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4660416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4660486Z self_outputs = self.self( 2025-08-14T21:38:07.4660746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:38:07.4660925Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:38:07.4660928Z 2025-08-14T21:38:07.4661021Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4661345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4661419Z layer_outputs = layer_module( 2025-08-14T21:38:07.4661624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4661705Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4661963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4662033Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4662301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:38:07.4662405Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:38:07.4662665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:38:07.4662751Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4662755Z 2025-08-14T21:38:07.4662849Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4663182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4663251Z layer_outputs = layer_module( 2025-08-14T21:38:07.4663456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4663536Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4663796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4663882Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4664122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4664194Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4664466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4664599Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4664862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:38:07.4664947Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4664950Z 2025-08-14T21:38:07.4665044Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4665377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4665443Z layer_outputs = layer_module( 2025-08-14T21:38:07.4665648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4665729Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4666023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4666112Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4666354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4666425Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4666693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4666793Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4667060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:38:07.4667166Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:38:07.4667366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:38:07.4667438Z return self.act(input) 2025-08-14T21:38:07.4667444Z 2025-08-14T21:38:07.4667537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4667863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4667937Z layer_outputs = layer_module( 2025-08-14T21:38:07.4668140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4668221Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4668480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4668558Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4668806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4668878Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4669148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:38:07.4669262Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:38:07.4669527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:38:07.4669609Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4669612Z 2025-08-14T21:38:07.4669705Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4670038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4670107Z layer_outputs = layer_module( 2025-08-14T21:38:07.4670310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4670426Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4670695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4670767Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4671045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4671112Z self_outputs = self.self( 2025-08-14T21:38:07.4671389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-14T21:38:07.4671467Z query_vectors = self.query(hidden_states) 2025-08-14T21:38:07.4671470Z 2025-08-14T21:38:07.4671595Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4671942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4672013Z layer_outputs = layer_module( 2025-08-14T21:38:07.4672232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4672307Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4672583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4672666Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4672949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4673017Z self_outputs = self.self( 2025-08-14T21:38:07.4673298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4673398Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4673729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4673906Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4673910Z 2025-08-14T21:38:07.4674017Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4674352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4674420Z layer_outputs = layer_module( 2025-08-14T21:38:07.4674635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4674712Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4674974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4675055Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4675322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4675397Z self_outputs = self.self( 2025-08-14T21:38:07.4675662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-14T21:38:07.4675739Z key_vectors = self.key(hidden_states) 2025-08-14T21:38:07.4675742Z 2025-08-14T21:38:07.4675845Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4676191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4676265Z layer_outputs = layer_module( 2025-08-14T21:38:07.4676516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4676590Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4676866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4676937Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4677204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4677278Z self_outputs = self.self( 2025-08-14T21:38:07.4677544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4677677Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4678010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4678188Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4678192Z 2025-08-14T21:38:07.4678297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4678630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4678704Z layer_outputs = layer_module( 2025-08-14T21:38:07.4678913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4678989Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4679261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4679333Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4679602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4679675Z self_outputs = self.self( 2025-08-14T21:38:07.4679940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4680042Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4680367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4680538Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4680549Z 2025-08-14T21:38:07.4680649Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4680982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4681056Z layer_outputs = layer_module( 2025-08-14T21:38:07.4681267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4681341Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4681614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4681688Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4681959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4682026Z self_outputs = self.self( 2025-08-14T21:38:07.4682292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4682426Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4682757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4682936Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4682939Z 2025-08-14T21:38:07.4683016Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4683093Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4683173Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4683246Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4683342Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4683716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4683787Z layer_outputs = layer_module( 2025-08-14T21:38:07.4684011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4684085Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4684350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4684431Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4684697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4684762Z self_outputs = self.self( 2025-08-14T21:38:07.4685037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:38:07.4685144Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4685577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4685739Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:38:07.4686094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:38:07.4686258Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:38:07.4686263Z 2025-08-14T21:38:07.4686344Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4686456Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4686823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4686894Z layer_outputs = layer_module( 2025-08-14T21:38:07.4687115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4687192Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4687469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4687542Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4687838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4687915Z self_outputs = self.self( 2025-08-14T21:38:07.4688184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:38:07.4688255Z attn_scores += diagonal_mask 2025-08-14T21:38:07.4688267Z 2025-08-14T21:38:07.4688367Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4688703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4688816Z layer_outputs = layer_module( 2025-08-14T21:38:07.4689026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4689101Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4689377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4689449Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4689726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4689793Z self_outputs = self.self( 2025-08-14T21:38:07.4690091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:38:07.4690181Z attn_probs = nn.functional.softmax( 2025-08-14T21:38:07.4690184Z 2025-08-14T21:38:07.4690282Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4690632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4690703Z layer_outputs = layer_module( 2025-08-14T21:38:07.4690921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4691004Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4691269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4691338Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4691619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4691688Z self_outputs = self.self( 2025-08-14T21:38:07.4691959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-14T21:38:07.4692038Z value_vectors = self.value(hidden_states) 2025-08-14T21:38:07.4692041Z 2025-08-14T21:38:07.4692134Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4692467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4692533Z layer_outputs = layer_module( 2025-08-14T21:38:07.4692744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4692821Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4693086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4693166Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4693428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4693494Z self_outputs = self.self( 2025-08-14T21:38:07.4693761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4693871Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4694209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4694373Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:38:07.4694556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4694687Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4694690Z 2025-08-14T21:38:07.4694783Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4695119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4695187Z layer_outputs = layer_module( 2025-08-14T21:38:07.4695393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4695473Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4695736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4695843Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4696113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4696182Z self_outputs = self.self( 2025-08-14T21:38:07.4696451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4696561Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4696894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4697028Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:38:07.4697332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:38:07.4697426Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:38:07.4697606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4697700Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4697704Z 2025-08-14T21:38:07.4697807Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4698134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4698207Z layer_outputs = layer_module( 2025-08-14T21:38:07.4698417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4698490Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4698759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4698832Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4699102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4699169Z self_outputs = self.self( 2025-08-14T21:38:07.4699430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4699541Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4699869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4700010Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4700022Z 2025-08-14T21:38:07.4700117Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4700447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4700562Z layer_outputs = layer_module( 2025-08-14T21:38:07.4700772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4700846Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4701123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4701195Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4701468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4701534Z self_outputs = self.self( 2025-08-14T21:38:07.4701833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4701949Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4702286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4702436Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4702440Z 2025-08-14T21:38:07.4702537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4702876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4702950Z layer_outputs = layer_module( 2025-08-14T21:38:07.4703155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4703233Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4703494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4703567Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4703833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4703897Z self_outputs = self.self( 2025-08-14T21:38:07.4704154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:38:07.4704339Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:38:07.4704342Z 2025-08-14T21:38:07.4704437Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4704771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4704839Z layer_outputs = layer_module( 2025-08-14T21:38:07.4705044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4705123Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4705384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4705463Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4705724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:38:07.4705827Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:38:07.4706104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:38:07.4706181Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4706184Z 2025-08-14T21:38:07.4706282Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4706633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4706697Z layer_outputs = layer_module( 2025-08-14T21:38:07.4706902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4706973Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4707226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4707309Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4707547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4707655Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4707914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4708016Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4708277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:38:07.4708351Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4708354Z 2025-08-14T21:38:07.4708453Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4708769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4708834Z layer_outputs = layer_module( 2025-08-14T21:38:07.4709040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4709109Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4709374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4709449Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4709684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4709762Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4710022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4710120Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4710382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:38:07.4710486Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:38:07.4710687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:38:07.4710756Z return self.act(input) 2025-08-14T21:38:07.4710759Z 2025-08-14T21:38:07.4710850Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4711178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4711245Z layer_outputs = layer_module( 2025-08-14T21:38:07.4711454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4711528Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4711799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4711883Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4712117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4712220Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4712481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:38:07.4712592Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:38:07.4712855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:38:07.4712930Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4712933Z 2025-08-14T21:38:07.4713028Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4713394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4713466Z layer_outputs = layer_module( 2025-08-14T21:38:07.4713679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4713752Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4714015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4714095Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4714359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4714432Z self_outputs = self.self( 2025-08-14T21:38:07.4714696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 509, in forward 2025-08-14T21:38:07.4714774Z query_vectors = self.query(hidden_states) 2025-08-14T21:38:07.4714777Z 2025-08-14T21:38:07.4714877Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4715208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4715275Z layer_outputs = layer_module( 2025-08-14T21:38:07.4715490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4715565Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4715836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4715906Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4716170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4716242Z self_outputs = self.self( 2025-08-14T21:38:07.4716505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4716607Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4716927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4717102Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4717106Z 2025-08-14T21:38:07.4717206Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4717542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4717614Z layer_outputs = layer_module( 2025-08-14T21:38:07.4717823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4717929Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4718206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4718279Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4718545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4718619Z self_outputs = self.self( 2025-08-14T21:38:07.4718883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 510, in forward 2025-08-14T21:38:07.4718965Z key_vectors = self.key(hidden_states) 2025-08-14T21:38:07.4718969Z 2025-08-14T21:38:07.4719066Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4719436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4719512Z layer_outputs = layer_module( 2025-08-14T21:38:07.4719720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4719798Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4720061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4720132Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4720428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4720494Z self_outputs = self.self( 2025-08-14T21:38:07.4720767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4720871Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4721204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4721384Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4721388Z 2025-08-14T21:38:07.4721486Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4721824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4721901Z layer_outputs = layer_module( 2025-08-14T21:38:07.4722116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4722201Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4722474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4722549Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4722826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4722892Z self_outputs = self.self( 2025-08-14T21:38:07.4723168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4723265Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4723596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4723780Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4723783Z 2025-08-14T21:38:07.4723881Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4724262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4724331Z layer_outputs = layer_module( 2025-08-14T21:38:07.4724544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4724627Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4724898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4724970Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4725313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4725384Z self_outputs = self.self( 2025-08-14T21:38:07.4725779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 524, in forward 2025-08-14T21:38:07.4725894Z attn_scores = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4726270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 796, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4726467Z diagonal_chunked_attention_scores = torch.einsum("bcxd,bcyd->bcxy", (query, key)) # multiply 2025-08-14T21:38:07.4726471Z 2025-08-14T21:38:07.4726557Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4726651Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4726734Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4726816Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4726945Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4727303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4727377Z layer_outputs = layer_module( 2025-08-14T21:38:07.4727619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4727695Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4727979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4728053Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4728328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4728403Z self_outputs = self.self( 2025-08-14T21:38:07.4728679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 536, in forward 2025-08-14T21:38:07.4728794Z diagonal_mask = self._sliding_chunks_query_key_matmul( 2025-08-14T21:38:07.4729132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 834, in _sliding_chunks_query_key_matmul 2025-08-14T21:38:07.4729267Z self._mask_invalid_locations(diagonal_attention_scores, window_overlap) 2025-08-14T21:38:07.4729593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 762, in _mask_invalid_locations 2025-08-14T21:38:07.4729739Z input_tensor[:, :affected_seq_len, :, : affected_seq_len + 1] = torch.full_like( 2025-08-14T21:38:07.4729743Z 2025-08-14T21:38:07.4729826Z cudagraph partition due to non gpu ops 2025-08-14T21:38:07.4729926Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4730274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4730391Z layer_outputs = layer_module( 2025-08-14T21:38:07.4730605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4730681Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4730960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4731032Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4731310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4731378Z self_outputs = self.self( 2025-08-14T21:38:07.4731644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 541, in forward 2025-08-14T21:38:07.4731753Z attn_scores += diagonal_mask 2025-08-14T21:38:07.4731757Z 2025-08-14T21:38:07.4731860Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4732202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4732269Z layer_outputs = layer_module( 2025-08-14T21:38:07.4732480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4732560Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4732829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4732901Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4733181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4733247Z self_outputs = self.self( 2025-08-14T21:38:07.4733520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 579, in forward 2025-08-14T21:38:07.4733599Z attn_probs = nn.functional.softmax( 2025-08-14T21:38:07.4733603Z 2025-08-14T21:38:07.4733699Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4734044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4734109Z layer_outputs = layer_module( 2025-08-14T21:38:07.4734326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4734399Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4734672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4734752Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4735026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4735098Z self_outputs = self.self( 2025-08-14T21:38:07.4735367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 511, in forward 2025-08-14T21:38:07.4735447Z value_vectors = self.value(hidden_states) 2025-08-14T21:38:07.4735450Z 2025-08-14T21:38:07.4735552Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4735892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4735959Z layer_outputs = layer_module( 2025-08-14T21:38:07.4736180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4736254Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4736564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4736637Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4736906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4736980Z self_outputs = self.self( 2025-08-14T21:38:07.4737247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4737365Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4737918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 863, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4738095Z padded_value = nn.functional.pad(value, (0, 0, window_overlap, window_overlap), value=-1) 2025-08-14T21:38:07.4738296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4738397Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4738401Z 2025-08-14T21:38:07.4738510Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4738847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4738920Z layer_outputs = layer_module( 2025-08-14T21:38:07.4739143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4739222Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4739497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4739586Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4739860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4739939Z self_outputs = self.self( 2025-08-14T21:38:07.4740219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4740333Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4740687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 876, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4740822Z chunked_attn_probs = self._pad_and_diagonalize(chunked_attn_probs) 2025-08-14T21:38:07.4741148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 699, in _pad_and_diagonalize 2025-08-14T21:38:07.4741241Z chunked_hidden_states = nn.functional.pad( 2025-08-14T21:38:07.4741429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/nn/functional.py", line 5294, in pad 2025-08-14T21:38:07.4741537Z return torch._C._nn.pad(input, pad, mode, value) 2025-08-14T21:38:07.4741541Z 2025-08-14T21:38:07.4741641Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4741988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4742061Z layer_outputs = layer_module( 2025-08-14T21:38:07.4742278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4742362Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4742638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4742779Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4743060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4743128Z self_outputs = self.self( 2025-08-14T21:38:07.4743403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4743514Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4743858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4744006Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4744046Z 2025-08-14T21:38:07.4744142Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4744486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4744552Z layer_outputs = layer_module( 2025-08-14T21:38:07.4744757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4744838Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4745101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4745178Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4745440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4745507Z self_outputs = self.self( 2025-08-14T21:38:07.4745775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 613, in forward 2025-08-14T21:38:07.4745882Z attn_output = self._sliding_chunks_matmul_attn_probs_value( 2025-08-14T21:38:07.4746212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 878, in _sliding_chunks_matmul_attn_probs_value 2025-08-14T21:38:07.4746361Z context = torch.einsum("bcwd,bcdh->bcwh", (chunked_attn_probs, chunked_value)) 2025-08-14T21:38:07.4746364Z 2025-08-14T21:38:07.4746460Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4746795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4746862Z layer_outputs = layer_module( 2025-08-14T21:38:07.4747070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4747151Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4747418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4747497Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4747759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1135, in forward 2025-08-14T21:38:07.4747823Z self_outputs = self.self( 2025-08-14T21:38:07.4748094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 618, in forward 2025-08-14T21:38:07.4748267Z attn_output = attn_output.transpose(0, 1).reshape(seq_len, batch_size, embed_dim).contiguous() 2025-08-14T21:38:07.4748271Z 2025-08-14T21:38:07.4748374Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4748706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4748803Z layer_outputs = layer_module( 2025-08-14T21:38:07.4749017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4749090Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4749351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1199, in forward 2025-08-14T21:38:07.4749430Z self_attn_outputs = self.attention( 2025-08-14T21:38:07.4749693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1144, in forward 2025-08-14T21:38:07.4749802Z attn_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:38:07.4750096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1094, in forward 2025-08-14T21:38:07.4750179Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4750182Z 2025-08-14T21:38:07.4750283Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4750613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4750687Z layer_outputs = layer_module( 2025-08-14T21:38:07.4750895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4750969Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4751244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4751324Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4751583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4751660Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4751933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4752044Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4752315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1160, in forward 2025-08-14T21:38:07.4752392Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4752402Z 2025-08-14T21:38:07.4752499Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4752834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4752911Z layer_outputs = layer_module( 2025-08-14T21:38:07.4753131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4753206Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4753476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4753552Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4753797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4753869Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4754132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1218, in ff_chunk 2025-08-14T21:38:07.4754238Z intermediate_output = self.intermediate(attn_output) 2025-08-14T21:38:07.4754505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1161, in forward 2025-08-14T21:38:07.4754648Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:38:07.4754848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:38:07.4754915Z return self.act(input) 2025-08-14T21:38:07.4754918Z 2025-08-14T21:38:07.4755019Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:38:07.4755347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1259, in torch_dynamo_resume_in_forward_at_1244 2025-08-14T21:38:07.4755413Z layer_outputs = layer_module( 2025-08-14T21:38:07.4755625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:38:07.4755698Z return super().__call__(*args, **kwargs) 2025-08-14T21:38:07.4755997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1211, in forward 2025-08-14T21:38:07.4756079Z layer_output = apply_chunking_to_forward( 2025-08-14T21:38:07.4756331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:38:07.4756413Z return forward_fn(*input_tensors) 2025-08-14T21:38:07.4756729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1219, in ff_chunk 2025-08-14T21:38:07.4756848Z layer_output = self.output(intermediate_output, attn_output) 2025-08-14T21:38:07.4757111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1174, in forward 2025-08-14T21:38:07.4757187Z hidden_states = self.dense(hidden_states) 2025-08-14T21:38:07.4757190Z 2025-08-14T21:39:13.3603871Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:13.3606694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1716, in torch_dynamo_resume_in_forward_at_1703 2025-08-14T21:39:13.3608720Z prediction_scores = self.lm_head(sequence_output) 2025-08-14T21:39:13.3609312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1333, in forward 2025-08-14T21:39:13.3609821Z x = self.dense(features) 2025-08-14T21:39:13.3615590Z 2025-08-14T21:39:13.3619877Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:13.3623182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1716, in torch_dynamo_resume_in_forward_at_1703 2025-08-14T21:39:13.3627240Z prediction_scores = self.lm_head(sequence_output) 2025-08-14T21:39:13.3627928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1338, in forward 2025-08-14T21:39:13.3628494Z x = self.decoder(x) 2025-08-14T21:39:13.3628980Z 2025-08-14T21:39:13.3629410Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:13.3629988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/longformer/modeling_longformer.py", line 1723, in torch_dynamo_resume_in_forward_at_1703 2025-08-14T21:39:13.3630581Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:39:13.3630829Z 2025-08-14T21:39:14.9539202Z Compilation time (from dynamo_timed): 96.553074058 2025-08-14T21:39:14.9765217Z pass 2025-08-14T21:39:14.9765957Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:39:14.9767010Z TIMING: gc:0.00845 entire_frame_compile:96.55307 _recursive_pre_grad_passes:0.02027 _recursive_joint_graph_passes:0.94831 _recursive_post_grad_passes:1.76448 async_compile.wait:3.02419 code_gen:75.32178 inductor_compile:82.32085 backend_compile:91.52692 total_wall_time:96.55307 2025-08-14T21:39:14.9768360Z STATS: call_* op count: 1787 | FakeTensorMode.__torch_dispatch__:56224 | FakeTensor.__torch_dispatch__:16842 | ProxyTorchDispatchMode.__torch_dispatch__:17446 2025-08-14T21:39:14.9768908Z Dynamo produced 4 graphs covering 1787 ops with 4 graph breaks (1 unique) 2025-08-14T21:39:20.7597499Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:39:20.7598638Z from pkg_resources import resource_filename 2025-08-14T21:39:21.3611394Z 2025-08-14T21:39:24.1465392Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:39:24.1466025Z loading model: 0it [00:02, ?it/s] 2025-08-14T21:39:24.1484460Z cpu eval BartForCausalLM 2025-08-14T21:39:25.8358582Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:39:26.5124002Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:39:27.2039353Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:39:34.5787942Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5788418Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5788716Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5788999Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5789234Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5789537Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5789773Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5790089Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5790450Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5790719Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5792723Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5793011Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5793351Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5793898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5794824Z return mod(**inputs) 2025-08-14T21:39:34.5795366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5795777Z outputs = self.model.decoder( 2025-08-14T21:39:34.5796173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5796558Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5796972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5797347Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5797757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5798149Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5798557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:39:34.5799015Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:39:34.5799221Z 2025-08-14T21:39:34.5799339Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5799695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5800019Z return mod(**inputs) 2025-08-14T21:39:34.5800391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5800781Z outputs = self.model.decoder( 2025-08-14T21:39:34.5801544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5801935Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5802324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5802725Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5803168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5803611Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5804046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:39:34.5804686Z key_states = self.k_proj(current_states) 2025-08-14T21:39:34.5804949Z 2025-08-14T21:39:34.5805069Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5805491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5805907Z return mod(**inputs) 2025-08-14T21:39:34.5806301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5806725Z outputs = self.model.decoder( 2025-08-14T21:39:34.5807128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5807542Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5807921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5808303Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5808718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5809152Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5809586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:39:34.5810022Z value_states = self.v_proj(current_states) 2025-08-14T21:39:34.5810177Z 2025-08-14T21:39:34.5810267Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5810499Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5810718Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5810939Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5811191Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5811597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5812017Z return mod(**inputs) 2025-08-14T21:39:34.5812427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5812859Z outputs = self.model.decoder( 2025-08-14T21:39:34.5813243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5813618Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5813971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5814336Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5814714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5815124Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5815531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.5815938Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.5816382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:39:34.5816907Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:39:34.5817083Z 2025-08-14T21:39:34.5817193Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5817544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5817861Z return mod(**inputs) 2025-08-14T21:39:34.5818209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5818582Z outputs = self.model.decoder( 2025-08-14T21:39:34.5818937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5819307Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5819681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5820033Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5820393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5820775Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5821159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.5821733Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.5822158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:39:34.5822598Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:39:34.5822764Z 2025-08-14T21:39:34.5822872Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5823202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5823515Z return mod(**inputs) 2025-08-14T21:39:34.5823862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5824235Z outputs = self.model.decoder( 2025-08-14T21:39:34.5824587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5824948Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5825271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5825601Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5825969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5826363Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5826747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:39:34.5827116Z attn_output = self.out_proj(attn_output) 2025-08-14T21:39:34.5827252Z 2025-08-14T21:39:34.5827350Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5827692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5827995Z return mod(**inputs) 2025-08-14T21:39:34.5828336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5828709Z outputs = self.model.decoder( 2025-08-14T21:39:34.5829072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5829422Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5829746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5830120Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5830480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.5830898Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.5831072Z 2025-08-14T21:39:34.5831172Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5831519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5831828Z return mod(**inputs) 2025-08-14T21:39:34.5832176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5832556Z outputs = self.model.decoder( 2025-08-14T21:39:34.5832942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5833318Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5833661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5834014Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5834383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.5834799Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.5835178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:34.5835517Z return self.act(input) 2025-08-14T21:39:34.5835626Z 2025-08-14T21:39:34.5835728Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5836085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5836405Z return mod(**inputs) 2025-08-14T21:39:34.5836751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5837130Z outputs = self.model.decoder( 2025-08-14T21:39:34.5837497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5838084Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5838416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5838770Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5839144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:39:34.5839523Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:39:34.5839656Z 2025-08-14T21:39:34.5839760Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5840111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5840431Z return mod(**inputs) 2025-08-14T21:39:34.5840772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5841146Z outputs = self.model.decoder( 2025-08-14T21:39:34.5841507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5841882Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5842217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5842576Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5842960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5843360Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5843855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:39:34.5844313Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:39:34.5844633Z 2025-08-14T21:39:34.5844751Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5845119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5845457Z return mod(**inputs) 2025-08-14T21:39:34.5845836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5846251Z outputs = self.model.decoder( 2025-08-14T21:39:34.5846617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5847065Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5847402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5847744Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5848114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5848504Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5848888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:39:34.5849256Z key_states = self.k_proj(current_states) 2025-08-14T21:39:34.5849393Z 2025-08-14T21:39:34.5849492Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5849836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5850143Z return mod(**inputs) 2025-08-14T21:39:34.5850493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5850855Z outputs = self.model.decoder( 2025-08-14T21:39:34.5851206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5851566Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5851899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5852245Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5852610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5852993Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5853378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:39:34.5853753Z value_states = self.v_proj(current_states) 2025-08-14T21:39:34.5853891Z 2025-08-14T21:39:34.5853970Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5854180Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5854383Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5854584Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5854807Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5855163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5855483Z return mod(**inputs) 2025-08-14T21:39:34.5855829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5856208Z outputs = self.model.decoder( 2025-08-14T21:39:34.5856570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5856941Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5857266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5857652Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5858016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5858400Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5858795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.5859184Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.5859612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:39:34.5860103Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:39:34.5860292Z 2025-08-14T21:39:34.5860393Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5860748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5861067Z return mod(**inputs) 2025-08-14T21:39:34.5861406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5861781Z outputs = self.model.decoder( 2025-08-14T21:39:34.5862149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5862520Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5862859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5863218Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5863605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5863986Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5864375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.5864768Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.5865190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:39:34.5865637Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:39:34.5865798Z 2025-08-14T21:39:34.5865901Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5866279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5866622Z return mod(**inputs) 2025-08-14T21:39:34.5866985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5867369Z outputs = self.model.decoder( 2025-08-14T21:39:34.5867755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5868121Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5868455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5868808Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5869172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5869564Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5869953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:39:34.5870352Z attn_output = self.out_proj(attn_output) 2025-08-14T21:39:34.5870484Z 2025-08-14T21:39:34.5870584Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5870979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5871309Z return mod(**inputs) 2025-08-14T21:39:34.5871659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5872043Z outputs = self.model.decoder( 2025-08-14T21:39:34.5872418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5872800Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5873136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5873503Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5873908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.5874329Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.5874496Z 2025-08-14T21:39:34.5874595Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5874985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5875482Z return mod(**inputs) 2025-08-14T21:39:34.5875843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5876225Z outputs = self.model.decoder( 2025-08-14T21:39:34.5876597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5876973Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5877312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5877666Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5878048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.5878461Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.5878842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:34.5879184Z return self.act(input) 2025-08-14T21:39:34.5879292Z 2025-08-14T21:39:34.5879398Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5879743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5880066Z return mod(**inputs) 2025-08-14T21:39:34.5880426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5880825Z outputs = self.model.decoder( 2025-08-14T21:39:34.5881198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5881594Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5881945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5882314Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5882722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:39:34.5883150Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:39:34.5883296Z 2025-08-14T21:39:34.5883412Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5883783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5884124Z return mod(**inputs) 2025-08-14T21:39:34.5884589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5885114Z outputs = self.model.decoder( 2025-08-14T21:39:34.5885522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5885949Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5886315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5886687Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5887094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5887531Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5887944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:39:34.5888479Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:39:34.5888705Z 2025-08-14T21:39:34.5888814Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5889189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5889523Z return mod(**inputs) 2025-08-14T21:39:34.5889899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5890313Z outputs = self.model.decoder( 2025-08-14T21:39:34.5890702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5891101Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5891462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5891843Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5892242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5892672Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5893103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:39:34.5893479Z key_states = self.k_proj(current_states) 2025-08-14T21:39:34.5893608Z 2025-08-14T21:39:34.5893706Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5894055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5894360Z return mod(**inputs) 2025-08-14T21:39:34.5894706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5895070Z outputs = self.model.decoder( 2025-08-14T21:39:34.5895433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5895813Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5896131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5896468Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5896827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5897203Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5897574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:39:34.5897945Z value_states = self.v_proj(current_states) 2025-08-14T21:39:34.5898075Z 2025-08-14T21:39:34.5898156Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5898353Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5898549Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5898745Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5899000Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5899332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5899639Z return mod(**inputs) 2025-08-14T21:39:34.5899980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5900340Z outputs = self.model.decoder( 2025-08-14T21:39:34.5900700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5901064Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5901392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5901755Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5902120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5902511Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5902889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.5903282Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.5903711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:39:34.5904173Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:39:34.5904350Z 2025-08-14T21:39:34.5904451Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5904795Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5905108Z return mod(**inputs) 2025-08-14T21:39:34.5905452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5905827Z outputs = self.model.decoder( 2025-08-14T21:39:34.5906177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5906536Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5906853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5907190Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5907544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5907931Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5908306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.5908686Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.5909104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:39:34.5909532Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:39:34.5909683Z 2025-08-14T21:39:34.5909777Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5910111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5910415Z return mod(**inputs) 2025-08-14T21:39:34.5910743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5911104Z outputs = self.model.decoder( 2025-08-14T21:39:34.5911456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5911810Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5912205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5912548Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5912917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5913308Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5913702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:39:34.5914094Z attn_output = self.out_proj(attn_output) 2025-08-14T21:39:34.5914225Z 2025-08-14T21:39:34.5914334Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5914700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5915011Z return mod(**inputs) 2025-08-14T21:39:34.5915359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5915737Z outputs = self.model.decoder( 2025-08-14T21:39:34.5916091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5916464Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5916799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5917144Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5917517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.5917935Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.5918103Z 2025-08-14T21:39:34.5918212Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5918554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5918870Z return mod(**inputs) 2025-08-14T21:39:34.5919215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5919586Z outputs = self.model.decoder( 2025-08-14T21:39:34.5919952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5920382Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5920720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5921072Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5921444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.5921856Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.5922230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:34.5922556Z return self.act(input) 2025-08-14T21:39:34.5922668Z 2025-08-14T21:39:34.5922768Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5923114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5923424Z return mod(**inputs) 2025-08-14T21:39:34.5923772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5924151Z outputs = self.model.decoder( 2025-08-14T21:39:34.5924590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5924964Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5925306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5925706Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5926112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:39:34.5926541Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:39:34.5926698Z 2025-08-14T21:39:34.5926812Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5927208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5927538Z return mod(**inputs) 2025-08-14T21:39:34.5927914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5928294Z outputs = self.model.decoder( 2025-08-14T21:39:34.5928687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5929061Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5929395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5929739Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5930099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5930491Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5930882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:39:34.5931320Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:39:34.5931522Z 2025-08-14T21:39:34.5931617Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5931958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5932266Z return mod(**inputs) 2025-08-14T21:39:34.5932600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5932968Z outputs = self.model.decoder( 2025-08-14T21:39:34.5933324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5933687Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5934006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5934347Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5934710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5935086Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5935471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:39:34.5935850Z key_states = self.k_proj(current_states) 2025-08-14T21:39:34.5935978Z 2025-08-14T21:39:34.5936080Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5936408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5936714Z return mod(**inputs) 2025-08-14T21:39:34.5937058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5937427Z outputs = self.model.decoder( 2025-08-14T21:39:34.5937915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5938300Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5938650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5939005Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5939447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5939841Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5940231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:39:34.5940608Z value_states = self.v_proj(current_states) 2025-08-14T21:39:34.5940751Z 2025-08-14T21:39:34.5940832Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5941040Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5941238Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5941440Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5941670Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5942061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5942384Z return mod(**inputs) 2025-08-14T21:39:34.5942730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5943103Z outputs = self.model.decoder( 2025-08-14T21:39:34.5943454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5943820Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5944152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5944498Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5944862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5945262Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5945658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.5946055Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.5946488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:39:34.5946958Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:39:34.5947136Z 2025-08-14T21:39:34.5947242Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5947584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5947903Z return mod(**inputs) 2025-08-14T21:39:34.5948258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5948642Z outputs = self.model.decoder( 2025-08-14T21:39:34.5949004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5949387Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5949732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5950082Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5950464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5950875Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5951272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.5951664Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.5952116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:39:34.5952571Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:39:34.5952764Z 2025-08-14T21:39:34.5952871Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5953210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5953524Z return mod(**inputs) 2025-08-14T21:39:34.5953871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5954241Z outputs = self.model.decoder( 2025-08-14T21:39:34.5954603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5954973Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5955310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5955682Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5956066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5956476Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5956871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:39:34.5957259Z attn_output = self.out_proj(attn_output) 2025-08-14T21:39:34.5957402Z 2025-08-14T21:39:34.5957503Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5957858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5958184Z return mod(**inputs) 2025-08-14T21:39:34.5958523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5958891Z outputs = self.model.decoder( 2025-08-14T21:39:34.5959245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5959602Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5959932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5960275Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5960629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.5961031Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.5961196Z 2025-08-14T21:39:34.5961294Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5961626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5961931Z return mod(**inputs) 2025-08-14T21:39:34.5962281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5962657Z outputs = self.model.decoder( 2025-08-14T21:39:34.5963020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5963400Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5963743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5964099Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5964532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.5964993Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.5965400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:34.5965757Z return self.act(input) 2025-08-14T21:39:34.5965873Z 2025-08-14T21:39:34.5965976Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5966369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5966678Z return mod(**inputs) 2025-08-14T21:39:34.5967009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5967373Z outputs = self.model.decoder( 2025-08-14T21:39:34.5967726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5968086Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5968400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5968739Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5969131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:39:34.5969498Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:39:34.5969637Z 2025-08-14T21:39:34.5969736Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5970082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5970410Z return mod(**inputs) 2025-08-14T21:39:34.5970741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5971106Z outputs = self.model.decoder( 2025-08-14T21:39:34.5971457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5971818Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5972156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5972507Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5972886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5973279Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5973672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:39:34.5974107Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:39:34.5974299Z 2025-08-14T21:39:34.5974402Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5974733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5975040Z return mod(**inputs) 2025-08-14T21:39:34.5975378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5975733Z outputs = self.model.decoder( 2025-08-14T21:39:34.5976086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5976451Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5976776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5977106Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5977465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5977847Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5978224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:39:34.5978584Z key_states = self.k_proj(current_states) 2025-08-14T21:39:34.5978717Z 2025-08-14T21:39:34.5978817Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5979158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5979496Z return mod(**inputs) 2025-08-14T21:39:34.5979843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5980211Z outputs = self.model.decoder( 2025-08-14T21:39:34.5980572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5980935Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5981274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5981617Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5982006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5982396Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5982779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:39:34.5983150Z value_states = self.v_proj(current_states) 2025-08-14T21:39:34.5983281Z 2025-08-14T21:39:34.5983357Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5983560Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5983759Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5983946Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.5984166Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5984503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5984811Z return mod(**inputs) 2025-08-14T21:39:34.5985145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5985507Z outputs = self.model.decoder( 2025-08-14T21:39:34.5985864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5986221Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5986547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5986888Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5987254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5987639Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5988028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.5988419Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.5988852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:39:34.5989375Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:39:34.5989556Z 2025-08-14T21:39:34.5989653Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5989993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5990292Z return mod(**inputs) 2025-08-14T21:39:34.5990634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5991006Z outputs = self.model.decoder( 2025-08-14T21:39:34.5991361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5991715Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5992044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5992427Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5992778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5993157Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5993534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.5993912Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.5994320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:39:34.5994747Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:39:34.5994904Z 2025-08-14T21:39:34.5994999Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5995362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.5995668Z return mod(**inputs) 2025-08-14T21:39:34.5996005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.5996372Z outputs = self.model.decoder( 2025-08-14T21:39:34.5996719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.5997078Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.5997405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.5997748Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.5998107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.5998493Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.5998875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:39:34.5999250Z attn_output = self.out_proj(attn_output) 2025-08-14T21:39:34.5999378Z 2025-08-14T21:39:34.5999476Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.5999811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6000118Z return mod(**inputs) 2025-08-14T21:39:34.6000452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6000817Z outputs = self.model.decoder( 2025-08-14T21:39:34.6001173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6001537Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6001866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6002216Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6002587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.6002989Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.6003161Z 2025-08-14T21:39:34.6003266Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6003614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6003927Z return mod(**inputs) 2025-08-14T21:39:34.6004276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6004741Z outputs = self.model.decoder( 2025-08-14T21:39:34.6005130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6005518Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6005906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6006259Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6006642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.6007066Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.6007455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:34.6007799Z return self.act(input) 2025-08-14T21:39:34.6007917Z 2025-08-14T21:39:34.6008024Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6008405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6008725Z return mod(**inputs) 2025-08-14T21:39:34.6009071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6009439Z outputs = self.model.decoder( 2025-08-14T21:39:34.6009809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6010173Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6010500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6010832Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6011193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:39:34.6011567Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:39:34.6011697Z 2025-08-14T21:39:34.6011802Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6012140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6012460Z return mod(**inputs) 2025-08-14T21:39:34.6012805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6013173Z outputs = self.model.decoder( 2025-08-14T21:39:34.6013536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6013917Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6014241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6014572Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6014935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6015326Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6015703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:39:34.6016139Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:39:34.6016337Z 2025-08-14T21:39:34.6016433Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6016777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6017084Z return mod(**inputs) 2025-08-14T21:39:34.6017432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6017805Z outputs = self.model.decoder( 2025-08-14T21:39:34.6018163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6018545Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6018870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6019251Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6019613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6020006Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6020395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:39:34.6020771Z key_states = self.k_proj(current_states) 2025-08-14T21:39:34.6020900Z 2025-08-14T21:39:34.6020997Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6021351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6021662Z return mod(**inputs) 2025-08-14T21:39:34.6022021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6022396Z outputs = self.model.decoder( 2025-08-14T21:39:34.6022750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6023110Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6023426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6023764Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6024123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6024508Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6024884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:39:34.6025258Z value_states = self.v_proj(current_states) 2025-08-14T21:39:34.6025392Z 2025-08-14T21:39:34.6025474Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6025666Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6025861Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6026054Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6026265Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6026599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6026904Z return mod(**inputs) 2025-08-14T21:39:34.6027242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6027597Z outputs = self.model.decoder( 2025-08-14T21:39:34.6027952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6028319Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6028636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6028977Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6029341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6029725Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6030098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.6030480Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.6030906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:39:34.6031378Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:39:34.6031555Z 2025-08-14T21:39:34.6031656Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6032050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6032361Z return mod(**inputs) 2025-08-14T21:39:34.6032692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6033061Z outputs = self.model.decoder( 2025-08-14T21:39:34.6033421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6033792Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6034117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6034459Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6034854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6035245Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6035621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.6036002Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.6036418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:39:34.6036848Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:39:34.6037007Z 2025-08-14T21:39:34.6037103Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6037447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6037950Z return mod(**inputs) 2025-08-14T21:39:34.6038296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6038665Z outputs = self.model.decoder( 2025-08-14T21:39:34.6039025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6039386Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6039714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6040065Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6040430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6040804Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6041189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:39:34.6041564Z attn_output = self.out_proj(attn_output) 2025-08-14T21:39:34.6041694Z 2025-08-14T21:39:34.6041794Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6042144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6042466Z return mod(**inputs) 2025-08-14T21:39:34.6042822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6043235Z outputs = self.model.decoder( 2025-08-14T21:39:34.6043600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6043973Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6044307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6044704Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6045101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.6045560Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.6045814Z 2025-08-14T21:39:34.6045919Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6046258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6046574Z return mod(**inputs) 2025-08-14T21:39:34.6046930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6047281Z outputs = self.model.decoder( 2025-08-14T21:39:34.6047629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6047987Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6048348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6048676Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6049033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.6049424Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.6049774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:34.6050094Z return self.act(input) 2025-08-14T21:39:34.6050204Z 2025-08-14T21:39:34.6050299Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6050633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6050926Z return mod(**inputs) 2025-08-14T21:39:34.6051257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6051614Z outputs = self.model.decoder( 2025-08-14T21:39:34.6051955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6052316Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6052635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6052966Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6053318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:39:34.6053678Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:39:34.6053803Z 2025-08-14T21:39:34.6053908Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6054243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6054541Z return mod(**inputs) 2025-08-14T21:39:34.6054883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6055254Z outputs = self.model.decoder( 2025-08-14T21:39:34.6055592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6055957Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6056283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6056623Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6056977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6057368Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6057736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:39:34.6058152Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:39:34.6058346Z 2025-08-14T21:39:34.6058468Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6058796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6059092Z return mod(**inputs) 2025-08-14T21:39:34.6059415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6059768Z outputs = self.model.decoder( 2025-08-14T21:39:34.6060113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6060462Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6060769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6061125Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6061478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6061844Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6062215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:39:34.6062583Z key_states = self.k_proj(current_states) 2025-08-14T21:39:34.6062708Z 2025-08-14T21:39:34.6062811Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6063138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6063446Z return mod(**inputs) 2025-08-14T21:39:34.6063784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6064146Z outputs = self.model.decoder( 2025-08-14T21:39:34.6064498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6064860Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6065185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6065513Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6065870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6066251Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6066627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:39:34.6066991Z value_states = self.v_proj(current_states) 2025-08-14T21:39:34.6067133Z 2025-08-14T21:39:34.6067211Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6067414Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6067607Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6067805Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6068034Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6068362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6068664Z return mod(**inputs) 2025-08-14T21:39:34.6069003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6069366Z outputs = self.model.decoder( 2025-08-14T21:39:34.6069712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6070074Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6070404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6070747Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6071104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6071520Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6071895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.6072272Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.6072687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:39:34.6073140Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:39:34.6073310Z 2025-08-14T21:39:34.6073414Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6073741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6074076Z return mod(**inputs) 2025-08-14T21:39:34.6074411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6074778Z outputs = self.model.decoder( 2025-08-14T21:39:34.6075122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6075480Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6075801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6076131Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6076494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6076873Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6077251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.6077624Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.6078038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:39:34.6078467Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:39:34.6078617Z 2025-08-14T21:39:34.6078721Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6079049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6079350Z return mod(**inputs) 2025-08-14T21:39:34.6079688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6080041Z outputs = self.model.decoder( 2025-08-14T21:39:34.6080430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6080782Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6081107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6081440Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6081808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6082191Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6082559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:39:34.6082927Z attn_output = self.out_proj(attn_output) 2025-08-14T21:39:34.6083061Z 2025-08-14T21:39:34.6083156Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6083498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6083818Z return mod(**inputs) 2025-08-14T21:39:34.6084177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6084687Z outputs = self.model.decoder( 2025-08-14T21:39:34.6085097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6085510Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6085857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6086220Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6086596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.6087024Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.6087187Z 2025-08-14T21:39:34.6087344Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6087682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6087983Z return mod(**inputs) 2025-08-14T21:39:34.6088325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6088693Z outputs = self.model.decoder( 2025-08-14T21:39:34.6089095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6089457Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6089774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6090106Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6090449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.6090844Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.6091204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:34.6091519Z return self.act(input) 2025-08-14T21:39:34.6091628Z 2025-08-14T21:39:34.6091723Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6092062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6092366Z return mod(**inputs) 2025-08-14T21:39:34.6092692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6093051Z outputs = self.model.decoder( 2025-08-14T21:39:34.6093402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6093760Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6094080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6094419Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6094776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:39:34.6095134Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:39:34.6095269Z 2025-08-14T21:39:34.6095365Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6095702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6096005Z return mod(**inputs) 2025-08-14T21:39:34.6096343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6096713Z outputs = self.model.decoder( 2025-08-14T21:39:34.6097081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6097449Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6097835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6098192Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6098580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6098961Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6099339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:39:34.6099773Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:39:34.6099962Z 2025-08-14T21:39:34.6100066Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6100421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6100740Z return mod(**inputs) 2025-08-14T21:39:34.6101095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6101469Z outputs = self.model.decoder( 2025-08-14T21:39:34.6101846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6102234Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6102579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6102932Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6103320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6103729Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6104139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:39:34.6104523Z key_states = self.k_proj(current_states) 2025-08-14T21:39:34.6104659Z 2025-08-14T21:39:34.6104759Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6105109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6105421Z return mod(**inputs) 2025-08-14T21:39:34.6105773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6106148Z outputs = self.model.decoder( 2025-08-14T21:39:34.6106511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6106880Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6107218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6107569Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6107940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6108333Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6108727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:39:34.6109114Z value_states = self.v_proj(current_states) 2025-08-14T21:39:34.6109251Z 2025-08-14T21:39:34.6109330Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6109541Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6109747Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6109946Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6110178Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6110538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6110869Z return mod(**inputs) 2025-08-14T21:39:34.6111280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6111656Z outputs = self.model.decoder( 2025-08-14T21:39:34.6112019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6112386Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6112726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6113085Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6113453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6113832Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6114274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.6114659Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.6115065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:39:34.6115516Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:39:34.6115695Z 2025-08-14T21:39:34.6115793Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6116129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6116429Z return mod(**inputs) 2025-08-14T21:39:34.6116765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6117126Z outputs = self.model.decoder( 2025-08-14T21:39:34.6117479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6117835Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6118160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6118504Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6118857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6119247Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6119631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.6120018Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.6120437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:39:34.6120878Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:39:34.6121034Z 2025-08-14T21:39:34.6121142Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6121483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6121787Z return mod(**inputs) 2025-08-14T21:39:34.6122139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6122517Z outputs = self.model.decoder( 2025-08-14T21:39:34.6122883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6123248Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6123579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6123931Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6124301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6124823Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6125251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:39:34.6125659Z attn_output = self.out_proj(attn_output) 2025-08-14T21:39:34.6125806Z 2025-08-14T21:39:34.6125908Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6126267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6126598Z return mod(**inputs) 2025-08-14T21:39:34.6126940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6127389Z outputs = self.model.decoder( 2025-08-14T21:39:34.6127798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6128173Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6128541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6128900Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6129276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.6129685Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.6129859Z 2025-08-14T21:39:34.6129960Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6130306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6130623Z return mod(**inputs) 2025-08-14T21:39:34.6130967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6131342Z outputs = self.model.decoder( 2025-08-14T21:39:34.6131709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6132073Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6132412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6132761Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6133134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.6133541Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.6133919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:34.6134260Z return self.act(input) 2025-08-14T21:39:34.6134366Z 2025-08-14T21:39:34.6134470Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6134810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6135127Z return mod(**inputs) 2025-08-14T21:39:34.6135473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6135836Z outputs = self.model.decoder( 2025-08-14T21:39:34.6136202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6136569Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6136901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6137242Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6137744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:39:34.6138204Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:39:34.6138338Z 2025-08-14T21:39:34.6138436Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6138784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6139099Z return mod(**inputs) 2025-08-14T21:39:34.6139449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6139803Z outputs = self.model.decoder( 2025-08-14T21:39:34.6140160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6140522Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6140890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6141229Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6141586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6141981Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6142372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:39:34.6142825Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:39:34.6143030Z 2025-08-14T21:39:34.6143132Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6143487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6143801Z return mod(**inputs) 2025-08-14T21:39:34.6144215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6144578Z outputs = self.model.decoder( 2025-08-14T21:39:34.6144922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6145282Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6145609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6145957Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6146319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6146710Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6147100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:39:34.6147465Z key_states = self.k_proj(current_states) 2025-08-14T21:39:34.6147590Z 2025-08-14T21:39:34.6147690Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6148027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6148343Z return mod(**inputs) 2025-08-14T21:39:34.6148681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6149047Z outputs = self.model.decoder( 2025-08-14T21:39:34.6149407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6149776Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6150097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6150447Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6150808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6151190Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6151620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:39:34.6151991Z value_states = self.v_proj(current_states) 2025-08-14T21:39:34.6152122Z 2025-08-14T21:39:34.6152207Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6152483Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6152899Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6153168Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6153425Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6153862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6153981Z return mod(**inputs) 2025-08-14T21:39:34.6154276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6154369Z outputs = self.model.decoder( 2025-08-14T21:39:34.6154643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6154749Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6155017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6155115Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6155368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6155527Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6155769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.6155905Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.6156241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:39:34.6156392Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:39:34.6156396Z 2025-08-14T21:39:34.6156539Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6156751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6156823Z return mod(**inputs) 2025-08-14T21:39:34.6157138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6157228Z outputs = self.model.decoder( 2025-08-14T21:39:34.6157517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6157604Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6157830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6157978Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6158241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6158379Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6158631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.6172852Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.6173357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:39:34.6173489Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:39:34.6173498Z 2025-08-14T21:39:34.6173631Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6173848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6174025Z return mod(**inputs) 2025-08-14T21:39:34.6174291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6174371Z outputs = self.model.decoder( 2025-08-14T21:39:34.6174620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6174698Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6174921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6175015Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6175267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6176322Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6176599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:39:34.6176691Z attn_output = self.out_proj(attn_output) 2025-08-14T21:39:34.6176697Z 2025-08-14T21:39:34.6176815Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6177018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6177088Z return mod(**inputs) 2025-08-14T21:39:34.6177343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6177425Z outputs = self.model.decoder( 2025-08-14T21:39:34.6177681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6177757Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6177983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6178075Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6178320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.6178444Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.6178457Z 2025-08-14T21:39:34.6178562Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6178762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6178838Z return mod(**inputs) 2025-08-14T21:39:34.6179083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6179160Z outputs = self.model.decoder( 2025-08-14T21:39:34.6179416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6179488Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6179719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6179799Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6180041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.6180168Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.6180378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:34.6180447Z return self.act(input) 2025-08-14T21:39:34.6180452Z 2025-08-14T21:39:34.6180561Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6180757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6180833Z return mod(**inputs) 2025-08-14T21:39:34.6181080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6181192Z outputs = self.model.decoder( 2025-08-14T21:39:34.6181445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6181518Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6181733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6181817Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6182056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:39:34.6182145Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:39:34.6182148Z 2025-08-14T21:39:34.6182281Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6182479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6182555Z return mod(**inputs) 2025-08-14T21:39:34.6182803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6182883Z outputs = self.model.decoder( 2025-08-14T21:39:34.6183140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6183208Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6183426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6183516Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6183760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6183861Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6184111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:39:34.6184263Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:39:34.6184267Z 2025-08-14T21:39:34.6184384Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6184575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6184636Z return mod(**inputs) 2025-08-14T21:39:34.6184883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6184952Z outputs = self.model.decoder( 2025-08-14T21:39:34.6185198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6185268Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6185478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6185563Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6185806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6185902Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6186150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:39:34.6186234Z key_states = self.k_proj(current_states) 2025-08-14T21:39:34.6186238Z 2025-08-14T21:39:34.6186349Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6186558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6186626Z return mod(**inputs) 2025-08-14T21:39:34.6186904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6187014Z outputs = self.model.decoder( 2025-08-14T21:39:34.6187277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6187350Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6187576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6187659Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6187910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6188009Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6188265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:39:34.6188387Z value_states = self.v_proj(current_states) 2025-08-14T21:39:34.6188391Z 2025-08-14T21:39:34.6188490Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6188576Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6188656Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6188741Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6188848Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6189053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6189131Z return mod(**inputs) 2025-08-14T21:39:34.6189389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6189473Z outputs = self.model.decoder( 2025-08-14T21:39:34.6189731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6189808Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6190045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6190130Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6190384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6190492Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6190747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.6190858Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.6191151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:39:34.6191284Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:39:34.6191291Z 2025-08-14T21:39:34.6191394Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6191588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6191659Z return mod(**inputs) 2025-08-14T21:39:34.6191893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6191963Z outputs = self.model.decoder( 2025-08-14T21:39:34.6192205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6192273Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6192478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6192559Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6192803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6192900Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6193201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.6193298Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.6193585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:39:34.6193691Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:39:34.6193695Z 2025-08-14T21:39:34.6193798Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6193987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6194051Z return mod(**inputs) 2025-08-14T21:39:34.6194320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6194391Z outputs = self.model.decoder( 2025-08-14T21:39:34.6194631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6194709Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6194923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6195007Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6195259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6195350Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6195591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:39:34.6195670Z attn_output = self.out_proj(attn_output) 2025-08-14T21:39:34.6195676Z 2025-08-14T21:39:34.6195781Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6195976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6196040Z return mod(**inputs) 2025-08-14T21:39:34.6196291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6196362Z outputs = self.model.decoder( 2025-08-14T21:39:34.6196612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6196688Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6196895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6197053Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6197286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.6197401Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.6197407Z 2025-08-14T21:39:34.6197510Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6197698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6197768Z return mod(**inputs) 2025-08-14T21:39:34.6198001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6198071Z outputs = self.model.decoder( 2025-08-14T21:39:34.6198317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6198388Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6198609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6198695Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6198932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.6199092Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.6199302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:34.6199372Z return self.act(input) 2025-08-14T21:39:34.6199376Z 2025-08-14T21:39:34.6199483Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6199683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6199760Z return mod(**inputs) 2025-08-14T21:39:34.6200025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6200134Z outputs = self.model.decoder( 2025-08-14T21:39:34.6200411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6200491Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6200725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6200817Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6201080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:39:34.6201173Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:39:34.6201177Z 2025-08-14T21:39:34.6201285Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6201498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6201577Z return mod(**inputs) 2025-08-14T21:39:34.6201847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6201930Z outputs = self.model.decoder( 2025-08-14T21:39:34.6202205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6202280Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6202523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6202606Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6202870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6202984Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6203247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:39:34.6203419Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:39:34.6203423Z 2025-08-14T21:39:34.6203535Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6203748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6203827Z return mod(**inputs) 2025-08-14T21:39:34.6204096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6204175Z outputs = self.model.decoder( 2025-08-14T21:39:34.6204533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6204619Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6204864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6204948Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6205216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6205369Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6205646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:39:34.6205751Z key_states = self.k_proj(current_states) 2025-08-14T21:39:34.6205755Z 2025-08-14T21:39:34.6205856Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6206050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6206121Z return mod(**inputs) 2025-08-14T21:39:34.6206367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6206438Z outputs = self.model.decoder( 2025-08-14T21:39:34.6206722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6206797Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6207021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6207100Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6207346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6207447Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6207688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:39:34.6207772Z value_states = self.v_proj(current_states) 2025-08-14T21:39:34.6207783Z 2025-08-14T21:39:34.6207859Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6207935Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6208020Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6208095Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6208197Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6208399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6208464Z return mod(**inputs) 2025-08-14T21:39:34.6208707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6208786Z outputs = self.model.decoder( 2025-08-14T21:39:34.6209027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6209102Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6209319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6209398Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6209646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6209742Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6209986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.6210081Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.6210363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:39:34.6210502Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:39:34.6210505Z 2025-08-14T21:39:34.6210604Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6210797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6210871Z return mod(**inputs) 2025-08-14T21:39:34.6211114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6211227Z outputs = self.model.decoder( 2025-08-14T21:39:34.6211471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6211542Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6211764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6211841Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6212091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6212187Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6212458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.6212563Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.6212849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:39:34.6212958Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:39:34.6212962Z 2025-08-14T21:39:34.6213069Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6213264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6213335Z return mod(**inputs) 2025-08-14T21:39:34.6213584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6213655Z outputs = self.model.decoder( 2025-08-14T21:39:34.6213914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6213986Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6214203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6214292Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6214537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6214640Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6214885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:39:34.6214964Z attn_output = self.out_proj(attn_output) 2025-08-14T21:39:34.6214968Z 2025-08-14T21:39:34.6215077Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6215272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6215346Z return mod(**inputs) 2025-08-14T21:39:34.6215599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6215671Z outputs = self.model.decoder( 2025-08-14T21:39:34.6215918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6215987Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6216199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6216281Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6216517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.6216636Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.6216639Z 2025-08-14T21:39:34.6216739Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6216931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6217042Z return mod(**inputs) 2025-08-14T21:39:34.6217292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6217370Z outputs = self.model.decoder( 2025-08-14T21:39:34.6217615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6217684Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6217905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6217980Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6218224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.6218373Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.6218577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:34.6218656Z return self.act(input) 2025-08-14T21:39:34.6218660Z 2025-08-14T21:39:34.6218758Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6218946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6219016Z return mod(**inputs) 2025-08-14T21:39:34.6219254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6219326Z outputs = self.model.decoder( 2025-08-14T21:39:34.6219573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6219642Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6219864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6219943Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6220178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:39:34.6220265Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:39:34.6220268Z 2025-08-14T21:39:34.6220366Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6220564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6220627Z return mod(**inputs) 2025-08-14T21:39:34.6220865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6220941Z outputs = self.model.decoder( 2025-08-14T21:39:34.6221181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6221249Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6221471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6221544Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6221787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6221886Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6222127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:39:34.6222282Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:39:34.6222285Z 2025-08-14T21:39:34.6222385Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6222585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6222649Z return mod(**inputs) 2025-08-14T21:39:34.6222929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6223010Z outputs = self.model.decoder( 2025-08-14T21:39:34.6223267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6223343Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6223579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6223670Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6223917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6224011Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6224282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:39:34.6224373Z key_states = self.k_proj(current_states) 2025-08-14T21:39:34.6224377Z 2025-08-14T21:39:34.6224477Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6224676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6224740Z return mod(**inputs) 2025-08-14T21:39:34.6224984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6225061Z outputs = self.model.decoder( 2025-08-14T21:39:34.6225304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6225374Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6225603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6225681Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6225934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6226029Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6226269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:39:34.6226362Z value_states = self.v_proj(current_states) 2025-08-14T21:39:34.6226366Z 2025-08-14T21:39:34.6226443Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6226523Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6226607Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6226682Z cudagraph partition due to non gpu ops 2025-08-14T21:39:34.6226791Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6226989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6227055Z return mod(**inputs) 2025-08-14T21:39:34.6227311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6227384Z outputs = self.model.decoder( 2025-08-14T21:39:34.6227630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6227708Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6227926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6228009Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6228261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6228352Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6228597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.6228723Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.6229010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:39:34.6229139Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:39:34.6229143Z 2025-08-14T21:39:34.6229240Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6229437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6229499Z return mod(**inputs) 2025-08-14T21:39:34.6229737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6229815Z outputs = self.model.decoder( 2025-08-14T21:39:34.6230082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6230161Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6230372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6230449Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6230694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6230789Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6231035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:39:34.6231128Z attn_output, attn_weights = attention_interface( 2025-08-14T21:39:34.6231417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:39:34.6231532Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:39:34.6231539Z 2025-08-14T21:39:34.6231637Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6231840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6231910Z return mod(**inputs) 2025-08-14T21:39:34.6232145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6232221Z outputs = self.model.decoder( 2025-08-14T21:39:34.6232455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6232531Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6232747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6232826Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6233071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:39:34.6233166Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:39:34.6233405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:39:34.6233492Z attn_output = self.out_proj(attn_output) 2025-08-14T21:39:34.6233495Z 2025-08-14T21:39:34.6233593Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6233793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6233859Z return mod(**inputs) 2025-08-14T21:39:34.6234120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6234199Z outputs = self.model.decoder( 2025-08-14T21:39:34.6234460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6234564Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6234795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6234877Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6235142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.6235266Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.6235270Z 2025-08-14T21:39:34.6235377Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6235596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6235662Z return mod(**inputs) 2025-08-14T21:39:34.6235967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6236050Z outputs = self.model.decoder( 2025-08-14T21:39:34.6236306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6236387Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6236606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6236684Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6236939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:39:34.6237057Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:39:34.6237277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:39:34.6237348Z return self.act(input) 2025-08-14T21:39:34.6237354Z 2025-08-14T21:39:34.6237456Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6237829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6237906Z return mod(**inputs) 2025-08-14T21:39:34.6238166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1901, in forward 2025-08-14T21:39:34.6238252Z outputs = self.model.decoder( 2025-08-14T21:39:34.6238508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:39:34.6238592Z layer_outputs = decoder_layer( 2025-08-14T21:39:34.6238818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:39:34.6238902Z return super().__call__(*args, **kwargs) 2025-08-14T21:39:34.6239172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:39:34.6239259Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:39:34.6239266Z 2025-08-14T21:39:34.6239381Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6239588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6239657Z return mod(**inputs) 2025-08-14T21:39:34.6239926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1917, in forward 2025-08-14T21:39:34.6240010Z logits = self.lm_head(outputs[0]) 2025-08-14T21:39:34.6240014Z 2025-08-14T21:39:34.6240119Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:39:34.6240333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:39:34.6240401Z return mod(**inputs) 2025-08-14T21:39:34.6240668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1923, in forward 2025-08-14T21:39:34.6240823Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:39:34.6240927Z 2025-08-14T21:39:43.8477593Z Compilation time (from dynamo_timed): 14.384173459 2025-08-14T21:39:43.8824034Z pass 2025-08-14T21:39:43.8827394Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:39:43.8828157Z TIMING: _recursive_pre_grad_passes:0.00706 _recursive_joint_graph_passes:0.6561 _recursive_post_grad_passes:0.08354 async_compile.wait:0.71329 code_gen:7.80504 inductor_compile:9.05566 backend_compile:12.23739 gc:0.00105 entire_frame_compile:14.38417 total_wall_time:14.38417 2025-08-14T21:39:43.8829161Z STATS: call_* op count: 372 | FakeTensorMode.__torch_dispatch__:13198 | FakeTensor.__torch_dispatch__:4868 | ProxyTorchDispatchMode.__torch_dispatch__:4813 2025-08-14T21:39:43.8829942Z Dynamo produced 1 graphs covering 372 ops with 0 graph breaks (0 unique) 2025-08-14T21:39:48.8098557Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:39:48.8099808Z from pkg_resources import resource_filename 2025-08-14T21:39:49.4363738Z 2025-08-14T21:39:54.5850955Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:39:54.5851356Z loading model: 0it [00:05, ?it/s] 2025-08-14T21:39:54.5878958Z cpu eval BartForConditionalGeneration 2025-08-14T21:39:57.9339963Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:39:59.1490088Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:40:00.3663366Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:40:16.6206201Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6211846Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6212138Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6212351Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6212559Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6212767Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6212978Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6213182Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6214595Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6214888Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6215108Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6215358Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6215698Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6216123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6216463Z return mod(**inputs) 2025-08-14T21:40:16.6216903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6217327Z outputs = self.model( 2025-08-14T21:40:16.6217724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6218152Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6218572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6219055Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6219420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6219802Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6220195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6220969Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6221365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6221832Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6222033Z 2025-08-14T21:40:16.6222148Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6222502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6222835Z return mod(**inputs) 2025-08-14T21:40:16.6223210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6223592Z outputs = self.model( 2025-08-14T21:40:16.6224030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6224480Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6224886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6225334Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6225721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6226114Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6226529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6226968Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6227399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6227853Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6227993Z 2025-08-14T21:40:16.6228102Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6228484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6228837Z return mod(**inputs) 2025-08-14T21:40:16.6229228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6229687Z outputs = self.model( 2025-08-14T21:40:16.6230088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6230513Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6230964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6231421Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6231801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6232180Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6232595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6233029Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6233461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6233890Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6234036Z 2025-08-14T21:40:16.6234123Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6234542Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6234763Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6234973Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6235217Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6235602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6235993Z return mod(**inputs) 2025-08-14T21:40:16.6236383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6236785Z outputs = self.model( 2025-08-14T21:40:16.6237164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6237576Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6238141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6238527Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6238877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6239233Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6239679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6240080Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6240462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6240869Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6241310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6241785Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6241976Z 2025-08-14T21:40:16.6242089Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6242465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6242817Z return mod(**inputs) 2025-08-14T21:40:16.6243195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6243596Z outputs = self.model( 2025-08-14T21:40:16.6243974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6244390Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6244774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6245185Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6245649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6246074Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6246485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6246915Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6247304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6247701Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6248142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6248599Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6248760Z 2025-08-14T21:40:16.6248871Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6249224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6249547Z return mod(**inputs) 2025-08-14T21:40:16.6249901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6250281Z outputs = self.model( 2025-08-14T21:40:16.6250629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6251090Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6251464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6251836Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6252186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6252533Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6252929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6253342Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6253797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6254210Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6254354Z 2025-08-14T21:40:16.6254463Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6254838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6255190Z return mod(**inputs) 2025-08-14T21:40:16.6255541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6255917Z outputs = self.model( 2025-08-14T21:40:16.6256257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6256624Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6256984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6257347Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6257679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6258026Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6258386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6258802Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6258982Z 2025-08-14T21:40:16.6259084Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6259443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6259755Z return mod(**inputs) 2025-08-14T21:40:16.6260113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6260479Z outputs = self.model( 2025-08-14T21:40:16.6260830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6261210Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6261576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6261951Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6262287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6262647Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6263033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6263481Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6263876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.6264236Z return self.act(input) 2025-08-14T21:40:16.6264352Z 2025-08-14T21:40:16.6264466Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6265657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6265981Z return mod(**inputs) 2025-08-14T21:40:16.6266341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6266730Z outputs = self.model( 2025-08-14T21:40:16.6267098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6267484Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6267860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6268232Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6268645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6269072Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6269475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:40:16.6269848Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.6269989Z 2025-08-14T21:40:16.6270090Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6270443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6270756Z return mod(**inputs) 2025-08-14T21:40:16.6271121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6271498Z outputs = self.model( 2025-08-14T21:40:16.6271990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6272376Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6272751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6273129Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6273477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6273831Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6274213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6274621Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6275009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6275516Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6275726Z 2025-08-14T21:40:16.6275830Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6276196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6276510Z return mod(**inputs) 2025-08-14T21:40:16.6276874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6277253Z outputs = self.model( 2025-08-14T21:40:16.6277603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6277989Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6278359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6278736Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6279080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6279439Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6279875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6280267Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6280649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6281036Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6281169Z 2025-08-14T21:40:16.6281279Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6281626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6281946Z return mod(**inputs) 2025-08-14T21:40:16.6282343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6282729Z outputs = self.model( 2025-08-14T21:40:16.6283088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6283475Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6283842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6284214Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6284580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6284966Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6285372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6285944Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6286386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6286831Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6286989Z 2025-08-14T21:40:16.6287087Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6287324Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6287544Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6287762Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6288002Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6288383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6288723Z return mod(**inputs) 2025-08-14T21:40:16.6289092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6289489Z outputs = self.model( 2025-08-14T21:40:16.6289870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6290278Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6290673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6291074Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6291440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6291814Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6292209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6292628Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6293040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6293457Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6293931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6294487Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6294678Z 2025-08-14T21:40:16.6294794Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6295163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6295505Z return mod(**inputs) 2025-08-14T21:40:16.6295859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6296237Z outputs = self.model( 2025-08-14T21:40:16.6296585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6296966Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6297373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6297747Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6298092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6298455Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6298837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6299223Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6299613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6300011Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6300441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6300897Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6301066Z 2025-08-14T21:40:16.6301167Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6301519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6301835Z return mod(**inputs) 2025-08-14T21:40:16.6302189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6302562Z outputs = self.model( 2025-08-14T21:40:16.6302915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6303287Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6303657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6304053Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6304391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6304754Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6305131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6305524Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6305905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6306285Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6306417Z 2025-08-14T21:40:16.6306524Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6306877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6307209Z return mod(**inputs) 2025-08-14T21:40:16.6307587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6308030Z outputs = self.model( 2025-08-14T21:40:16.6308404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6308787Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6309158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6309539Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6309876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6310234Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6310612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6311068Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6311249Z 2025-08-14T21:40:16.6311352Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6311709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6312036Z return mod(**inputs) 2025-08-14T21:40:16.6312385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6312779Z outputs = self.model( 2025-08-14T21:40:16.6313145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6313530Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6313896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6314282Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6314628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6314984Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6315365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6315787Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6316169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.6316513Z return self.act(input) 2025-08-14T21:40:16.6316629Z 2025-08-14T21:40:16.6316732Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6317087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6317401Z return mod(**inputs) 2025-08-14T21:40:16.6317764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6318144Z outputs = self.model( 2025-08-14T21:40:16.6318512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6318892Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6319267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6319647Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6319984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6320345Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6320727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:40:16.6321115Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.6321251Z 2025-08-14T21:40:16.6321364Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6321721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6322102Z return mod(**inputs) 2025-08-14T21:40:16.6322454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6322820Z outputs = self.model( 2025-08-14T21:40:16.6323173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6323584Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6323990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6324408Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6324794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6325217Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6325717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6326177Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6326619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6327105Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6327309Z 2025-08-14T21:40:16.6327412Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6327771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6328119Z return mod(**inputs) 2025-08-14T21:40:16.6328511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6328946Z outputs = self.model( 2025-08-14T21:40:16.6329345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6329782Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6330190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6330605Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6330981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6331377Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6331802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6332251Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6332720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6333143Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6333297Z 2025-08-14T21:40:16.6333409Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6333804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6334154Z return mod(**inputs) 2025-08-14T21:40:16.6334545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6334960Z outputs = self.model( 2025-08-14T21:40:16.6335351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6335770Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6336141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6336524Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6336866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6337257Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6337860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6338271Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6338718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6339115Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6339263Z 2025-08-14T21:40:16.6339345Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6339558Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6339761Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6340046Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6340279Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6340632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6340956Z return mod(**inputs) 2025-08-14T21:40:16.6341307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6341681Z outputs = self.model( 2025-08-14T21:40:16.6342032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6342416Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6342790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6343165Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6343515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6343874Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6344255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6344639Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6345033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6345435Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6345870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6346353Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6346546Z 2025-08-14T21:40:16.6346647Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6347006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6347321Z return mod(**inputs) 2025-08-14T21:40:16.6347687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6348053Z outputs = self.model( 2025-08-14T21:40:16.6348400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6348768Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6349129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6349499Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6349836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6350186Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6350561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6351003Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6351376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6351769Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6352194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6352636Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6352792Z 2025-08-14T21:40:16.6352892Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6353239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6353558Z return mod(**inputs) 2025-08-14T21:40:16.6353939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6354312Z outputs = self.model( 2025-08-14T21:40:16.6354655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6355025Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6355380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6355747Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6356090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6356435Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6356817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6357211Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6357599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6357971Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6358109Z 2025-08-14T21:40:16.6358207Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6358557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6358872Z return mod(**inputs) 2025-08-14T21:40:16.6359217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6359587Z outputs = self.model( 2025-08-14T21:40:16.6359938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6360311Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6360683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6361058Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6361400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6361746Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6362124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6362542Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6362711Z 2025-08-14T21:40:16.6362810Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6363162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6363482Z return mod(**inputs) 2025-08-14T21:40:16.6363839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6364212Z outputs = self.model( 2025-08-14T21:40:16.6364606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6364989Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6365350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6365810Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6366151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6366507Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6366888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6367316Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6367741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.6368094Z return self.act(input) 2025-08-14T21:40:16.6368208Z 2025-08-14T21:40:16.6368315Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6368681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6369011Z return mod(**inputs) 2025-08-14T21:40:16.6369365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6369750Z outputs = self.model( 2025-08-14T21:40:16.6370119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6370508Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6370883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6371270Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6371627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6371992Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6372429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:40:16.6372812Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.6372946Z 2025-08-14T21:40:16.6373056Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6373401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6373723Z return mod(**inputs) 2025-08-14T21:40:16.6374072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6374444Z outputs = self.model( 2025-08-14T21:40:16.6374796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6375181Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6375554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6375921Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6376317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6376683Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6377056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6377437Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6377822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6378264Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6378511Z 2025-08-14T21:40:16.6378613Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6378963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6379274Z return mod(**inputs) 2025-08-14T21:40:16.6379621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6379980Z outputs = self.model( 2025-08-14T21:40:16.6380326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6380697Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6381059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6381453Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6381786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6382138Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6382499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6382879Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6383261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6383634Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6383767Z 2025-08-14T21:40:16.6383869Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6384224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6384545Z return mod(**inputs) 2025-08-14T21:40:16.6384925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6385323Z outputs = self.model( 2025-08-14T21:40:16.6385688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6386064Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6386423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6386809Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6387144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6387483Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6387851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6388254Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6388649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6389023Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6389167Z 2025-08-14T21:40:16.6389246Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6389456Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6389663Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6389860Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6390089Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6390450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6390755Z return mod(**inputs) 2025-08-14T21:40:16.6391102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6391467Z outputs = self.model( 2025-08-14T21:40:16.6391807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6392218Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6392574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6392934Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6393259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6393605Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6393973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6394353Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6394759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6395158Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6395590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6396048Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6396233Z 2025-08-14T21:40:16.6396334Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6396682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6396996Z return mod(**inputs) 2025-08-14T21:40:16.6397335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6397712Z outputs = self.model( 2025-08-14T21:40:16.6398057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6398413Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6398765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6399122Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6399443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6399772Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6400131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6400497Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6400869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6401266Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6401683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6402117Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6402266Z 2025-08-14T21:40:16.6402362Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6402697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6403000Z return mod(**inputs) 2025-08-14T21:40:16.6403341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6403697Z outputs = self.model( 2025-08-14T21:40:16.6404043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6404417Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6404789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6405157Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6405611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6405992Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6406388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6406825Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6407215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6407601Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6407746Z 2025-08-14T21:40:16.6407845Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6408228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6408543Z return mod(**inputs) 2025-08-14T21:40:16.6408882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6409251Z outputs = self.model( 2025-08-14T21:40:16.6409593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6409961Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6410316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6410685Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6411023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6411365Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6411739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6412154Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6412320Z 2025-08-14T21:40:16.6412425Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6412765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6413077Z return mod(**inputs) 2025-08-14T21:40:16.6413426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6413795Z outputs = self.model( 2025-08-14T21:40:16.6414137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6414506Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6414870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6415235Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6415569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6415918Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6416287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6416700Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6417075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.6417405Z return self.act(input) 2025-08-14T21:40:16.6417509Z 2025-08-14T21:40:16.6417608Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6417952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6418266Z return mod(**inputs) 2025-08-14T21:40:16.6418608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6419001Z outputs = self.model( 2025-08-14T21:40:16.6419349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6419721Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6420090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6420454Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6420788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6421136Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6421497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:40:16.6421918Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.6422057Z 2025-08-14T21:40:16.6422162Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6422505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6422810Z return mod(**inputs) 2025-08-14T21:40:16.6423153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6423515Z outputs = self.model( 2025-08-14T21:40:16.6423851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6424221Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6424582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6424947Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6425278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6425623Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6425988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6426366Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6426752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6427189Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6427383Z 2025-08-14T21:40:16.6427489Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6427826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6428139Z return mod(**inputs) 2025-08-14T21:40:16.6428490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6428857Z outputs = self.model( 2025-08-14T21:40:16.6429197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6429570Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6429934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6430293Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6430636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6430976Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6431334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6431707Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6432081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6432500Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6432629Z 2025-08-14T21:40:16.6432736Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6433074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6433389Z return mod(**inputs) 2025-08-14T21:40:16.6433736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6434092Z outputs = self.model( 2025-08-14T21:40:16.6434443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6434808Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6435205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6435580Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6435906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6436248Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6436605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6436988Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6437363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6437918Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6438062Z 2025-08-14T21:40:16.6438143Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6438352Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6438562Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6438755Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6438985Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6439334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6439658Z return mod(**inputs) 2025-08-14T21:40:16.6440007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6440377Z outputs = self.model( 2025-08-14T21:40:16.6440724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6441093Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6441453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6441822Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6442160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6442506Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6442876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6443263Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6443637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6444029Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6444466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6444953Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6445136Z 2025-08-14T21:40:16.6445244Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6445660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6446090Z return mod(**inputs) 2025-08-14T21:40:16.6446442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6446812Z outputs = self.model( 2025-08-14T21:40:16.6447226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6447598Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6447959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6448341Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6448685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6449095Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6449481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6449887Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6450286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6450683Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6451131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6451592Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6451760Z 2025-08-14T21:40:16.6451872Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6452228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6452558Z return mod(**inputs) 2025-08-14T21:40:16.6452924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6453307Z outputs = self.model( 2025-08-14T21:40:16.6453664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6454052Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6454431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6454806Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6455154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6455518Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6455906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6456297Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6456698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6457092Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6457230Z 2025-08-14T21:40:16.6457340Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6457695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6458025Z return mod(**inputs) 2025-08-14T21:40:16.6458384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6458765Z outputs = self.model( 2025-08-14T21:40:16.6459131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6459522Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6459878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6460289Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6460632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6460991Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6461361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6461782Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6461961Z 2025-08-14T21:40:16.6462065Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6462423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6462778Z return mod(**inputs) 2025-08-14T21:40:16.6463128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6463503Z outputs = self.model( 2025-08-14T21:40:16.6463842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6464227Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6464599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6464977Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6465331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6465688Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6466070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6466474Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6466847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.6467181Z return self.act(input) 2025-08-14T21:40:16.6467286Z 2025-08-14T21:40:16.6467392Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6467732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6468050Z return mod(**inputs) 2025-08-14T21:40:16.6468397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6468764Z outputs = self.model( 2025-08-14T21:40:16.6469109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6469483Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6469852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6470220Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6470564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6470913Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6471283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:40:16.6471652Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.6471792Z 2025-08-14T21:40:16.6471892Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6472239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6472559Z return mod(**inputs) 2025-08-14T21:40:16.6472892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6473249Z outputs = self.model( 2025-08-14T21:40:16.6473627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6473992Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6474355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6474732Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6475075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6475426Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6475804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6476194Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6476670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6477135Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6477341Z 2025-08-14T21:40:16.6477443Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6477800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6478114Z return mod(**inputs) 2025-08-14T21:40:16.6478477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6478861Z outputs = self.model( 2025-08-14T21:40:16.6479210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6479595Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6479973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6480356Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6480697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6481058Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6481443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6481841Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6482228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6482613Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6482746Z 2025-08-14T21:40:16.6482854Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6483205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6483535Z return mod(**inputs) 2025-08-14T21:40:16.6483901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6484275Z outputs = self.model( 2025-08-14T21:40:16.6484626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6485011Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6485386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6485833Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6486193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6486587Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6487013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6487490Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6487903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6488295Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6488434Z 2025-08-14T21:40:16.6488522Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6488728Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6488942Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6489150Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6489374Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6489741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6490065Z return mod(**inputs) 2025-08-14T21:40:16.6490461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6490840Z outputs = self.model( 2025-08-14T21:40:16.6491193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6491552Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6491893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6492248Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6492573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6492908Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6493255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6493634Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6494001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6494384Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6494813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6495264Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6495435Z 2025-08-14T21:40:16.6495539Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6495866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6496173Z return mod(**inputs) 2025-08-14T21:40:16.6496509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6496857Z outputs = self.model( 2025-08-14T21:40:16.6497206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6497580Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6497939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6498295Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6498627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6498975Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6499331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6499699Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6500070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6500448Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6500897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6501331Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6501488Z 2025-08-14T21:40:16.6501585Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6501931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6502237Z return mod(**inputs) 2025-08-14T21:40:16.6502588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6502966Z outputs = self.model( 2025-08-14T21:40:16.6503299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6503705Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6504068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6504430Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6504750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6505089Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6505449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6505824Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6506190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6506559Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6506686Z 2025-08-14T21:40:16.6506791Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6507123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6507436Z return mod(**inputs) 2025-08-14T21:40:16.6507771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6508126Z outputs = self.model( 2025-08-14T21:40:16.6508453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6508814Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6509165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6509516Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6509843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6510185Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6510545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6510944Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6511112Z 2025-08-14T21:40:16.6511207Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6511548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6511855Z return mod(**inputs) 2025-08-14T21:40:16.6512186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6512543Z outputs = self.model( 2025-08-14T21:40:16.6512877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6513238Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6513599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6514009Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6514337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6514673Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6515037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6515439Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6515796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.6516116Z return self.act(input) 2025-08-14T21:40:16.6516224Z 2025-08-14T21:40:16.6516324Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6516700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6517002Z return mod(**inputs) 2025-08-14T21:40:16.6517347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6517706Z outputs = self.model( 2025-08-14T21:40:16.6518034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6518399Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6518750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6519111Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6519429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6519767Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6520131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:40:16.6520499Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.6520627Z 2025-08-14T21:40:16.6520723Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6521060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6521365Z return mod(**inputs) 2025-08-14T21:40:16.6521692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6522048Z outputs = self.model( 2025-08-14T21:40:16.6522383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6522744Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6523094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6523449Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6523774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6524113Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6524472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6524850Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6525230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6525786Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6526009Z 2025-08-14T21:40:16.6526118Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6526499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6526847Z return mod(**inputs) 2025-08-14T21:40:16.6527268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6527632Z outputs = self.model( 2025-08-14T21:40:16.6527976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6528341Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6528701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6529068Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6529407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6529750Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6530165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6530564Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6530953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6531337Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6531479Z 2025-08-14T21:40:16.6531581Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6531937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6532253Z return mod(**inputs) 2025-08-14T21:40:16.6532611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6532989Z outputs = self.model( 2025-08-14T21:40:16.6533347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6533721Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6534094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6534473Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6534814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6535175Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6535561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6535968Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6536350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6536737Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6536876Z 2025-08-14T21:40:16.6536969Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6537177Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6537390Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6537597Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6538074Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6538422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6538742Z return mod(**inputs) 2025-08-14T21:40:16.6539093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6539458Z outputs = self.model( 2025-08-14T21:40:16.6539808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6540183Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6540551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6540915Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6541324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6541661Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6542020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6542397Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6542780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6543183Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6543598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6544107Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6544286Z 2025-08-14T21:40:16.6544392Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6544731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6545031Z return mod(**inputs) 2025-08-14T21:40:16.6545370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6545740Z outputs = self.model( 2025-08-14T21:40:16.6546083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6546463Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6546834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6547206Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6547535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6547890Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6548265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6548641Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6549033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6549415Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6549836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6550263Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6550421Z 2025-08-14T21:40:16.6550521Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6550862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6551170Z return mod(**inputs) 2025-08-14T21:40:16.6551502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6551862Z outputs = self.model( 2025-08-14T21:40:16.6552213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6552580Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6552948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6553317Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6553655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6553999Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6554371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6554805Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6555175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6555550Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6555689Z 2025-08-14T21:40:16.6555788Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6556135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6556448Z return mod(**inputs) 2025-08-14T21:40:16.6556781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6557135Z outputs = self.model( 2025-08-14T21:40:16.6557508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6557889Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6558254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6558621Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6558952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6559304Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6559679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6560106Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6560279Z 2025-08-14T21:40:16.6560380Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6560741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6561077Z return mod(**inputs) 2025-08-14T21:40:16.6561424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6561791Z outputs = self.model( 2025-08-14T21:40:16.6562141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6562522Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6562890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6563269Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6563612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6563977Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6564354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6564783Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6565168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.6565509Z return self.act(input) 2025-08-14T21:40:16.6565703Z 2025-08-14T21:40:16.6565819Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6566224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6566565Z return mod(**inputs) 2025-08-14T21:40:16.6566932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6567307Z outputs = self.model( 2025-08-14T21:40:16.6567667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6568051Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6568473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6568845Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6569181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6569523Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6569896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:40:16.6570274Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.6570405Z 2025-08-14T21:40:16.6570516Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6570858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6571213Z return mod(**inputs) 2025-08-14T21:40:16.6571567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6571934Z outputs = self.model( 2025-08-14T21:40:16.6572288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6572664Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6573030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6573399Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6573725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6574071Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6574440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6574830Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6575218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6575662Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6575861Z 2025-08-14T21:40:16.6575961Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6576311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6576636Z return mod(**inputs) 2025-08-14T21:40:16.6576967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6577328Z outputs = self.model( 2025-08-14T21:40:16.6577670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6578036Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6578382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6578746Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6579076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6579418Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6579774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6580153Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6580526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6580888Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6581024Z 2025-08-14T21:40:16.6581123Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6581463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6581830Z return mod(**inputs) 2025-08-14T21:40:16.6582169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6582544Z outputs = self.model( 2025-08-14T21:40:16.6582881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6583242Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6583601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6583968Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6584304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6584417Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6584663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6584761Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6585001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6585091Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6585095Z 2025-08-14T21:40:16.6585173Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6585249Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6585330Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6585403Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6585501Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6585703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6585769Z return mod(**inputs) 2025-08-14T21:40:16.6586018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6586086Z outputs = self.model( 2025-08-14T21:40:16.6586327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6586408Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6586651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6586720Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6586934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6587008Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6587248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6587333Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6587565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6587664Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6587941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6588065Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6588076Z 2025-08-14T21:40:16.6588173Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6588364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6588433Z return mod(**inputs) 2025-08-14T21:40:16.6588672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6588739Z outputs = self.model( 2025-08-14T21:40:16.6589022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6589094Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6589340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6589410Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6589624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6589706Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6589944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6590041Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6590313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6590408Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6590687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6590788Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6590792Z 2025-08-14T21:40:16.6590887Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6591080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6591144Z return mod(**inputs) 2025-08-14T21:40:16.6591391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6591458Z outputs = self.model( 2025-08-14T21:40:16.6591737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6591820Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6592056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6592126Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6592342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6592419Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6592664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6592749Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6592982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6593072Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6593076Z 2025-08-14T21:40:16.6593172Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6593373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6593436Z return mod(**inputs) 2025-08-14T21:40:16.6593673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6593747Z outputs = self.model( 2025-08-14T21:40:16.6593987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6594057Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6594300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6594372Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6594591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6594702Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6594937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6595059Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6595063Z 2025-08-14T21:40:16.6595161Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6595353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6595424Z return mod(**inputs) 2025-08-14T21:40:16.6595666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6595739Z outputs = self.model( 2025-08-14T21:40:16.6596021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6596095Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6596346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6596416Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6596638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6596714Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6596950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6597069Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6597275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.6597341Z return self.act(input) 2025-08-14T21:40:16.6597345Z 2025-08-14T21:40:16.6597452Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6597643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6597714Z return mod(**inputs) 2025-08-14T21:40:16.6597957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6598021Z outputs = self.model( 2025-08-14T21:40:16.6598266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6598337Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6598574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6598654Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6598869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6598954Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6599191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:40:16.6599271Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.6599275Z 2025-08-14T21:40:16.6599380Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6599570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6599639Z return mod(**inputs) 2025-08-14T21:40:16.6599878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6599944Z outputs = self.model( 2025-08-14T21:40:16.6600190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6600263Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6600502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6600612Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6600827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6600908Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6601145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6601232Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6601476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6601620Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6601624Z 2025-08-14T21:40:16.6601765Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6601958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6602024Z return mod(**inputs) 2025-08-14T21:40:16.6602275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6602340Z outputs = self.model( 2025-08-14T21:40:16.6602580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6602658Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6602897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6602983Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6603189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6603266Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6603505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6603593Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6603823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6603903Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6603906Z 2025-08-14T21:40:16.6604520Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6604714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6604776Z return mod(**inputs) 2025-08-14T21:40:16.6605011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6605083Z outputs = self.model( 2025-08-14T21:40:16.6605379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6605466Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6605820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6605902Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6606141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6606222Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6606488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6606601Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6606842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6606933Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6606980Z 2025-08-14T21:40:16.6607060Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6607137Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6607220Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6607294Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6607393Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6607599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6607662Z return mod(**inputs) 2025-08-14T21:40:16.6607905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6607970Z outputs = self.model( 2025-08-14T21:40:16.6608209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6608323Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6608557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6608628Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6608843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6608918Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6609154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6609238Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6609469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6609567Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6609844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6609982Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6609985Z 2025-08-14T21:40:16.6610083Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6610272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6610341Z return mod(**inputs) 2025-08-14T21:40:16.6610577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6610642Z outputs = self.model( 2025-08-14T21:40:16.6610886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6610962Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6611206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6611274Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6611482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6611562Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6611794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6611882Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6612114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6612203Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6612482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6612589Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6612592Z 2025-08-14T21:40:16.6612688Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6612923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6612985Z return mod(**inputs) 2025-08-14T21:40:16.6613226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6613290Z outputs = self.model( 2025-08-14T21:40:16.6613521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6613597Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6613828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6613905Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6614147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6614224Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6614463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6614548Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6614781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6614865Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6614869Z 2025-08-14T21:40:16.6614964Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6615159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6615225Z return mod(**inputs) 2025-08-14T21:40:16.6615479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6615553Z outputs = self.model( 2025-08-14T21:40:16.6615819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6615889Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6616131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6616199Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6616416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6616490Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6616725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6616844Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6616850Z 2025-08-14T21:40:16.6616945Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6617144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6617206Z return mod(**inputs) 2025-08-14T21:40:16.6617447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6617519Z outputs = self.model( 2025-08-14T21:40:16.6617759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6617827Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6618071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6618139Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6618361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6618433Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6618704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6618823Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6619019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.6619083Z return self.act(input) 2025-08-14T21:40:16.6619094Z 2025-08-14T21:40:16.6619189Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6619374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6619441Z return mod(**inputs) 2025-08-14T21:40:16.6619670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6619762Z outputs = self.model( 2025-08-14T21:40:16.6620003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6620074Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6620309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6620378Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6620586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6620666Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6620895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:40:16.6620970Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.6620981Z 2025-08-14T21:40:16.6621075Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6621261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6621332Z return mod(**inputs) 2025-08-14T21:40:16.6621566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6621631Z outputs = self.model( 2025-08-14T21:40:16.6621869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6621938Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6622175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6622243Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6622449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6622533Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6622762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6622849Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6623085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6623225Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6623229Z 2025-08-14T21:40:16.6623331Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6623514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6623575Z return mod(**inputs) 2025-08-14T21:40:16.6623820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6623885Z outputs = self.model( 2025-08-14T21:40:16.6624126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6624243Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6624491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6624564Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6624772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6624844Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6625089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6625176Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6625422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6625541Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6625546Z 2025-08-14T21:40:16.6625647Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6625842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6625905Z return mod(**inputs) 2025-08-14T21:40:16.6626144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6626217Z outputs = self.model( 2025-08-14T21:40:16.6626456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6626532Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6626766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6626835Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6627056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6627132Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6627370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6627465Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6627704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6627793Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6627796Z 2025-08-14T21:40:16.6627873Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6627948Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6628031Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6628103Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6628212Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6628403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6628470Z return mod(**inputs) 2025-08-14T21:40:16.6628719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6628784Z outputs = self.model( 2025-08-14T21:40:16.6629024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6629102Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6629338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6629413Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6629626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6629704Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6629947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6630073Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6630311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6630412Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6630693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6630828Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6630832Z 2025-08-14T21:40:16.6630930Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6631121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6631223Z return mod(**inputs) 2025-08-14T21:40:16.6631470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6631544Z outputs = self.model( 2025-08-14T21:40:16.6631784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6631854Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6632094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6632163Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6632371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6632451Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6632689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6632781Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6633017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6633114Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6633400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6633505Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6633508Z 2025-08-14T21:40:16.6633610Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6633799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6633862Z return mod(**inputs) 2025-08-14T21:40:16.6634112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6634178Z outputs = self.model( 2025-08-14T21:40:16.6634416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6634496Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6634732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6634811Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6635021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6635097Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6635340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6635424Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6635662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6635783Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6635786Z 2025-08-14T21:40:16.6635885Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6636083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6636147Z return mod(**inputs) 2025-08-14T21:40:16.6636388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6636462Z outputs = self.model( 2025-08-14T21:40:16.6636704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6636783Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6637056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6637127Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6637349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6637425Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6637873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6638001Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6638004Z 2025-08-14T21:40:16.6638101Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6638294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6638358Z return mod(**inputs) 2025-08-14T21:40:16.6638593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6638669Z outputs = self.model( 2025-08-14T21:40:16.6638909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6638986Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6639237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6639306Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6639518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6639591Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6639820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6639940Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6640139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.6640210Z return self.act(input) 2025-08-14T21:40:16.6640216Z 2025-08-14T21:40:16.6640310Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6640493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6640562Z return mod(**inputs) 2025-08-14T21:40:16.6640792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6640855Z outputs = self.model( 2025-08-14T21:40:16.6641092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6641161Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6641396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6641466Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6641669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6641816Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6642049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:40:16.6642125Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.6642135Z 2025-08-14T21:40:16.6642229Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6642416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6642484Z return mod(**inputs) 2025-08-14T21:40:16.6642721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6642784Z outputs = self.model( 2025-08-14T21:40:16.6643070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6643143Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6643382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6643450Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6643657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6643738Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6643969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6644054Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6644295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6644440Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6644444Z 2025-08-14T21:40:16.6644551Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6644741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6644803Z return mod(**inputs) 2025-08-14T21:40:16.6645052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6645117Z outputs = self.model( 2025-08-14T21:40:16.6645368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6645440Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6645736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6645820Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6646037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6646115Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6646363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6646450Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6646692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6646769Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6646773Z 2025-08-14T21:40:16.6646869Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6647069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6647133Z return mod(**inputs) 2025-08-14T21:40:16.6647380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6647452Z outputs = self.model( 2025-08-14T21:40:16.6647745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6647820Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6648050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6648119Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6648330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6648402Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6648636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6648720Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6648980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6649076Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6649079Z 2025-08-14T21:40:16.6649154Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6649228Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6649307Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6649379Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6649491Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6649672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6649731Z return mod(**inputs) 2025-08-14T21:40:16.6649962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6650023Z outputs = self.model( 2025-08-14T21:40:16.6650252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6650328Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6650554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6650627Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6650831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6650904Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6651137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6651221Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6651444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6651547Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6651825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6651955Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6651958Z 2025-08-14T21:40:16.6652051Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6652232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6652300Z return mod(**inputs) 2025-08-14T21:40:16.6652523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6652590Z outputs = self.model( 2025-08-14T21:40:16.6652816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6652885Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6653110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6653212Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6653413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6653490Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6653712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6653799Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6654020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6654108Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6654411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6654515Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6654521Z 2025-08-14T21:40:16.6654622Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6654809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6654870Z return mod(**inputs) 2025-08-14T21:40:16.6655110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6655173Z outputs = self.model( 2025-08-14T21:40:16.6655404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6655480Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6655715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6655789Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6655993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6656069Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6656316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6656398Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6656621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6656704Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6656708Z 2025-08-14T21:40:16.6656804Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6656994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6657059Z return mod(**inputs) 2025-08-14T21:40:16.6657293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6657367Z outputs = self.model( 2025-08-14T21:40:16.6657599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6657674Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6657902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6657968Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6658180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6658253Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6658485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6658603Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6658639Z 2025-08-14T21:40:16.6658735Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6658926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6658986Z return mod(**inputs) 2025-08-14T21:40:16.6659219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6659288Z outputs = self.model( 2025-08-14T21:40:16.6659521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6659595Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6659826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6659924Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6660136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6660215Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6660445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6660563Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6660768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.6660843Z return self.act(input) 2025-08-14T21:40:16.6660847Z 2025-08-14T21:40:16.6660944Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6661132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6661205Z return mod(**inputs) 2025-08-14T21:40:16.6661444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6661512Z outputs = self.model( 2025-08-14T21:40:16.6661755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6661832Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6662074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6662145Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6662351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6662436Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6662667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:40:16.6662755Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.6662758Z 2025-08-14T21:40:16.6662856Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6663047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6663121Z return mod(**inputs) 2025-08-14T21:40:16.6663355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6663421Z outputs = self.model( 2025-08-14T21:40:16.6663661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6663734Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6663971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6664042Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6664252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6664335Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6664603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6664688Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6664922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6665061Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6665064Z 2025-08-14T21:40:16.6665166Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6665351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6665413Z return mod(**inputs) 2025-08-14T21:40:16.6665689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6665754Z outputs = self.model( 2025-08-14T21:40:16.6666001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6666070Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6666300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6666377Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6666586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6666661Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6666898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6666980Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6667217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6667293Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6667296Z 2025-08-14T21:40:16.6667389Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6667581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6667641Z return mod(**inputs) 2025-08-14T21:40:16.6667885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6667948Z outputs = self.model( 2025-08-14T21:40:16.6668182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6668259Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6668491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6668558Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6668772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6668844Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6669082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6669166Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6669397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6669488Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6669492Z 2025-08-14T21:40:16.6669564Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6669636Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6669715Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6669786Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6669887Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6670116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6670176Z return mod(**inputs) 2025-08-14T21:40:16.6670467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6670528Z outputs = self.model( 2025-08-14T21:40:16.6670755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6670828Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6671052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6671122Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6671931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6672032Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6672264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6672345Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6672574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6672660Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6672923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6673048Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6673051Z 2025-08-14T21:40:16.6673142Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6673324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6673395Z return mod(**inputs) 2025-08-14T21:40:16.6673628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6673698Z outputs = self.model( 2025-08-14T21:40:16.6673969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6674042Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6674281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6674351Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6674572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6674656Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6674884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6674975Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6675204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6675290Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6675570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6675672Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6675675Z 2025-08-14T21:40:16.6675786Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6675968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6676030Z return mod(**inputs) 2025-08-14T21:40:16.6676261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6676362Z outputs = self.model( 2025-08-14T21:40:16.6676590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6676664Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6676889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6676963Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6677163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6677234Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6677461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 312, in forward 2025-08-14T21:40:16.6677577Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:40:16.6677808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6677885Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6677888Z 2025-08-14T21:40:16.6677981Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6678167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6678228Z return mod(**inputs) 2025-08-14T21:40:16.6678452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6678521Z outputs = self.model( 2025-08-14T21:40:16.6678746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6678822Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6679043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6679110Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6679316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6679386Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6679605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6679719Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6679723Z 2025-08-14T21:40:16.6679813Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6680000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6680060Z return mod(**inputs) 2025-08-14T21:40:16.6680285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6680355Z outputs = self.model( 2025-08-14T21:40:16.6680578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6680651Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6680870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6680936Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6681140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6681211Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6681430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 323, in forward 2025-08-14T21:40:16.6681546Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6681741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.6681849Z return self.act(input) 2025-08-14T21:40:16.6681852Z 2025-08-14T21:40:16.6681944Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6682125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6682195Z return mod(**inputs) 2025-08-14T21:40:16.6682417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6682485Z outputs = self.model( 2025-08-14T21:40:16.6682710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1270, in forward 2025-08-14T21:40:16.6682776Z encoder_outputs = self.encoder( 2025-08-14T21:40:16.6683037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 869, in forward 2025-08-14T21:40:16.6683104Z layer_outputs = encoder_layer( 2025-08-14T21:40:16.6683309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6683389Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6683612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 325, in forward 2025-08-14T21:40:16.6683691Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.6683695Z 2025-08-14T21:40:16.6683788Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6683974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6684041Z return mod(**inputs) 2025-08-14T21:40:16.6684277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6684341Z outputs = self.model( 2025-08-14T21:40:16.6684582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6684653Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6684897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6684964Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6685167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6685247Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6685476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6685651Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6685891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6686032Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6686039Z 2025-08-14T21:40:16.6686146Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6686344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6686407Z return mod(**inputs) 2025-08-14T21:40:16.6686659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6686726Z outputs = self.model( 2025-08-14T21:40:16.6686973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6687045Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6687290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6687369Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6687636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6687711Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6687950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6688048Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6688296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6688370Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6688373Z 2025-08-14T21:40:16.6688467Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6688702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6688769Z return mod(**inputs) 2025-08-14T21:40:16.6689010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6689076Z outputs = self.model( 2025-08-14T21:40:16.6689306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6689383Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6689619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6689689Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6689907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6689982Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6690231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6690323Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6690553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6690639Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6690642Z 2025-08-14T21:40:16.6690716Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6690795Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6690867Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6690938Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6691038Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6691221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6691282Z return mod(**inputs) 2025-08-14T21:40:16.6691524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6691587Z outputs = self.model( 2025-08-14T21:40:16.6691821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6691898Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6692129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6692204Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6692407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6692478Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6692715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6692805Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6693043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6693168Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6693443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6693577Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6693580Z 2025-08-14T21:40:16.6693675Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6693868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6693929Z return mod(**inputs) 2025-08-14T21:40:16.6694165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6694236Z outputs = self.model( 2025-08-14T21:40:16.6694502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6694575Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6694819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6694887Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6695110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6695184Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6695416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6695514Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6695747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6695843Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6696129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6696237Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6696240Z 2025-08-14T21:40:16.6696343Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6696693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6696764Z return mod(**inputs) 2025-08-14T21:40:16.6697010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6697074Z outputs = self.model( 2025-08-14T21:40:16.6697318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6697391Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6697625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6697706Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6697916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6697990Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6698231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6698322Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6698561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6698637Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6698640Z 2025-08-14T21:40:16.6698737Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6698933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6699033Z return mod(**inputs) 2025-08-14T21:40:16.6699274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6699345Z outputs = self.model( 2025-08-14T21:40:16.6699583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6699660Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6699894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6699961Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6700179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6700282Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6700521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6700625Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6700856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6701004Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6701008Z 2025-08-14T21:40:16.6701103Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6701288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6701357Z return mod(**inputs) 2025-08-14T21:40:16.6701588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6701660Z outputs = self.model( 2025-08-14T21:40:16.6701893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6701963Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6702202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6702269Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6702482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6702555Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6702784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6702890Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6703123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6703196Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6703201Z 2025-08-14T21:40:16.6703301Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6703485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6703553Z return mod(**inputs) 2025-08-14T21:40:16.6703783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6703848Z outputs = self.model( 2025-08-14T21:40:16.6704086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6704155Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6704386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6704463Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6704667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6704784Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6705015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6705114Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6705352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6705432Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6705435Z 2025-08-14T21:40:16.6705516Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6705589Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6705661Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6705740Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6705869Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6706058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6706133Z return mod(**inputs) 2025-08-14T21:40:16.6706367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6706437Z outputs = self.model( 2025-08-14T21:40:16.6706671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6706740Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6706980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6707047Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6707257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6707341Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6707573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6707682Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6707926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6708014Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6708284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6708403Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6708406Z 2025-08-14T21:40:16.6708505Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6708688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6708749Z return mod(**inputs) 2025-08-14T21:40:16.6708984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6709046Z outputs = self.model( 2025-08-14T21:40:16.6709273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6709348Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6709574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6709646Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6709849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6709922Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6710157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6710369Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6710593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6710688Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6710953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6711059Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6711062Z 2025-08-14T21:40:16.6711156Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6711338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6711407Z return mod(**inputs) 2025-08-14T21:40:16.6711677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6711753Z outputs = self.model( 2025-08-14T21:40:16.6711980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6712049Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6712281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6712348Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6712546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6712625Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6712847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6712954Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6713178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6713253Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6713256Z 2025-08-14T21:40:16.6713357Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6713538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6713605Z return mod(**inputs) 2025-08-14T21:40:16.6713827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6713890Z outputs = self.model( 2025-08-14T21:40:16.6714128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6714196Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6714430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6714507Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6714712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6714792Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6715024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.6715137Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6715141Z 2025-08-14T21:40:16.6715242Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6715430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6715492Z return mod(**inputs) 2025-08-14T21:40:16.6715732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6715796Z outputs = self.model( 2025-08-14T21:40:16.6716072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6716141Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6716374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6716449Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6716657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6716749Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6716974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.6717080Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6717312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.6717380Z return self.act(input) 2025-08-14T21:40:16.6717383Z 2025-08-14T21:40:16.6717475Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6717659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6717718Z return mod(**inputs) 2025-08-14T21:40:16.6717946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6718009Z outputs = self.model( 2025-08-14T21:40:16.6718235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6718313Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6718539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6718606Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6718812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6718887Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6719121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:16.6719196Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.6719200Z 2025-08-14T21:40:16.6719294Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6719489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6719553Z return mod(**inputs) 2025-08-14T21:40:16.6719792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6719859Z outputs = self.model( 2025-08-14T21:40:16.6720090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6720170Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6720399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6720467Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6720679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6720752Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6720988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6721082Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6721312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6721460Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6721497Z 2025-08-14T21:40:16.6721593Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6721789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6721857Z return mod(**inputs) 2025-08-14T21:40:16.6722091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6722161Z outputs = self.model( 2025-08-14T21:40:16.6722393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6722462Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6722703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6722800Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6723015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6723102Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6723323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6723420Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6723658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6723732Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6723742Z 2025-08-14T21:40:16.6723836Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6724022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6724092Z return mod(**inputs) 2025-08-14T21:40:16.6724321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6724387Z outputs = self.model( 2025-08-14T21:40:16.6724626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6724696Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6724933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6725003Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6725210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6725291Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6725621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6725738Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6726004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6726097Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6726101Z 2025-08-14T21:40:16.6726192Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6726275Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6726356Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6726445Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6726553Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6726767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6726840Z return mod(**inputs) 2025-08-14T21:40:16.6727088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6727164Z outputs = self.model( 2025-08-14T21:40:16.6727417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6727523Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6727768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6727836Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6728047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6728126Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6728361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6728460Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6728728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6728820Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6729101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6729224Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6729228Z 2025-08-14T21:40:16.6729329Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6729515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6729578Z return mod(**inputs) 2025-08-14T21:40:16.6729813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6729878Z outputs = self.model( 2025-08-14T21:40:16.6730119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6730197Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6730430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6730505Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6730713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6730787Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6731029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6731119Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6731354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6731446Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6731717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6731831Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6731835Z 2025-08-14T21:40:16.6731929Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6732116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6735130Z return mod(**inputs) 2025-08-14T21:40:16.6735387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6735462Z outputs = self.model( 2025-08-14T21:40:16.6735705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6735778Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6736045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6736151Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6736362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6736444Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6736676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6736800Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6737039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6737115Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6737118Z 2025-08-14T21:40:16.6737210Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6737438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6737502Z return mod(**inputs) 2025-08-14T21:40:16.6737924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6737994Z outputs = self.model( 2025-08-14T21:40:16.6738229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6738308Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6738542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6738611Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6738827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6738900Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6739148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6739258Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6739504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6739661Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6739665Z 2025-08-14T21:40:16.6739770Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6739976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6740042Z return mod(**inputs) 2025-08-14T21:40:16.6740299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6740374Z outputs = self.model( 2025-08-14T21:40:16.6740616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6740690Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6740944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6741012Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6741226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6741382Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6741621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6741731Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6741966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6742045Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6742056Z 2025-08-14T21:40:16.6742153Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6742374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6742444Z return mod(**inputs) 2025-08-14T21:40:16.6742683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6742748Z outputs = self.model( 2025-08-14T21:40:16.6742996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6743066Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6743312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6743381Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6743636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6743723Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6743962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6744065Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6744310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6744393Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6744396Z 2025-08-14T21:40:16.6744479Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6744555Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6744628Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6744708Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6744812Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6745002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6745077Z return mod(**inputs) 2025-08-14T21:40:16.6745315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6745386Z outputs = self.model( 2025-08-14T21:40:16.6745626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6745700Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6745947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6746016Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6746226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6746311Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6746550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6746663Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6746898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6746990Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6747317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6747442Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6747446Z 2025-08-14T21:40:16.6747550Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6747739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6747805Z return mod(**inputs) 2025-08-14T21:40:16.6748051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6748136Z outputs = self.model( 2025-08-14T21:40:16.6748379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6748457Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6748699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6748777Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6748992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6749068Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6749347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6749451Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6749697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6749793Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6750072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6750186Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6750190Z 2025-08-14T21:40:16.6750289Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6750477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6750547Z return mod(**inputs) 2025-08-14T21:40:16.6750789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6750861Z outputs = self.model( 2025-08-14T21:40:16.6751100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6751173Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6751416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6751485Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6751699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6751781Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6752017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6752125Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6752409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6752484Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6752487Z 2025-08-14T21:40:16.6752587Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6752768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6752833Z return mod(**inputs) 2025-08-14T21:40:16.6753096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6753185Z outputs = self.model( 2025-08-14T21:40:16.6753429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6753496Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6753734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6753807Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6754031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6754108Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6754334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.6754445Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6754449Z 2025-08-14T21:40:16.6754549Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6754732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6754799Z return mod(**inputs) 2025-08-14T21:40:16.6755033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6755126Z outputs = self.model( 2025-08-14T21:40:16.6755362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6755434Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6755664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6755738Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6755940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6756021Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6756247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.6756356Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6756570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.6756634Z return self.act(input) 2025-08-14T21:40:16.6756639Z 2025-08-14T21:40:16.6756730Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6756917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6756978Z return mod(**inputs) 2025-08-14T21:40:16.6757209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6757273Z outputs = self.model( 2025-08-14T21:40:16.6757496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6757573Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6757798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6757873Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6758070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6758144Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6758373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:16.6758447Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.6758450Z 2025-08-14T21:40:16.6758572Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6758761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6758819Z return mod(**inputs) 2025-08-14T21:40:16.6759051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6759113Z outputs = self.model( 2025-08-14T21:40:16.6759339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6759434Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6759663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6759727Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6759937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6760009Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6760242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6760333Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6760560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6760734Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6760738Z 2025-08-14T21:40:16.6760833Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6761017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6761077Z return mod(**inputs) 2025-08-14T21:40:16.6761301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6761370Z outputs = self.model( 2025-08-14T21:40:16.6761596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6761663Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6761897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6761963Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6762173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6762245Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6762468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6762565Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6762790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6762871Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6762874Z 2025-08-14T21:40:16.6762966Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6763145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6763211Z return mod(**inputs) 2025-08-14T21:40:16.6763437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6763501Z outputs = self.model( 2025-08-14T21:40:16.6763740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6763810Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6764048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6764135Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6764346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6764427Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6764660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6764757Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6764994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6765095Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6765099Z 2025-08-14T21:40:16.6765181Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6765259Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6765335Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6765417Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6765587Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6765788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6765863Z return mod(**inputs) 2025-08-14T21:40:16.6766106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6766229Z outputs = self.model( 2025-08-14T21:40:16.6766490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6766568Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6766828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6766896Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6767100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6767182Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6767409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6767508Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6767738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6767828Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6768116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6768237Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6768241Z 2025-08-14T21:40:16.6768343Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6768522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6768584Z return mod(**inputs) 2025-08-14T21:40:16.6768812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6768875Z outputs = self.model( 2025-08-14T21:40:16.6769099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6769175Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6769399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6769475Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6769673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6769746Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6769998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6770085Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6770316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6770407Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6770679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6770807Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6770811Z 2025-08-14T21:40:16.6770907Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6771100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6771162Z return mod(**inputs) 2025-08-14T21:40:16.6771442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6771513Z outputs = self.model( 2025-08-14T21:40:16.6771739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6771806Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6772072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6772146Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6772359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6772433Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6772661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6772759Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6772992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6773067Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6773070Z 2025-08-14T21:40:16.6773173Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6773361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6773428Z return mod(**inputs) 2025-08-14T21:40:16.6773662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6773726Z outputs = self.model( 2025-08-14T21:40:16.6773962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6774030Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6774261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6774337Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6774541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6774619Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6774850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6774950Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6775187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6775325Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6775329Z 2025-08-14T21:40:16.6775429Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6775630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6775693Z return mod(**inputs) 2025-08-14T21:40:16.6775933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6775995Z outputs = self.model( 2025-08-14T21:40:16.6776228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6776303Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6776555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6776630Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6776835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6776909Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6777147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6777244Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6777481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6777556Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6777598Z 2025-08-14T21:40:16.6777696Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6777889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6777951Z return mod(**inputs) 2025-08-14T21:40:16.6778183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6778256Z outputs = self.model( 2025-08-14T21:40:16.6778489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6778567Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6778800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6778866Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6779083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6779156Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6779386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6779492Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6779721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6779808Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6779812Z 2025-08-14T21:40:16.6779887Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6779960Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6780038Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6780108Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6780217Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6780403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6780471Z return mod(**inputs) 2025-08-14T21:40:16.6780703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6780767Z outputs = self.model( 2025-08-14T21:40:16.6781003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6781072Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6781324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6781399Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6781604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6781686Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6781913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6782028Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6782268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6782360Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6782632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6782766Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6782769Z 2025-08-14T21:40:16.6782864Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6783056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6783117Z return mod(**inputs) 2025-08-14T21:40:16.6783377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6783451Z outputs = self.model( 2025-08-14T21:40:16.6783686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6783763Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6783993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6784063Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6784282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6784356Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6784593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6784706Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6784942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6785044Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6785332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6785432Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6785437Z 2025-08-14T21:40:16.6785542Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6785726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6785794Z return mod(**inputs) 2025-08-14T21:40:16.6786025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6786091Z outputs = self.model( 2025-08-14T21:40:16.6786328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6786401Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6786632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6786707Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6786913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6787013Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6787242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6787339Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6787577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6787654Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6787677Z 2025-08-14T21:40:16.6787781Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6787965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6788029Z return mod(**inputs) 2025-08-14T21:40:16.6788271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6788336Z outputs = self.model( 2025-08-14T21:40:16.6788568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6788644Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6788873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6788981Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6789188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6789263Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6789500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.6789611Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6789616Z 2025-08-14T21:40:16.6789711Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6789903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6789964Z return mod(**inputs) 2025-08-14T21:40:16.6790202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6790264Z outputs = self.model( 2025-08-14T21:40:16.6790498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6790574Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6790803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6790876Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6791083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6791161Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6791400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.6791510Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6791710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.6791784Z return self.act(input) 2025-08-14T21:40:16.6791787Z 2025-08-14T21:40:16.6791885Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6792081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6792143Z return mod(**inputs) 2025-08-14T21:40:16.6792374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6792471Z outputs = self.model( 2025-08-14T21:40:16.6792703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6792771Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6793009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6793076Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6793289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6793382Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6793614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:16.6793698Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.6793702Z 2025-08-14T21:40:16.6793795Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6793986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6794048Z return mod(**inputs) 2025-08-14T21:40:16.6794280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6794348Z outputs = self.model( 2025-08-14T21:40:16.6794611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6794680Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6794921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6794989Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6795211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6795281Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6795506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6795603Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6795828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6795971Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6795981Z 2025-08-14T21:40:16.6796073Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6796255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6796323Z return mod(**inputs) 2025-08-14T21:40:16.6796551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6796612Z outputs = self.model( 2025-08-14T21:40:16.6796848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6796915Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6797146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6797210Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6797413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6797491Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6797715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6797805Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6798036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6798127Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6798130Z 2025-08-14T21:40:16.6798229Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6798410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6798469Z return mod(**inputs) 2025-08-14T21:40:16.6798705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6798767Z outputs = self.model( 2025-08-14T21:40:16.6799017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6799085Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6799308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6799379Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6799582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6799653Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6799882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6799971Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6800241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6800322Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6800325Z 2025-08-14T21:40:16.6800399Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6800479Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6800551Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6800621Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6800723Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6800911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6800981Z return mod(**inputs) 2025-08-14T21:40:16.6801210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6801276Z outputs = self.model( 2025-08-14T21:40:16.6801511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6801580Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6801808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6801886Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6802091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6802172Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6802401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6802491Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6802725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6802817Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6803096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6803220Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6803224Z 2025-08-14T21:40:16.6803320Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6803510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6803589Z return mod(**inputs) 2025-08-14T21:40:16.6803818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6803888Z outputs = self.model( 2025-08-14T21:40:16.6804117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6804196Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6804422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6804504Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6804716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6804789Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6805024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6805126Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6805364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6805464Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6805863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6805977Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6805983Z 2025-08-14T21:40:16.6806091Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6806282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6806356Z return mod(**inputs) 2025-08-14T21:40:16.6806592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6806661Z outputs = self.model( 2025-08-14T21:40:16.6806905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6806978Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6807221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6807301Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6807511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6807597Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6807837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6807939Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6808180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6808258Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6808261Z 2025-08-14T21:40:16.6808364Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6808547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6808611Z return mod(**inputs) 2025-08-14T21:40:16.6808846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6808913Z outputs = self.model( 2025-08-14T21:40:16.6809153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6809227Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6809455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6809554Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6809761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6809835Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6810076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6810176Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6810425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6810573Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6810577Z 2025-08-14T21:40:16.6810672Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6810863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6810927Z return mod(**inputs) 2025-08-14T21:40:16.6811160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6811231Z outputs = self.model( 2025-08-14T21:40:16.6811465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6811575Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6811807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6811877Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6812092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6812165Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6812394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6812501Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6812728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6812809Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6812812Z 2025-08-14T21:40:16.6812908Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6813093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6813164Z return mod(**inputs) 2025-08-14T21:40:16.6813395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6813464Z outputs = self.model( 2025-08-14T21:40:16.6813693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6813763Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6813998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6814064Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6814268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6814352Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6814579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6814686Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6814914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6814993Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6815019Z 2025-08-14T21:40:16.6815101Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6815173Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6815243Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6815319Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6815414Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6815609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6815670Z return mod(**inputs) 2025-08-14T21:40:16.6815903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6816037Z outputs = self.model( 2025-08-14T21:40:16.6816271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6816339Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6816579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6816647Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6816858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6816930Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6817188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6817296Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6817528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6817626Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6817899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6818026Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6818029Z 2025-08-14T21:40:16.6818132Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6818319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6818381Z return mod(**inputs) 2025-08-14T21:40:16.6818625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6818690Z outputs = self.model( 2025-08-14T21:40:16.6818935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6819008Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6819242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6819320Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6819526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6819609Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6819841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6819942Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6820179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6820274Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6820551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6820659Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6820683Z 2025-08-14T21:40:16.6820779Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6820969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6821032Z return mod(**inputs) 2025-08-14T21:40:16.6821266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6821338Z outputs = self.model( 2025-08-14T21:40:16.6821572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6821676Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6821908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6821975Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6822191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6822265Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6822495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6822600Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6822864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6822950Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6822953Z 2025-08-14T21:40:16.6823050Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6823234Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6823304Z return mod(**inputs) 2025-08-14T21:40:16.6823533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6823597Z outputs = self.model( 2025-08-14T21:40:16.6823836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6823907Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6824144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6824212Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6824419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6824501Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6824732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.6824848Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6824852Z 2025-08-14T21:40:16.6824946Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6825133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6825212Z return mod(**inputs) 2025-08-14T21:40:16.6825439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6825499Z outputs = self.model( 2025-08-14T21:40:16.6825733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6825803Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6826039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6826105Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6826308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6826420Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6826643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.6826750Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6826949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.6827015Z return self.act(input) 2025-08-14T21:40:16.6827018Z 2025-08-14T21:40:16.6827114Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6827316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6827375Z return mod(**inputs) 2025-08-14T21:40:16.6827609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6827670Z outputs = self.model( 2025-08-14T21:40:16.6827904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6827971Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6828197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6828267Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6828497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6828570Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6828804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:16.6828880Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.6828883Z 2025-08-14T21:40:16.6828982Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6829164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6829228Z return mod(**inputs) 2025-08-14T21:40:16.6829462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6829521Z outputs = self.model( 2025-08-14T21:40:16.6829748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6829827Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6830065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6830142Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6830354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6830428Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6830681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6830774Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6831014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6831156Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6831161Z 2025-08-14T21:40:16.6831257Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6831452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6831515Z return mod(**inputs) 2025-08-14T21:40:16.6831759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6831828Z outputs = self.model( 2025-08-14T21:40:16.6832057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6832167Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6832393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6832460Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6832672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6832742Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6832991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6833082Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6833307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6833387Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6833392Z 2025-08-14T21:40:16.6833484Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6833667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6833735Z return mod(**inputs) 2025-08-14T21:40:16.6833963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6834061Z outputs = self.model( 2025-08-14T21:40:16.6834291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6834359Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6834595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6834663Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6834868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6834950Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6835183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6835287Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6835521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6835602Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6835607Z 2025-08-14T21:40:16.6835688Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6835761Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6835841Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6835914Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6836012Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6836210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6836273Z return mod(**inputs) 2025-08-14T21:40:16.6836522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6836592Z outputs = self.model( 2025-08-14T21:40:16.6836827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6836904Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6837139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6837206Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6837417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6837491Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6837873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6837980Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6838207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6838305Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6838577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6838741Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6838745Z 2025-08-14T21:40:16.6838849Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6839038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6839109Z return mod(**inputs) 2025-08-14T21:40:16.6839342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6839405Z outputs = self.model( 2025-08-14T21:40:16.6839642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6839712Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6839986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6840066Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6840276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6840358Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6840593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6840686Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6840926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6841016Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6841296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6841405Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6841410Z 2025-08-14T21:40:16.6841506Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6841702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6841765Z return mod(**inputs) 2025-08-14T21:40:16.6842008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6842082Z outputs = self.model( 2025-08-14T21:40:16.6842325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6842402Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6842640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6842711Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6842929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6843004Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6843240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6843340Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6843574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6843699Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6843702Z 2025-08-14T21:40:16.6843797Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6843986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6844059Z return mod(**inputs) 2025-08-14T21:40:16.6844296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6844384Z outputs = self.model( 2025-08-14T21:40:16.6844630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6844702Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6844954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6845026Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6845241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6845325Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6845711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6845868Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6846111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6846263Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6846267Z 2025-08-14T21:40:16.6846376Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6846572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6846648Z return mod(**inputs) 2025-08-14T21:40:16.6846898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6846962Z outputs = self.model( 2025-08-14T21:40:16.6847201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6847273Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6847506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6847585Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6847789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6847871Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6848100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6848201Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6848436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6848512Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6848515Z 2025-08-14T21:40:16.6848610Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6848803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6848866Z return mod(**inputs) 2025-08-14T21:40:16.6849104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6849167Z outputs = self.model( 2025-08-14T21:40:16.6849395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6849489Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6849720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6849794Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6849998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6850072Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6850307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6850424Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6850651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6850738Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6850741Z 2025-08-14T21:40:16.6850816Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6850898Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6850970Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6851039Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6851139Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6851319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6851417Z return mod(**inputs) 2025-08-14T21:40:16.6851662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6851728Z outputs = self.model( 2025-08-14T21:40:16.6851966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6852034Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6852270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6852350Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6852562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6852638Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6852883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6852986Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6853232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6853325Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6853604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6853741Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6853745Z 2025-08-14T21:40:16.6853843Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6854048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6854118Z return mod(**inputs) 2025-08-14T21:40:16.6854382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6854459Z outputs = self.model( 2025-08-14T21:40:16.6854728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6854801Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6855050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6855121Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6855367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6855447Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6855694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6855804Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6856052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6856166Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6856438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6856538Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6856541Z 2025-08-14T21:40:16.6856641Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6856827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6856889Z return mod(**inputs) 2025-08-14T21:40:16.6857126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6857190Z outputs = self.model( 2025-08-14T21:40:16.6857456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6857526Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6857755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6857829Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6858031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6858104Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6858339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6858436Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6858669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6858746Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6858750Z 2025-08-14T21:40:16.6858844Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6859039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6859101Z return mod(**inputs) 2025-08-14T21:40:16.6859338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6859401Z outputs = self.model( 2025-08-14T21:40:16.6859633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6859708Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6859934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6860003Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6860217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6860290Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6860527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.6860637Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6860641Z 2025-08-14T21:40:16.6860735Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6860949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6861012Z return mod(**inputs) 2025-08-14T21:40:16.6861251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6861314Z outputs = self.model( 2025-08-14T21:40:16.6861549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6861625Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6861883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6861949Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6862159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6862231Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6862468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.6862578Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6862774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.6862846Z return self.act(input) 2025-08-14T21:40:16.6862849Z 2025-08-14T21:40:16.6862977Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6863160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6863232Z return mod(**inputs) 2025-08-14T21:40:16.6863459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6863531Z outputs = self.model( 2025-08-14T21:40:16.6863759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6863830Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6864065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6864134Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6864342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6864417Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6864655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:16.6864745Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.6864748Z 2025-08-14T21:40:16.6864847Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6865039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6865114Z return mod(**inputs) 2025-08-14T21:40:16.6865352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6865424Z outputs = self.model( 2025-08-14T21:40:16.6865666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6865743Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6865990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6866063Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6866277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6866372Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6866603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6866725Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6866956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6867096Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6867101Z 2025-08-14T21:40:16.6867206Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6867392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6867480Z return mod(**inputs) 2025-08-14T21:40:16.6867717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6867780Z outputs = self.model( 2025-08-14T21:40:16.6868022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6868093Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6868324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6868399Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6868606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6868715Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6868944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6869037Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6869270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6869344Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6869349Z 2025-08-14T21:40:16.6869448Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6869631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6869692Z return mod(**inputs) 2025-08-14T21:40:16.6869930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6869993Z outputs = self.model( 2025-08-14T21:40:16.6870225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6870303Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6870530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6870604Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6870810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6870893Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6871121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6871211Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6871433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6871518Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6871522Z 2025-08-14T21:40:16.6871596Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6871673Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6871744Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6871812Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6871912Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6872095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6872177Z return mod(**inputs) 2025-08-14T21:40:16.6872420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6872483Z outputs = self.model( 2025-08-14T21:40:16.6872722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6872793Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6873025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6873119Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6873328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6873410Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6873641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6873733Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6873970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6874061Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6874369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6874506Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6874509Z 2025-08-14T21:40:16.6874606Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6874798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6874860Z return mod(**inputs) 2025-08-14T21:40:16.6875093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6875164Z outputs = self.model( 2025-08-14T21:40:16.6875395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6875472Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6875707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6875774Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6875984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6876057Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6876285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6876385Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6876618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6876715Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6876987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6877091Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6877094Z 2025-08-14T21:40:16.6877198Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6877388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6877458Z return mod(**inputs) 2025-08-14T21:40:16.6877694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6877759Z outputs = self.model( 2025-08-14T21:40:16.6878030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6878102Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6878331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6878407Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6878612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6878707Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6878937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6879027Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6879263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6879341Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6879344Z 2025-08-14T21:40:16.6879438Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6879630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6879692Z return mod(**inputs) 2025-08-14T21:40:16.6879960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6880027Z outputs = self.model( 2025-08-14T21:40:16.6880259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6880338Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6880573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6880652Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6880862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6880936Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6881176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6881280Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6881517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6881668Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6881671Z 2025-08-14T21:40:16.6881766Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6881961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6882024Z return mod(**inputs) 2025-08-14T21:40:16.6882260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6882333Z outputs = self.model( 2025-08-14T21:40:16.6882567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6882636Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6882879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6882949Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6883163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6883236Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6883469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6883609Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6883844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6883928Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6883931Z 2025-08-14T21:40:16.6884031Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6884227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6884300Z return mod(**inputs) 2025-08-14T21:40:16.6884566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6884632Z outputs = self.model( 2025-08-14T21:40:16.6884881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6884953Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6885203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6885273Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6885486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6885645Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6885959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6886081Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6886341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6886434Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6886438Z 2025-08-14T21:40:16.6886532Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6886619Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6886703Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6886794Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6886903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6887130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6887198Z return mod(**inputs) 2025-08-14T21:40:16.6887449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6887528Z outputs = self.model( 2025-08-14T21:40:16.6887776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6887851Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6888106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6888183Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6888411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6888490Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6888734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6888854Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6889097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6889196Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6889492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6889625Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6889651Z 2025-08-14T21:40:16.6889762Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6889959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6890024Z return mod(**inputs) 2025-08-14T21:40:16.6890286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6890354Z outputs = self.model( 2025-08-14T21:40:16.6890599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6890688Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6890928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6891003Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6891211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6891289Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6891536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6891637Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6891916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6892010Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6892291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6892400Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6892403Z 2025-08-14T21:40:16.6892499Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6892697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6892760Z return mod(**inputs) 2025-08-14T21:40:16.6893001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6893078Z outputs = self.model( 2025-08-14T21:40:16.6893321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6893393Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6893641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6893710Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6893930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6894005Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6894244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6894353Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6894590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6894666Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6894680Z 2025-08-14T21:40:16.6894779Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6894976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6895048Z return mod(**inputs) 2025-08-14T21:40:16.6895296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6895363Z outputs = self.model( 2025-08-14T21:40:16.6895616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6895711Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6895962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6896032Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6896264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6896346Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6896599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.6896713Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6896724Z 2025-08-14T21:40:16.6896824Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6897010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6897083Z return mod(**inputs) 2025-08-14T21:40:16.6897382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6897448Z outputs = self.model( 2025-08-14T21:40:16.6897689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6898068Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6898319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6898391Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6898604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6898688Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6898926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.6899040Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6899252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.6899318Z return self.act(input) 2025-08-14T21:40:16.6899322Z 2025-08-14T21:40:16.6899430Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6899620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6899685Z return mod(**inputs) 2025-08-14T21:40:16.6899931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6899995Z outputs = self.model( 2025-08-14T21:40:16.6900287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6900366Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6900600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6900674Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6900879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6900956Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6901198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:16.6901277Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.6901281Z 2025-08-14T21:40:16.6901383Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6901576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6901639Z return mod(**inputs) 2025-08-14T21:40:16.6901909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6901974Z outputs = self.model( 2025-08-14T21:40:16.6902213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6902290Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6902530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6902624Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6902850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6902923Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6903160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6903255Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6903482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6903629Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6903633Z 2025-08-14T21:40:16.6903729Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6903953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6904019Z return mod(**inputs) 2025-08-14T21:40:16.6904253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6904324Z outputs = self.model( 2025-08-14T21:40:16.6904562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6904643Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6904885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6904956Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6905177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6905251Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6905490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6905595Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6905832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6905916Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6905919Z 2025-08-14T21:40:16.6906016Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6906208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6906278Z return mod(**inputs) 2025-08-14T21:40:16.6906518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6906593Z outputs = self.model( 2025-08-14T21:40:16.6906836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6906917Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6907153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6907221Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6907429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6927171Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6927605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6927735Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6928026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6928145Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6928153Z 2025-08-14T21:40:16.6928255Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6928455Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6928536Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6928625Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6928745Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6928983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6929060Z return mod(**inputs) 2025-08-14T21:40:16.6929323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6929406Z outputs = self.model( 2025-08-14T21:40:16.6929664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6929808Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6930066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6930143Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6930367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6930449Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6930689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6930795Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6931032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6931129Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6931428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6931561Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6931567Z 2025-08-14T21:40:16.6931681Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6931882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6931947Z return mod(**inputs) 2025-08-14T21:40:16.6932200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6932273Z outputs = self.model( 2025-08-14T21:40:16.6932520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6932593Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6932832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6932914Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6933132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6933215Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6933462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6933559Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6933826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6933920Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6934199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6934311Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6934318Z 2025-08-14T21:40:16.6934419Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6934641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6934707Z return mod(**inputs) 2025-08-14T21:40:16.6934948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6935022Z outputs = self.model( 2025-08-14T21:40:16.6935259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6935334Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6935583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6935655Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6935913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6935995Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6936243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6936347Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6936578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6936663Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6936667Z 2025-08-14T21:40:16.6936775Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6936968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6937033Z return mod(**inputs) 2025-08-14T21:40:16.6937277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6937343Z outputs = self.model( 2025-08-14T21:40:16.6937580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6937838Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6938082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6938162Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6938376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6938453Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6938696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6938803Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6939041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6939200Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6939204Z 2025-08-14T21:40:16.6939305Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6939503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6939568Z return mod(**inputs) 2025-08-14T21:40:16.6939805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6939938Z outputs = self.model( 2025-08-14T21:40:16.6940180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6940261Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6940503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6940573Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6940833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6940908Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6941138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6941249Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6941479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6941563Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6941567Z 2025-08-14T21:40:16.6941663Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6941903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6941976Z return mod(**inputs) 2025-08-14T21:40:16.6942212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6942286Z outputs = self.model( 2025-08-14T21:40:16.6942518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6942586Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6942826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6942896Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6943103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6943184Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6943419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6943527Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6943760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6943842Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6943845Z 2025-08-14T21:40:16.6943928Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6944002Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6944076Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6944153Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6944248Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6944444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6944508Z return mod(**inputs) 2025-08-14T21:40:16.6944742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6944814Z outputs = self.model( 2025-08-14T21:40:16.6945049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6945116Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6945352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6945450Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6945671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6945746Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6945982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6946093Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6946335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6946445Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6946726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6946850Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6946855Z 2025-08-14T21:40:16.6946956Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6947141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6947203Z return mod(**inputs) 2025-08-14T21:40:16.6947445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6947509Z outputs = self.model( 2025-08-14T21:40:16.6947780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6947852Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6948084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6948160Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6948364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6948439Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6948679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6948776Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6949015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6949105Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6949376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6949483Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6949486Z 2025-08-14T21:40:16.6949580Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6949770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6949833Z return mod(**inputs) 2025-08-14T21:40:16.6950063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6950135Z outputs = self.model( 2025-08-14T21:40:16.6950368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6950438Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6950676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6950745Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6950955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6951027Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6951253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6951383Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6951610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6951692Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6951695Z 2025-08-14T21:40:16.6951793Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6951978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6952064Z return mod(**inputs) 2025-08-14T21:40:16.6952296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6952359Z outputs = self.model( 2025-08-14T21:40:16.6952593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6952664Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6952898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6952966Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6953169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6953280Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6953511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.6953627Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6953637Z 2025-08-14T21:40:16.6953730Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6953916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6953986Z return mod(**inputs) 2025-08-14T21:40:16.6954219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6954283Z outputs = self.model( 2025-08-14T21:40:16.6954530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6954601Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6954848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6954919Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6955131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6955215Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6955452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.6955577Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6955783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.6955851Z return self.act(input) 2025-08-14T21:40:16.6955854Z 2025-08-14T21:40:16.6955958Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6956147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6956208Z return mod(**inputs) 2025-08-14T21:40:16.6956453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6956516Z outputs = self.model( 2025-08-14T21:40:16.6956747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6956824Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6957075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6957150Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6957356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6957429Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6957666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:16.6957761Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.6957765Z 2025-08-14T21:40:16.6957867Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6958051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6958113Z return mod(**inputs) 2025-08-14T21:40:16.6958352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6958416Z outputs = self.model( 2025-08-14T21:40:16.6958647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6958722Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6958984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6959062Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6959269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6959341Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6959578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6959671Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6959908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6960051Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6960055Z 2025-08-14T21:40:16.6960149Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6960361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6960421Z return mod(**inputs) 2025-08-14T21:40:16.6960654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6960725Z outputs = self.model( 2025-08-14T21:40:16.6960955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6961030Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6961262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6961330Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6961540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6961614Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6961850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6961951Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6962179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6962261Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6962264Z 2025-08-14T21:40:16.6962358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6962558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6962627Z return mod(**inputs) 2025-08-14T21:40:16.6962856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6962926Z outputs = self.model( 2025-08-14T21:40:16.6963158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6963226Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6963482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6963549Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6963753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6963830Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6964061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6964157Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6964388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6964469Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6964509Z 2025-08-14T21:40:16.6964595Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6964671Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6964747Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6964827Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6964924Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6965121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6965184Z return mod(**inputs) 2025-08-14T21:40:16.6965424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6965496Z outputs = self.model( 2025-08-14T21:40:16.6965819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6965895Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6966150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6966229Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6966472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6966555Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6966816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6966932Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6967195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6967318Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6967600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6967728Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6967734Z 2025-08-14T21:40:16.6967840Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6968030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6968096Z return mod(**inputs) 2025-08-14T21:40:16.6968340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6968429Z outputs = self.model( 2025-08-14T21:40:16.6968674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6968747Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6968980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6969059Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6969268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6969368Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6969605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6969699Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6969943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6970038Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6970318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6970430Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6970434Z 2025-08-14T21:40:16.6970563Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6970766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6970831Z return mod(**inputs) 2025-08-14T21:40:16.6971077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6971151Z outputs = self.model( 2025-08-14T21:40:16.6971388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6971469Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6971710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6971779Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6971998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6972076Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6972310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6972411Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6972645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6972730Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6972734Z 2025-08-14T21:40:16.6972830Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6973018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6973088Z return mod(**inputs) 2025-08-14T21:40:16.6973332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6973400Z outputs = self.model( 2025-08-14T21:40:16.6973648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6973720Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6973964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6974032Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6974243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6974342Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6974578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6974687Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6974925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6975071Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6975090Z 2025-08-14T21:40:16.6975197Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6975387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6975451Z return mod(**inputs) 2025-08-14T21:40:16.6975696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6975763Z outputs = self.model( 2025-08-14T21:40:16.6976053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6976126Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6976368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6976474Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6976691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6976772Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6977009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6977110Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6977354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6977440Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6977443Z 2025-08-14T21:40:16.6977540Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6977736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6977800Z return mod(**inputs) 2025-08-14T21:40:16.6978041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6978113Z outputs = self.model( 2025-08-14T21:40:16.6978351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6978421Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6978670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6978739Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6978963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6979035Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6979267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6979372Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6979611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.6979694Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.6979705Z 2025-08-14T21:40:16.6979781Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6979855Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6979934Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6980024Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.6980123Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6980319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6980384Z return mod(**inputs) 2025-08-14T21:40:16.6980624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6980698Z outputs = self.model( 2025-08-14T21:40:16.6980933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6981033Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6981271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6981340Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6981558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6981632Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6981867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6981976Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6982249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6982352Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6982624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.6982749Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.6982753Z 2025-08-14T21:40:16.6982855Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6983041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6983109Z return mod(**inputs) 2025-08-14T21:40:16.6983339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6983404Z outputs = self.model( 2025-08-14T21:40:16.6983645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6983714Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6983946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6984020Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6984222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6984305Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6984536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6984634Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6984869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.6984959Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.6985238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.6985341Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.6985344Z 2025-08-14T21:40:16.6985438Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6985630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6985711Z return mod(**inputs) 2025-08-14T21:40:16.6985940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6986012Z outputs = self.model( 2025-08-14T21:40:16.6986239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6986314Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6986543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6986635Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6986850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6986923Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6987161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.6987261Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.6987491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.6987574Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.6987577Z 2025-08-14T21:40:16.6987672Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6987890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6987963Z return mod(**inputs) 2025-08-14T21:40:16.6988202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6988275Z outputs = self.model( 2025-08-14T21:40:16.6988545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6988616Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6988857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6988923Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6989128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6989208Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6989438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.6989559Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6989563Z 2025-08-14T21:40:16.6989657Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6989842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6989911Z return mod(**inputs) 2025-08-14T21:40:16.6990143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6990213Z outputs = self.model( 2025-08-14T21:40:16.6990442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6990510Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6990747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6990816Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6991020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6991098Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6991325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.6991458Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.6991654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.6991718Z return self.act(input) 2025-08-14T21:40:16.6991722Z 2025-08-14T21:40:16.6991823Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6992009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6992076Z return mod(**inputs) 2025-08-14T21:40:16.6992313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6992394Z outputs = self.model( 2025-08-14T21:40:16.6992643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6992716Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6992960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6993037Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6993250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6993331Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6993597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:16.6993677Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.6993682Z 2025-08-14T21:40:16.6993785Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6993974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6994039Z return mod(**inputs) 2025-08-14T21:40:16.6994289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6994356Z outputs = self.model( 2025-08-14T21:40:16.6994608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6994687Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6994918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6994995Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6995200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6995283Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6995520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6995614Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6995860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.6996006Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.6996009Z 2025-08-14T21:40:16.6996107Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6996305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6996371Z return mod(**inputs) 2025-08-14T21:40:16.6996619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6996686Z outputs = self.model( 2025-08-14T21:40:16.6996922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6997001Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6997241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6997330Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6997550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6997625Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.6997868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.6997963Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.6998217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.6998303Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.6998306Z 2025-08-14T21:40:16.6998402Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.6998597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.6998662Z return mod(**inputs) 2025-08-14T21:40:16.6998894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.6998966Z outputs = self.model( 2025-08-14T21:40:16.6999200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.6999304Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.6999552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.6999623Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.6999839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.6999912Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7000149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.7000267Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.7000502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.7000591Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.7000595Z 2025-08-14T21:40:16.7000674Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7000749Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7000830Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7000900Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7000997Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7001193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7001258Z return mod(**inputs) 2025-08-14T21:40:16.7001496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7001569Z outputs = self.model( 2025-08-14T21:40:16.7001804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7001883Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7002121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7002190Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7002408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7002481Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7002725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.7002818Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.7003069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.7003170Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.7003450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.7003578Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.7003590Z 2025-08-14T21:40:16.7003687Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7003901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7003972Z return mod(**inputs) 2025-08-14T21:40:16.7004208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7004273Z outputs = self.model( 2025-08-14T21:40:16.7004515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7004585Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7004836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7004901Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7005136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7005219Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7005459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.7005647Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.7005931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.7006038Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.7006364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.7006481Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.7006486Z 2025-08-14T21:40:16.7006593Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7006816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7006889Z return mod(**inputs) 2025-08-14T21:40:16.7007163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7007236Z outputs = self.model( 2025-08-14T21:40:16.7007522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7007604Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7007839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7007910Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7008135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7008210Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7008451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.7008545Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.7008771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.7008856Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.7008862Z 2025-08-14T21:40:16.7008987Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7009180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7009243Z return mod(**inputs) 2025-08-14T21:40:16.7009473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7009544Z outputs = self.model( 2025-08-14T21:40:16.7009776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7009863Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7010103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7010172Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7010386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7010460Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7010690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7010798Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7011031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.7011239Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.7011253Z 2025-08-14T21:40:16.7011351Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7011537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7011608Z return mod(**inputs) 2025-08-14T21:40:16.7011841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7011907Z outputs = self.model( 2025-08-14T21:40:16.7012146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7012215Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7012455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7012526Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7012732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7012815Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7013046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7013145Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7013380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.7013454Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.7013458Z 2025-08-14T21:40:16.7013558Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7013745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7013806Z return mod(**inputs) 2025-08-14T21:40:16.7014057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7014126Z outputs = self.model( 2025-08-14T21:40:16.7014371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7014455Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7014700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7014796Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7015014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7015090Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7015343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7015450Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7015705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.7015809Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.7015813Z 2025-08-14T21:40:16.7015902Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7015986Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7016060Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7016134Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7016242Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7016441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7016511Z return mod(**inputs) 2025-08-14T21:40:16.7016745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7016837Z outputs = self.model( 2025-08-14T21:40:16.7017076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7017145Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7017374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7017443Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7017645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7017723Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7017948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7018045Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7018279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.7018368Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.7018639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.7018766Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.7018770Z 2025-08-14T21:40:16.7018861Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7019050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7019111Z return mod(**inputs) 2025-08-14T21:40:16.7019339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7019407Z outputs = self.model( 2025-08-14T21:40:16.7019650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7019725Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7019967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7020047Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7020262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7020336Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7020591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7020699Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7020940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.7021040Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.7021334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.7021451Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.7021455Z 2025-08-14T21:40:16.7021556Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7021741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7021810Z return mod(**inputs) 2025-08-14T21:40:16.7022056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7022124Z outputs = self.model( 2025-08-14T21:40:16.7022373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7022446Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7022716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7022791Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7023004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7023085Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7023325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7023430Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7023675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.7023756Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.7023759Z 2025-08-14T21:40:16.7023866Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7024059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7024141Z return mod(**inputs) 2025-08-14T21:40:16.7024387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7024455Z outputs = self.model( 2025-08-14T21:40:16.7024704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7024776Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7025024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7025094Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7025311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7025394Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7025637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.7025756Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.7025767Z 2025-08-14T21:40:16.7025866Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7026058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7026129Z return mod(**inputs) 2025-08-14T21:40:16.7026370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7026457Z outputs = self.model( 2025-08-14T21:40:16.7026714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7026785Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7027043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7027113Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7027347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7027431Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7027673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.7027788Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.7028003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.7028072Z return self.act(input) 2025-08-14T21:40:16.7028076Z 2025-08-14T21:40:16.7028183Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7028379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7028489Z return mod(**inputs) 2025-08-14T21:40:16.7028746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7028815Z outputs = self.model( 2025-08-14T21:40:16.7029060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7029139Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7029383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7029462Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7029678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7029755Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7030007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:16.7030086Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.7030091Z 2025-08-14T21:40:16.7030197Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7030394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7030458Z return mod(**inputs) 2025-08-14T21:40:16.7030710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7030778Z outputs = self.model( 2025-08-14T21:40:16.7031022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7031101Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7031345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7031425Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7031642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7031720Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7031969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.7032066Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.7032313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.7032480Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.7032484Z 2025-08-14T21:40:16.7032583Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7032785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7032849Z return mod(**inputs) 2025-08-14T21:40:16.7033099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7033192Z outputs = self.model( 2025-08-14T21:40:16.7033435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7033514Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7033757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7033830Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7034053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7034132Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7034380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.7034511Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.7034739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.7034823Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.7034827Z 2025-08-14T21:40:16.7034921Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7035104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7035177Z return mod(**inputs) 2025-08-14T21:40:16.7035406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7035477Z outputs = self.model( 2025-08-14T21:40:16.7035711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7035779Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7036012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7036079Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7036281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7036358Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7036578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.7036675Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.7036903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.7036984Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.7036987Z 2025-08-14T21:40:16.7037068Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7037144Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7037219Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7037309Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7037402Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7037588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7037835Z return mod(**inputs) 2025-08-14T21:40:16.7038070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7038192Z outputs = self.model( 2025-08-14T21:40:16.7038423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7038492Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7038731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7038799Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7039011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7039110Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7039342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.7039440Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.7039678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.7039776Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.7040051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.7040178Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.7040227Z 2025-08-14T21:40:16.7040331Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7040518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7040581Z return mod(**inputs) 2025-08-14T21:40:16.7040821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7040884Z outputs = self.model( 2025-08-14T21:40:16.7041122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7041194Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7041426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7041500Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7041705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7041786Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7042017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.7042107Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.7042348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.7042440Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.7042712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.7042819Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.7042823Z 2025-08-14T21:40:16.7042917Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7043109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7043170Z return mod(**inputs) 2025-08-14T21:40:16.7043402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7043472Z outputs = self.model( 2025-08-14T21:40:16.7043707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7043783Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7044030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7044097Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7044307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7044379Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7044609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.7044727Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.7044957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.7045039Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.7045042Z 2025-08-14T21:40:16.7045136Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7045327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7045399Z return mod(**inputs) 2025-08-14T21:40:16.7045700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7045773Z outputs = self.model( 2025-08-14T21:40:16.7046075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7046155Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7046444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7046521Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7046756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7046840Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7047086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7047209Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7047448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.7047595Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.7047599Z 2025-08-14T21:40:16.7047706Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7047907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7047969Z return mod(**inputs) 2025-08-14T21:40:16.7048216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7048281Z outputs = self.model( 2025-08-14T21:40:16.7048524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7048596Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7048831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7048907Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7049120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7049202Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7049434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7049535Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7049785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.7049877Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.7049880Z 2025-08-14T21:40:16.7049977Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7050169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7050233Z return mod(**inputs) 2025-08-14T21:40:16.7050470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7050534Z outputs = self.model( 2025-08-14T21:40:16.7050774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7050850Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7051075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7051141Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7051348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7051417Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7051644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7051743Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7052003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.7052092Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.7052096Z 2025-08-14T21:40:16.7052169Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7052246Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7052317Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7052389Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7052490Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7052671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7052732Z return mod(**inputs) 2025-08-14T21:40:16.7052966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7053028Z outputs = self.model( 2025-08-14T21:40:16.7053261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7053331Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7053555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7053627Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7053831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7053905Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7054143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7054242Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7054480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.7054573Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.7054842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.7054975Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.7054978Z 2025-08-14T21:40:16.7055073Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7055265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7055345Z return mod(**inputs) 2025-08-14T21:40:16.7055592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7055663Z outputs = self.model( 2025-08-14T21:40:16.7055890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7055960Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7056193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7056275Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7056490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7056562Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7056794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7056903Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7057137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.7057226Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.7057541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.7057643Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.7057647Z 2025-08-14T21:40:16.7057747Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7057932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7057993Z return mod(**inputs) 2025-08-14T21:40:16.7058230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7058294Z outputs = self.model( 2025-08-14T21:40:16.7058529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7058598Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7058829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7058905Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7059109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7059182Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7059413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7059509Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7059742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.7059817Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.7059821Z 2025-08-14T21:40:16.7059913Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7060100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7060163Z return mod(**inputs) 2025-08-14T21:40:16.7060402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7060466Z outputs = self.model( 2025-08-14T21:40:16.7060694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7060768Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7060999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7061089Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7061300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7061372Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7061604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.7061714Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.7061733Z 2025-08-14T21:40:16.7061829Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7062022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7062084Z return mod(**inputs) 2025-08-14T21:40:16.7062317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7062388Z outputs = self.model( 2025-08-14T21:40:16.7062621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7062696Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7062928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7063024Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7063241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7063316Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7063552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.7063663Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.7063864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.7063936Z return self.act(input) 2025-08-14T21:40:16.7063940Z 2025-08-14T21:40:16.7064035Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7064219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7064290Z return mod(**inputs) 2025-08-14T21:40:16.7064524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7064596Z outputs = self.model( 2025-08-14T21:40:16.7064835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7064904Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7065147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7065218Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7065430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7065511Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7065745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:16.7065846Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.7065850Z 2025-08-14T21:40:16.7065944Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7066132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7066201Z return mod(**inputs) 2025-08-14T21:40:16.7066431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7066501Z outputs = self.model( 2025-08-14T21:40:16.7066751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7066821Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7067055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7067122Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7067329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7067427Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7067657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.7067756Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.7067983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.7068124Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.7068128Z 2025-08-14T21:40:16.7068232Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7068415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7068487Z return mod(**inputs) 2025-08-14T21:40:16.7068745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7068813Z outputs = self.model( 2025-08-14T21:40:16.7069052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7069121Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7069352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7069428Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7069635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7069716Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7069945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.7070038Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.7070275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.7070353Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.7070358Z 2025-08-14T21:40:16.7070460Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7070644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7070705Z return mod(**inputs) 2025-08-14T21:40:16.7070953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7071016Z outputs = self.model( 2025-08-14T21:40:16.7071239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7071314Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7071539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7071614Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7071811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7071880Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7072110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.7072220Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.7072452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.7072540Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.7072544Z 2025-08-14T21:40:16.7072616Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7072706Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7072779Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7072849Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7072974Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7073158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7073219Z return mod(**inputs) 2025-08-14T21:40:16.7073459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7073524Z outputs = self.model( 2025-08-14T21:40:16.7073764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7073835Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7074072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7074179Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7074391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7074466Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7074708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.7074799Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.7075037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.7075130Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.7075414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.7075543Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.7075547Z 2025-08-14T21:40:16.7075643Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7075832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7075896Z return mod(**inputs) 2025-08-14T21:40:16.7076126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7076198Z outputs = self.model( 2025-08-14T21:40:16.7076426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7076498Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7076732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7076798Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7077015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7077088Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7077308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.7077407Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.7077632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.7077729Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.7078023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.7078122Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.7078125Z 2025-08-14T21:40:16.7078227Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7078416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7078477Z return mod(**inputs) 2025-08-14T21:40:16.7078726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7078806Z outputs = self.model( 2025-08-14T21:40:16.7079036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7079103Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7079326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7079401Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7079598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7079675Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7079931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.7080021Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.7080252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.7080327Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.7080330Z 2025-08-14T21:40:16.7080422Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7080608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7080671Z return mod(**inputs) 2025-08-14T21:40:16.7080903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7080965Z outputs = self.model( 2025-08-14T21:40:16.7081190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7081264Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7081488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7081555Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7081768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7081839Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7082079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7082179Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7082407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.7082555Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.7082562Z 2025-08-14T21:40:16.7082656Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7082850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7082913Z return mod(**inputs) 2025-08-14T21:40:16.7083146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7083217Z outputs = self.model( 2025-08-14T21:40:16.7083449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7083536Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7083776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7083846Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7084063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7084138Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7084394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7084506Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7084742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.7084828Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.7084831Z 2025-08-14T21:40:16.7084928Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7085116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7085187Z return mod(**inputs) 2025-08-14T21:40:16.7085458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7085586Z outputs = self.model( 2025-08-14T21:40:16.7085850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7085927Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7086179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7086250Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7086464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7086551Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7086793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7086897Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7087155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.7087239Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.7087243Z 2025-08-14T21:40:16.7087327Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7087404Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7087478Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7087559Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7087658Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7087855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7087926Z return mod(**inputs) 2025-08-14T21:40:16.7088154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7088228Z outputs = self.model( 2025-08-14T21:40:16.7088472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7088546Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7088798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7088868Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7089084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7089168Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7089435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7089545Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7089785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.7089883Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.7090178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.7090324Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.7090328Z 2025-08-14T21:40:16.7090437Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7090633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7090700Z return mod(**inputs) 2025-08-14T21:40:16.7090954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7091021Z outputs = self.model( 2025-08-14T21:40:16.7091266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7091345Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7091627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7091709Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7091931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7092008Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7092259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7092366Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7092612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.7092706Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.7092994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.7093108Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.7093113Z 2025-08-14T21:40:16.7093213Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7093414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7093480Z return mod(**inputs) 2025-08-14T21:40:16.7093724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7093800Z outputs = self.model( 2025-08-14T21:40:16.7094046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7094119Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7094373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7094446Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7094674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7094754Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7094996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7095105Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7095347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.7095457Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.7095467Z 2025-08-14T21:40:16.7095568Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7095764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7095835Z return mod(**inputs) 2025-08-14T21:40:16.7096084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7096170Z outputs = self.model( 2025-08-14T21:40:16.7096421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7096494Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7096747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7096819Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7097036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7097120Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7097371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.7097517Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.7097522Z 2025-08-14T21:40:16.7097629Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7097817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7097886Z return mod(**inputs) 2025-08-14T21:40:16.7098118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7098185Z outputs = self.model( 2025-08-14T21:40:16.7098422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7098489Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7098720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7098795Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7099003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7099087Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7099318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.7099429Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.7099634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.7099700Z return self.act(input) 2025-08-14T21:40:16.7099703Z 2025-08-14T21:40:16.7099801Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7099987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7100050Z return mod(**inputs) 2025-08-14T21:40:16.7100289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7100357Z outputs = self.model( 2025-08-14T21:40:16.7100587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7100663Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7100894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7100988Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7101194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7101266Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7101504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:16.7101582Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.7101588Z 2025-08-14T21:40:16.7101683Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7101939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7102001Z return mod(**inputs) 2025-08-14T21:40:16.7102239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7102303Z outputs = self.model( 2025-08-14T21:40:16.7102534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7102612Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7102843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7102916Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7103149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7103223Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7103460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.7103552Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.7103778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.7103933Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.7103936Z 2025-08-14T21:40:16.7104032Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7104225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7104288Z return mod(**inputs) 2025-08-14T21:40:16.7104520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7104594Z outputs = self.model( 2025-08-14T21:40:16.7104824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7104903Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7105133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7105201Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7105415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7105487Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7105716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.7105815Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.7106046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.7106130Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.7106134Z 2025-08-14T21:40:16.7106226Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7106410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7106480Z return mod(**inputs) 2025-08-14T21:40:16.7106707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7106791Z outputs = self.model( 2025-08-14T21:40:16.7107030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7107099Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7107334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7107402Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7107626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7107707Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7107937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.7108036Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.7108266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.7108346Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.7108350Z 2025-08-14T21:40:16.7108431Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7108505Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7108609Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7108692Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7108786Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7108979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7109042Z return mod(**inputs) 2025-08-14T21:40:16.7109273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7109344Z outputs = self.model( 2025-08-14T21:40:16.7109576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7109645Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7109880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7109948Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7110159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7110233Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7110462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.7110559Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.7110787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.7110879Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.7111157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.7111284Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.7111287Z 2025-08-14T21:40:16.7111392Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7111582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7111646Z return mod(**inputs) 2025-08-14T21:40:16.7111895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7111962Z outputs = self.model( 2025-08-14T21:40:16.7112209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7112305Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7112547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7112626Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7112840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7112921Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7113170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.7113293Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.7113528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.7113617Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.7113889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.7113996Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.7114000Z 2025-08-14T21:40:16.7114093Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7114288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7114394Z return mod(**inputs) 2025-08-14T21:40:16.7114639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7114716Z outputs = self.model( 2025-08-14T21:40:16.7114958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7115030Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7115277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7115351Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7115570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7115645Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7115887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 413, in forward 2025-08-14T21:40:16.7115988Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:40:16.7116230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.7116310Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.7116321Z 2025-08-14T21:40:16.7116420Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7116613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7116687Z return mod(**inputs) 2025-08-14T21:40:16.7116928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7116994Z outputs = self.model( 2025-08-14T21:40:16.7117244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7117318Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7117568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7117640Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7117855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7117939Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7118182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7118310Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7118565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 216, in forward 2025-08-14T21:40:16.7118713Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:40:16.7118717Z 2025-08-14T21:40:16.7118826Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7119026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7119109Z return mod(**inputs) 2025-08-14T21:40:16.7119363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7119432Z outputs = self.model( 2025-08-14T21:40:16.7119684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7119759Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7120004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7120082Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7120301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7120412Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7120664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7120769Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7121016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 235, in forward 2025-08-14T21:40:16.7121094Z key_states = self.k_proj(current_states) 2025-08-14T21:40:16.7121100Z 2025-08-14T21:40:16.7121202Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7121401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7121466Z return mod(**inputs) 2025-08-14T21:40:16.7121719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7121787Z outputs = self.model( 2025-08-14T21:40:16.7122033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7122113Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7122356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7122425Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7122646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7122725Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7122976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7123080Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7123324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 236, in forward 2025-08-14T21:40:16.7123414Z value_states = self.v_proj(current_states) 2025-08-14T21:40:16.7123420Z 2025-08-14T21:40:16.7123496Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7123574Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7123658Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7123733Z cudagraph partition due to non gpu ops 2025-08-14T21:40:16.7123838Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7124032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7124116Z return mod(**inputs) 2025-08-14T21:40:16.7124371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7124438Z outputs = self.model( 2025-08-14T21:40:16.7124688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7124769Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7125018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7125112Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7125330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7125407Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7125728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7125840Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7126084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.7126193Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.7126538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:40:16.7126686Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:16.7126690Z 2025-08-14T21:40:16.7126805Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7127000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7127075Z return mod(**inputs) 2025-08-14T21:40:16.7127320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7127393Z outputs = self.model( 2025-08-14T21:40:16.7127636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7127709Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7127964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7128035Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7128253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7128337Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7128578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7128692Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7128936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 254, in forward 2025-08-14T21:40:16.7129040Z attn_output, attn_weights = attention_interface( 2025-08-14T21:40:16.7129326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:40:16.7129428Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:40:16.7129432Z 2025-08-14T21:40:16.7129535Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7129724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7129787Z return mod(**inputs) 2025-08-14T21:40:16.7130030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7130155Z outputs = self.model( 2025-08-14T21:40:16.7130395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7130473Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7130714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7130788Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7131006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7131098Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7131340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 430, in forward 2025-08-14T21:40:16.7131441Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:40:16.7131685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 268, in forward 2025-08-14T21:40:16.7131764Z attn_output = self.out_proj(attn_output) 2025-08-14T21:40:16.7131768Z 2025-08-14T21:40:16.7131866Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7132067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7132130Z return mod(**inputs) 2025-08-14T21:40:16.7132402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7132479Z outputs = self.model( 2025-08-14T21:40:16.7132715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7132794Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7133029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7133100Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7133316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7133389Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7133623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.7133747Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.7133750Z 2025-08-14T21:40:16.7133847Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7134043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7134106Z return mod(**inputs) 2025-08-14T21:40:16.7134340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7134413Z outputs = self.model( 2025-08-14T21:40:16.7134648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7134725Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7134959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7135026Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7135244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7135319Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7135554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 445, in forward 2025-08-14T21:40:16.7135673Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:40:16.7135874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:16.7135971Z return self.act(input) 2025-08-14T21:40:16.7135975Z 2025-08-14T21:40:16.7136073Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7136268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7136341Z return mod(**inputs) 2025-08-14T21:40:16.7136584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1471, in forward 2025-08-14T21:40:16.7136655Z outputs = self.model( 2025-08-14T21:40:16.7136897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1288, in forward 2025-08-14T21:40:16.7136996Z decoder_outputs = self.decoder( 2025-08-14T21:40:16.7137239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1115, in forward 2025-08-14T21:40:16.7137308Z layer_outputs = decoder_layer( 2025-08-14T21:40:16.7137519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:16.7137817Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:16.7138070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 447, in forward 2025-08-14T21:40:16.7138158Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:40:16.7138162Z 2025-08-14T21:40:16.7138335Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7138529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7138602Z return mod(**inputs) 2025-08-14T21:40:16.7138837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1490, in forward 2025-08-14T21:40:16.7138914Z lm_logits = self.lm_head(outputs[0]) 2025-08-14T21:40:16.7138925Z 2025-08-14T21:40:16.7139020Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:16.7139210Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:16.7139281Z return mod(**inputs) 2025-08-14T21:40:16.7139517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bart/modeling_bart.py", line 1497, in forward 2025-08-14T21:40:16.7139680Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:40:16.7139686Z 2025-08-14T21:40:28.7553319Z Compilation time (from dynamo_timed): 25.934785759 2025-08-14T21:40:28.7641955Z pass 2025-08-14T21:40:28.7642416Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:40:28.7643261Z TIMING: _recursive_pre_grad_passes:0.01347 _recursive_joint_graph_passes:1.14142 _recursive_post_grad_passes:0.18016 async_compile.wait:0.83285 code_gen:10.38446 inductor_compile:13.34272 backend_compile:20.38888 gc:0.00087 entire_frame_compile:25.93479 total_wall_time:25.93479 2025-08-14T21:40:28.7644280Z STATS: call_* op count: 980 | FakeTensorMode.__torch_dispatch__:33505 | FakeTensor.__torch_dispatch__:11921 | ProxyTorchDispatchMode.__torch_dispatch__:12370 2025-08-14T21:40:28.7644823Z Dynamo produced 1 graphs covering 980 ops with 0 graph breaks (0 unique) 2025-08-14T21:40:34.3120252Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:40:34.3121247Z from pkg_resources import resource_filename 2025-08-14T21:40:34.9223769Z 2025-08-14T21:40:36.3386225Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:40:36.3386671Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:40:36.3399965Z cpu eval BertForMaskedLM 2025-08-14T21:40:36.8730568Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:40:37.1207262Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:40:37.3603434Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:40:44.8604847Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.8605189Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.8605627Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.8605864Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.8606489Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.8606719Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.8606931Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.8607164Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.8607383Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.8607602Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.8607819Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.8608030Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.8608283Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8608686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8609055Z return mod(**inputs) 2025-08-14T21:40:44.8609640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8610045Z outputs = self.bert( 2025-08-14T21:40:44.8610434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8610866Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8611280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8611709Z layer_outputs = layer_module( 2025-08-14T21:40:44.8612088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8617734Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8618365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8621318Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8621955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8626470Z return func(*args, **kwargs) 2025-08-14T21:40:44.8629796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8636429Z self_outputs = self.self( 2025-08-14T21:40:44.8638689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8639192Z return func(*args, **kwargs) 2025-08-14T21:40:44.8639618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:40:44.8640245Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:40:44.8640557Z 2025-08-14T21:40:44.8640679Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8641099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8641486Z return mod(**inputs) 2025-08-14T21:40:44.8641900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8642342Z outputs = self.bert( 2025-08-14T21:40:44.8642749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8643389Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8643808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8644243Z layer_outputs = layer_module( 2025-08-14T21:40:44.8644622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8645035Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8645665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8646188Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8646609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8647021Z return func(*args, **kwargs) 2025-08-14T21:40:44.8647426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8647849Z self_outputs = self.self( 2025-08-14T21:40:44.8648248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8648693Z return func(*args, **kwargs) 2025-08-14T21:40:44.8649215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:40:44.8649650Z self.key(current_states) 2025-08-14T21:40:44.8649788Z 2025-08-14T21:40:44.8649906Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8650301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8650648Z return mod(**inputs) 2025-08-14T21:40:44.8651037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8651446Z outputs = self.bert( 2025-08-14T21:40:44.8651832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8652277Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8652673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8653080Z layer_outputs = layer_module( 2025-08-14T21:40:44.8653460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8653853Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8654261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8654686Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8655104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8655505Z return func(*args, **kwargs) 2025-08-14T21:40:44.8655863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8656238Z self_outputs = self.self( 2025-08-14T21:40:44.8656595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8656968Z return func(*args, **kwargs) 2025-08-14T21:40:44.8657350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:40:44.8657747Z self.value(current_states) 2025-08-14T21:40:44.8657872Z 2025-08-14T21:40:44.8657966Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.8658209Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8658620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8658983Z return mod(**inputs) 2025-08-14T21:40:44.8659342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8659736Z outputs = self.bert( 2025-08-14T21:40:44.8660093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8660482Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8660858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8661257Z layer_outputs = layer_module( 2025-08-14T21:40:44.8661597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8661945Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8662322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8662711Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8663085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8663445Z return func(*args, **kwargs) 2025-08-14T21:40:44.8663845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8664232Z self_outputs = self.self( 2025-08-14T21:40:44.8664588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8664963Z return func(*args, **kwargs) 2025-08-14T21:40:44.8665347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:40:44.8665797Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:44.8665982Z 2025-08-14T21:40:44.8666086Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8666447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8666768Z return mod(**inputs) 2025-08-14T21:40:44.8667119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8667526Z outputs = self.bert( 2025-08-14T21:40:44.8667901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8668306Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8668691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8669106Z layer_outputs = layer_module( 2025-08-14T21:40:44.8669476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8669859Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8670256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8670686Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8671095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8671467Z return func(*args, **kwargs) 2025-08-14T21:40:44.8671842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:40:44.8672301Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:40:44.8672757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:40:44.8673161Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.8673336Z 2025-08-14T21:40:44.8673448Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8673830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8674170Z return mod(**inputs) 2025-08-14T21:40:44.8674539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8674941Z outputs = self.bert( 2025-08-14T21:40:44.8675315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8675733Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8676125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8676521Z layer_outputs = layer_module( 2025-08-14T21:40:44.8676884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8677257Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8677659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.8678073Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.8678542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.8678956Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.8679388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.8679868Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.8680307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:40:44.8680724Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.8680875Z 2025-08-14T21:40:44.8680983Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8681364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8681699Z return mod(**inputs) 2025-08-14T21:40:44.8682388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8682790Z outputs = self.bert( 2025-08-14T21:40:44.8683162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8683565Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8683958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8684356Z layer_outputs = layer_module( 2025-08-14T21:40:44.8684726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8685119Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8685701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.8686161Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.8686593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.8687017Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.8687451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.8687922Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.8688362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:40:44.8688856Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:40:44.8689240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:44.8689576Z return self.act(input) 2025-08-14T21:40:44.8689703Z 2025-08-14T21:40:44.8689811Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8690192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8690533Z return mod(**inputs) 2025-08-14T21:40:44.8690951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8691335Z outputs = self.bert( 2025-08-14T21:40:44.8691698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8692092Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8692498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8692908Z layer_outputs = layer_module( 2025-08-14T21:40:44.8693289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8693647Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8694076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.8694464Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.8694936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.8695332Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.8695735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:40:44.8696196Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:40:44.8696624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:40:44.8697013Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.8697155Z 2025-08-14T21:40:44.8697259Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8697616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8697933Z return mod(**inputs) 2025-08-14T21:40:44.8698285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8698660Z outputs = self.bert( 2025-08-14T21:40:44.8699005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8699397Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8699766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8700142Z layer_outputs = layer_module( 2025-08-14T21:40:44.8700477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8700838Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8701222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8701600Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8701981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8702348Z return func(*args, **kwargs) 2025-08-14T21:40:44.8702713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8703103Z self_outputs = self.self( 2025-08-14T21:40:44.8703462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8703832Z return func(*args, **kwargs) 2025-08-14T21:40:44.8704200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:40:44.8704715Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:40:44.8705004Z 2025-08-14T21:40:44.8705108Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8705469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8705782Z return mod(**inputs) 2025-08-14T21:40:44.8706141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8706518Z outputs = self.bert( 2025-08-14T21:40:44.8706870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8707245Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8707619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8708030Z layer_outputs = layer_module( 2025-08-14T21:40:44.8708370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8708736Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8709122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8709550Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8709942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8710339Z return func(*args, **kwargs) 2025-08-14T21:40:44.8710724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8711145Z self_outputs = self.self( 2025-08-14T21:40:44.8711520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8711916Z return func(*args, **kwargs) 2025-08-14T21:40:44.8712305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:40:44.8712717Z self.key(current_states) 2025-08-14T21:40:44.8712845Z 2025-08-14T21:40:44.8712954Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8713335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8713679Z return mod(**inputs) 2025-08-14T21:40:44.8714070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8714488Z outputs = self.bert( 2025-08-14T21:40:44.8714880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8715308Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8715728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8716151Z layer_outputs = layer_module( 2025-08-14T21:40:44.8716514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8716901Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8717330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8717785Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8718197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8718612Z return func(*args, **kwargs) 2025-08-14T21:40:44.8719026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8719445Z self_outputs = self.self( 2025-08-14T21:40:44.8719839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8720271Z return func(*args, **kwargs) 2025-08-14T21:40:44.8720688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:40:44.8721123Z self.value(current_states) 2025-08-14T21:40:44.8721248Z 2025-08-14T21:40:44.8721338Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.8721590Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8721978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8722339Z return mod(**inputs) 2025-08-14T21:40:44.8722721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8723168Z outputs = self.bert( 2025-08-14T21:40:44.8723558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8723972Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8724376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8724798Z layer_outputs = layer_module( 2025-08-14T21:40:44.8725153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8725637Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8726063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8726509Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8726901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8727294Z return func(*args, **kwargs) 2025-08-14T21:40:44.8727682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8728070Z self_outputs = self.self( 2025-08-14T21:40:44.8728445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8728835Z return func(*args, **kwargs) 2025-08-14T21:40:44.8729215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:40:44.8729664Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:44.8729863Z 2025-08-14T21:40:44.8729974Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8730350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8730695Z return mod(**inputs) 2025-08-14T21:40:44.8731064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8731460Z outputs = self.bert( 2025-08-14T21:40:44.8731832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8732225Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8732629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8733060Z layer_outputs = layer_module( 2025-08-14T21:40:44.8733420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8733794Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8734193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8734581Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8734977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8735356Z return func(*args, **kwargs) 2025-08-14T21:40:44.8735711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:40:44.8736131Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:40:44.8736545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:40:44.8736927Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.8737060Z 2025-08-14T21:40:44.8737171Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8737557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8738066Z return mod(**inputs) 2025-08-14T21:40:44.8738422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8738829Z outputs = self.bert( 2025-08-14T21:40:44.8739170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8739540Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8739902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8740270Z layer_outputs = layer_module( 2025-08-14T21:40:44.8740592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8740939Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8741311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.8741683Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.8742068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.8742449Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.8742844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.8743280Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.8743691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:40:44.8744067Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.8744202Z 2025-08-14T21:40:44.8744314Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8744666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8744998Z return mod(**inputs) 2025-08-14T21:40:44.8745342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8745695Z outputs = self.bert( 2025-08-14T21:40:44.8746037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8746406Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8746836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8747198Z layer_outputs = layer_module( 2025-08-14T21:40:44.8747534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8747882Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8748248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.8748650Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.8749043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.8749425Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.8749815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.8750261Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.8750681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:40:44.8751073Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:40:44.8751475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:44.8751806Z return self.act(input) 2025-08-14T21:40:44.8751912Z 2025-08-14T21:40:44.8752022Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8752355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8752660Z return mod(**inputs) 2025-08-14T21:40:44.8753002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8753356Z outputs = self.bert( 2025-08-14T21:40:44.8753685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8754050Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8754417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8754787Z layer_outputs = layer_module( 2025-08-14T21:40:44.8755133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8755495Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8755877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.8756258Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.8756656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.8757049Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.8757451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:40:44.8757919Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:40:44.8758356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:40:44.8758744Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.8758882Z 2025-08-14T21:40:44.8758987Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8759342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8759667Z return mod(**inputs) 2025-08-14T21:40:44.8760018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8760410Z outputs = self.bert( 2025-08-14T21:40:44.8760764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8761151Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8761516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8761898Z layer_outputs = layer_module( 2025-08-14T21:40:44.8762242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8762619Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8763001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8763394Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8763784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8764162Z return func(*args, **kwargs) 2025-08-14T21:40:44.8764531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8764909Z self_outputs = self.self( 2025-08-14T21:40:44.8765383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8765781Z return func(*args, **kwargs) 2025-08-14T21:40:44.8766167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:40:44.8766701Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:40:44.8766970Z 2025-08-14T21:40:44.8767086Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8767456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8767802Z return mod(**inputs) 2025-08-14T21:40:44.8768199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8768594Z outputs = self.bert( 2025-08-14T21:40:44.8768971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8769374Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8769770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8770162Z layer_outputs = layer_module( 2025-08-14T21:40:44.8770524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8770907Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8771305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8771712Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8772115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8772527Z return func(*args, **kwargs) 2025-08-14T21:40:44.8772908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8773302Z self_outputs = self.self( 2025-08-14T21:40:44.8773676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8774083Z return func(*args, **kwargs) 2025-08-14T21:40:44.8774456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:40:44.8774869Z self.key(current_states) 2025-08-14T21:40:44.8775015Z 2025-08-14T21:40:44.8775129Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8775499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8775838Z return mod(**inputs) 2025-08-14T21:40:44.8776205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8776570Z outputs = self.bert( 2025-08-14T21:40:44.8776904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8777301Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8777665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8778029Z layer_outputs = layer_module( 2025-08-14T21:40:44.8778370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8778723Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8779090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8779462Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8779857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8780225Z return func(*args, **kwargs) 2025-08-14T21:40:44.8780576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8780943Z self_outputs = self.self( 2025-08-14T21:40:44.8781282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8781638Z return func(*args, **kwargs) 2025-08-14T21:40:44.8781986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:40:44.8782350Z self.value(current_states) 2025-08-14T21:40:44.8782466Z 2025-08-14T21:40:44.8782555Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.8782790Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8783150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8783462Z return mod(**inputs) 2025-08-14T21:40:44.8783803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8784162Z outputs = self.bert( 2025-08-14T21:40:44.8784502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8784875Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8785236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8785597Z layer_outputs = layer_module( 2025-08-14T21:40:44.8785930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8786287Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8786665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8787057Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8787436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8787806Z return func(*args, **kwargs) 2025-08-14T21:40:44.8788170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8788566Z self_outputs = self.self( 2025-08-14T21:40:44.8788917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8789269Z return func(*args, **kwargs) 2025-08-14T21:40:44.8789626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:40:44.8790051Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:44.8790229Z 2025-08-14T21:40:44.8790339Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8790715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8791046Z return mod(**inputs) 2025-08-14T21:40:44.8791395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8791770Z outputs = self.bert( 2025-08-14T21:40:44.8792117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8792498Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8792874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8793239Z layer_outputs = layer_module( 2025-08-14T21:40:44.8793628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8793989Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8794367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8794745Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8795122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8795488Z return func(*args, **kwargs) 2025-08-14T21:40:44.8795846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:40:44.8796277Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:40:44.8796698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:40:44.8797087Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.8797226Z 2025-08-14T21:40:44.8797329Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8797688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8798009Z return mod(**inputs) 2025-08-14T21:40:44.8798364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8798736Z outputs = self.bert( 2025-08-14T21:40:44.8799090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8799466Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8799829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8800204Z layer_outputs = layer_module( 2025-08-14T21:40:44.8800551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8800905Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8801277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.8801666Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.8802062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.8802483Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.8802889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.8803339Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.8803764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:40:44.8804148Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.8804294Z 2025-08-14T21:40:44.8804418Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8804787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8805125Z return mod(**inputs) 2025-08-14T21:40:44.8805583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8806009Z outputs = self.bert( 2025-08-14T21:40:44.8806386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8806813Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8807189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8807572Z layer_outputs = layer_module( 2025-08-14T21:40:44.8808016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8808365Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8808734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.8809116Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.8809500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.8809891Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.8810290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.8810737Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.8811153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:40:44.8811564Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:40:44.8811944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:44.8812283Z return self.act(input) 2025-08-14T21:40:44.8812395Z 2025-08-14T21:40:44.8812497Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8812852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8813174Z return mod(**inputs) 2025-08-14T21:40:44.8813521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8813896Z outputs = self.bert( 2025-08-14T21:40:44.8814246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8814622Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8814984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8815358Z layer_outputs = layer_module( 2025-08-14T21:40:44.8815697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8816045Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8816415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.8816820Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.8817204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.8817576Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.8817970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:40:44.8818418Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:40:44.8818864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:40:44.8819245Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.8819385Z 2025-08-14T21:40:44.8819486Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8819833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8820142Z return mod(**inputs) 2025-08-14T21:40:44.8820491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8820859Z outputs = self.bert( 2025-08-14T21:40:44.8821206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8821622Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8822001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8822383Z layer_outputs = layer_module( 2025-08-14T21:40:44.8822720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8823079Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8823459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8823879Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8824274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8824674Z return func(*args, **kwargs) 2025-08-14T21:40:44.8825072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8825451Z self_outputs = self.self( 2025-08-14T21:40:44.8825803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8826170Z return func(*args, **kwargs) 2025-08-14T21:40:44.8826537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:40:44.8827041Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:40:44.8827308Z 2025-08-14T21:40:44.8827409Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8827766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8828085Z return mod(**inputs) 2025-08-14T21:40:44.8828433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8828810Z outputs = self.bert( 2025-08-14T21:40:44.8829163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8829536Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8829902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8830280Z layer_outputs = layer_module( 2025-08-14T21:40:44.8830677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8831029Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8831412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8831806Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8832190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8832577Z return func(*args, **kwargs) 2025-08-14T21:40:44.8832944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8833322Z self_outputs = self.self( 2025-08-14T21:40:44.8833671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8834044Z return func(*args, **kwargs) 2025-08-14T21:40:44.8834407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:40:44.8834787Z self.key(current_states) 2025-08-14T21:40:44.8834900Z 2025-08-14T21:40:44.8835003Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8835392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8835712Z return mod(**inputs) 2025-08-14T21:40:44.8836059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8836432Z outputs = self.bert( 2025-08-14T21:40:44.8836789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8837172Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8837543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8838058Z layer_outputs = layer_module( 2025-08-14T21:40:44.8838406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8838757Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8839139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8839529Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8839907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8840271Z return func(*args, **kwargs) 2025-08-14T21:40:44.8840639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8841017Z self_outputs = self.self( 2025-08-14T21:40:44.8841373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8841739Z return func(*args, **kwargs) 2025-08-14T21:40:44.8842104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:40:44.8842477Z self.value(current_states) 2025-08-14T21:40:44.8842592Z 2025-08-14T21:40:44.8842677Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.8842916Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8843274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8843591Z return mod(**inputs) 2025-08-14T21:40:44.8843937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8844310Z outputs = self.bert( 2025-08-14T21:40:44.8844711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8845089Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8845564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8845996Z layer_outputs = layer_module( 2025-08-14T21:40:44.8846373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8846741Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8847159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8847550Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8847964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8848377Z return func(*args, **kwargs) 2025-08-14T21:40:44.8848780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8849188Z self_outputs = self.self( 2025-08-14T21:40:44.8849574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8849975Z return func(*args, **kwargs) 2025-08-14T21:40:44.8850430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:40:44.8850902Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:44.8851112Z 2025-08-14T21:40:44.8851222Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8851611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8851988Z return mod(**inputs) 2025-08-14T21:40:44.8852368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8852784Z outputs = self.bert( 2025-08-14T21:40:44.8853164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8853587Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8853990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8854399Z layer_outputs = layer_module( 2025-08-14T21:40:44.8854730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8855069Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8855437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8855815Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8856181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8856535Z return func(*args, **kwargs) 2025-08-14T21:40:44.8856888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:40:44.8857306Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:40:44.8857714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:40:44.8858092Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.8858236Z 2025-08-14T21:40:44.8858338Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8858686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8859018Z return mod(**inputs) 2025-08-14T21:40:44.8859366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8859733Z outputs = self.bert( 2025-08-14T21:40:44.8860075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8860442Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8860817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8861217Z layer_outputs = layer_module( 2025-08-14T21:40:44.8861539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8861882Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8862251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.8862629Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.8863009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.8863393Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.8863785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.8864254Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.8864669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:40:44.8865054Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.8865185Z 2025-08-14T21:40:44.8865290Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8865630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8865950Z return mod(**inputs) 2025-08-14T21:40:44.8866295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8866658Z outputs = self.bert( 2025-08-14T21:40:44.8866992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8867365Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8867731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8868086Z layer_outputs = layer_module( 2025-08-14T21:40:44.8868410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8868746Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8869105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.8869468Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.8869840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.8870209Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.8870585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.8871007Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.8871415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:40:44.8871807Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:40:44.8872160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:44.8872503Z return self.act(input) 2025-08-14T21:40:44.8872608Z 2025-08-14T21:40:44.8872712Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8873050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8873450Z return mod(**inputs) 2025-08-14T21:40:44.8873792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8874157Z outputs = self.bert( 2025-08-14T21:40:44.8874486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8874923Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8875293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8875666Z layer_outputs = layer_module( 2025-08-14T21:40:44.8876003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8876368Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8876735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.8877096Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.8877513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.8877890Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.8878280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:40:44.8878713Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:40:44.8879127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:40:44.8879501Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.8879634Z 2025-08-14T21:40:44.8879740Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8880078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8880391Z return mod(**inputs) 2025-08-14T21:40:44.8880738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8881089Z outputs = self.bert( 2025-08-14T21:40:44.8881434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8881810Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8882168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8882525Z layer_outputs = layer_module( 2025-08-14T21:40:44.8882861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8883218Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8883587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8883969Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8884347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8884716Z return func(*args, **kwargs) 2025-08-14T21:40:44.8885070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8885512Z self_outputs = self.self( 2025-08-14T21:40:44.8885905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8886329Z return func(*args, **kwargs) 2025-08-14T21:40:44.8886702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:40:44.8887220Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:40:44.8887474Z 2025-08-14T21:40:44.8887580Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8887923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8888259Z return mod(**inputs) 2025-08-14T21:40:44.8888610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8889044Z outputs = self.bert( 2025-08-14T21:40:44.8889381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8889760Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8890128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8890492Z layer_outputs = layer_module( 2025-08-14T21:40:44.8890828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8891178Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8891589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8891969Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8892338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8892700Z return func(*args, **kwargs) 2025-08-14T21:40:44.8893055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8893413Z self_outputs = self.self( 2025-08-14T21:40:44.8893759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8894117Z return func(*args, **kwargs) 2025-08-14T21:40:44.8894462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:40:44.8894829Z self.key(current_states) 2025-08-14T21:40:44.8894948Z 2025-08-14T21:40:44.8895050Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8895398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8895697Z return mod(**inputs) 2025-08-14T21:40:44.8896037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8896394Z outputs = self.bert( 2025-08-14T21:40:44.8896727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8897101Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8897461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8897827Z layer_outputs = layer_module( 2025-08-14T21:40:44.8898154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8898501Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8898868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8899237Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8899601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8899977Z return func(*args, **kwargs) 2025-08-14T21:40:44.8900333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8900695Z self_outputs = self.self( 2025-08-14T21:40:44.8901055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8901409Z return func(*args, **kwargs) 2025-08-14T21:40:44.8901758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:40:44.8902134Z self.value(current_states) 2025-08-14T21:40:44.8902252Z 2025-08-14T21:40:44.8902331Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.8902557Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8902884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8903189Z return mod(**inputs) 2025-08-14T21:40:44.8903524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8903934Z outputs = self.bert( 2025-08-14T21:40:44.8904271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8904652Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8905092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8905454Z layer_outputs = layer_module( 2025-08-14T21:40:44.8905786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8906136Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8906501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8906881Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8907240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8907592Z return func(*args, **kwargs) 2025-08-14T21:40:44.8907933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8908292Z self_outputs = self.self( 2025-08-14T21:40:44.8908632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8908989Z return func(*args, **kwargs) 2025-08-14T21:40:44.8909331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:40:44.8909761Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:44.8909937Z 2025-08-14T21:40:44.8910057Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8910395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8910696Z return mod(**inputs) 2025-08-14T21:40:44.8911030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8911390Z outputs = self.bert( 2025-08-14T21:40:44.8911719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8912086Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8912446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8912823Z layer_outputs = layer_module( 2025-08-14T21:40:44.8913153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8913537Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8913944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8914359Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8914767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8915183Z return func(*args, **kwargs) 2025-08-14T21:40:44.8915583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:40:44.8916066Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:40:44.8916538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:40:44.8916958Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.8917109Z 2025-08-14T21:40:44.8917235Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8917604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8917952Z return mod(**inputs) 2025-08-14T21:40:44.8918339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8918745Z outputs = self.bert( 2025-08-14T21:40:44.8919164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8919583Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8919982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8920388Z layer_outputs = layer_module( 2025-08-14T21:40:44.8920754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8921143Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8921543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.8921965Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.8922399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.8922824Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.8923259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.8923751Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.8924208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:40:44.8924629Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.8924781Z 2025-08-14T21:40:44.8924891Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8925277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8925716Z return mod(**inputs) 2025-08-14T21:40:44.8926104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8926525Z outputs = self.bert( 2025-08-14T21:40:44.8926914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8927335Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8927735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8928152Z layer_outputs = layer_module( 2025-08-14T21:40:44.8928532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8928949Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8929364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.8929799Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.8930237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.8930655Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.8931127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.8931620Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.8932079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:40:44.8932527Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:40:44.8932939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:44.8933312Z return self.act(input) 2025-08-14T21:40:44.8933433Z 2025-08-14T21:40:44.8933544Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8933969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8934323Z return mod(**inputs) 2025-08-14T21:40:44.8934714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8935127Z outputs = self.bert( 2025-08-14T21:40:44.8935523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8935945Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8936342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8936748Z layer_outputs = layer_module( 2025-08-14T21:40:44.8937116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8937509Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8938071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.8938487Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.8938908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.8939324Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.8939745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:40:44.8940231Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:40:44.8940690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:40:44.8941091Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.8941245Z 2025-08-14T21:40:44.8941355Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8941736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8942078Z return mod(**inputs) 2025-08-14T21:40:44.8942443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8942837Z outputs = self.bert( 2025-08-14T21:40:44.8943208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8943612Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8944049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8944447Z layer_outputs = layer_module( 2025-08-14T21:40:44.8944808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8945180Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8945544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8945942Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8946297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8946652Z return func(*args, **kwargs) 2025-08-14T21:40:44.8947011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8947380Z self_outputs = self.self( 2025-08-14T21:40:44.8947725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8948080Z return func(*args, **kwargs) 2025-08-14T21:40:44.8948429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:40:44.8948959Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:40:44.8949212Z 2025-08-14T21:40:44.8949308Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8949657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8949968Z return mod(**inputs) 2025-08-14T21:40:44.8950306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8950674Z outputs = self.bert( 2025-08-14T21:40:44.8951018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8951393Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8951753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8952118Z layer_outputs = layer_module( 2025-08-14T21:40:44.8952456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8952808Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8953179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8953550Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8953918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8954273Z return func(*args, **kwargs) 2025-08-14T21:40:44.8954631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8955004Z self_outputs = self.self( 2025-08-14T21:40:44.8955342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8955687Z return func(*args, **kwargs) 2025-08-14T21:40:44.8956038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:40:44.8956399Z self.key(current_states) 2025-08-14T21:40:44.8956506Z 2025-08-14T21:40:44.8956603Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8956946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8957289Z return mod(**inputs) 2025-08-14T21:40:44.8957624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8957969Z outputs = self.bert( 2025-08-14T21:40:44.8958302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8958665Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8959008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8959386Z layer_outputs = layer_module( 2025-08-14T21:40:44.8959708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8960050Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8960404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8960774Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8961132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8961492Z return func(*args, **kwargs) 2025-08-14T21:40:44.8961840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8962237Z self_outputs = self.self( 2025-08-14T21:40:44.8962588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8962942Z return func(*args, **kwargs) 2025-08-14T21:40:44.8963297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:40:44.8963668Z self.value(current_states) 2025-08-14T21:40:44.8963784Z 2025-08-14T21:40:44.8963873Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.8964103Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8964457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8964802Z return mod(**inputs) 2025-08-14T21:40:44.8965147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8965586Z outputs = self.bert( 2025-08-14T21:40:44.8965997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8966414Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8966801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8967202Z layer_outputs = layer_module( 2025-08-14T21:40:44.8967557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8967920Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8968292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8968673Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8969042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8969400Z return func(*args, **kwargs) 2025-08-14T21:40:44.8969755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.8970124Z self_outputs = self.self( 2025-08-14T21:40:44.8970464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8970823Z return func(*args, **kwargs) 2025-08-14T21:40:44.8971175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:40:44.8971622Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:44.8971801Z 2025-08-14T21:40:44.8971904Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8972250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8972572Z return mod(**inputs) 2025-08-14T21:40:44.8972926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8973332Z outputs = self.bert( 2025-08-14T21:40:44.8973732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8974116Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8974478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8974853Z layer_outputs = layer_module( 2025-08-14T21:40:44.8975192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8975547Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8975917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.8976333Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.8976700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.8977056Z return func(*args, **kwargs) 2025-08-14T21:40:44.8977410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:40:44.8977842Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:40:44.8978271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:40:44.8978644Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.8978788Z 2025-08-14T21:40:44.8978887Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8979231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8979546Z return mod(**inputs) 2025-08-14T21:40:44.8979886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8980254Z outputs = self.bert( 2025-08-14T21:40:44.8980594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8980959Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8981318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8981683Z layer_outputs = layer_module( 2025-08-14T21:40:44.8982010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8982349Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8982715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.8983090Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.8983473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.8983862Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.8984255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.8984699Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.8985140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:40:44.8985529Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.8985667Z 2025-08-14T21:40:44.8985787Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8986132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8986441Z return mod(**inputs) 2025-08-14T21:40:44.8986788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8987170Z outputs = self.bert( 2025-08-14T21:40:44.8987507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8987881Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8988247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8988615Z layer_outputs = layer_module( 2025-08-14T21:40:44.8988943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8989291Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8989690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.8990063Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.8990454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.8990836Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.8991229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.8991667Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.8992074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:40:44.8992475Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:40:44.8992837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:44.8993166Z return self.act(input) 2025-08-14T21:40:44.8993276Z 2025-08-14T21:40:44.8993373Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.8993717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.8994015Z return mod(**inputs) 2025-08-14T21:40:44.8994354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.8994713Z outputs = self.bert( 2025-08-14T21:40:44.8995058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.8995422Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.8995786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.8996156Z layer_outputs = layer_module( 2025-08-14T21:40:44.8996494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.8996834Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.8997200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.8997572Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.8997943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.8999390Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.8999797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:40:44.9000329Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:40:44.9000755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:40:44.9001138Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.9001274Z 2025-08-14T21:40:44.9001413Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9001762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9002091Z return mod(**inputs) 2025-08-14T21:40:44.9002450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9002834Z outputs = self.bert( 2025-08-14T21:40:44.9003182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9003574Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9003952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9004350Z layer_outputs = layer_module( 2025-08-14T21:40:44.9004763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9005147Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9005611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9006015Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9006424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9006816Z return func(*args, **kwargs) 2025-08-14T21:40:44.9007179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9007543Z self_outputs = self.self( 2025-08-14T21:40:44.9007953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9008326Z return func(*args, **kwargs) 2025-08-14T21:40:44.9008673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:40:44.9009178Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:40:44.9009449Z 2025-08-14T21:40:44.9009552Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9009908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9010221Z return mod(**inputs) 2025-08-14T21:40:44.9010573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9010945Z outputs = self.bert( 2025-08-14T21:40:44.9011287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9011673Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9012041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9012417Z layer_outputs = layer_module( 2025-08-14T21:40:44.9012750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9013109Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9013488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9013898Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9014269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9014348Z return func(*args, **kwargs) 2025-08-14T21:40:44.9014592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9014664Z self_outputs = self.self( 2025-08-14T21:40:44.9014927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9014996Z return func(*args, **kwargs) 2025-08-14T21:40:44.9015250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:40:44.9015320Z self.key(current_states) 2025-08-14T21:40:44.9015325Z 2025-08-14T21:40:44.9015428Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9015635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9015701Z return mod(**inputs) 2025-08-14T21:40:44.9015946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9016019Z outputs = self.bert( 2025-08-14T21:40:44.9016326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9016412Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9016660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9016731Z layer_outputs = layer_module( 2025-08-14T21:40:44.9016957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9017037Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9017277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9017364Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9017605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9017681Z return func(*args, **kwargs) 2025-08-14T21:40:44.9017920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9017991Z self_outputs = self.self( 2025-08-14T21:40:44.9018234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9018304Z return func(*args, **kwargs) 2025-08-14T21:40:44.9018545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:40:44.9018626Z self.value(current_states) 2025-08-14T21:40:44.9018630Z 2025-08-14T21:40:44.9018713Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.9018828Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9019028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9019096Z return mod(**inputs) 2025-08-14T21:40:44.9019348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9019416Z outputs = self.bert( 2025-08-14T21:40:44.9019666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9019740Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9019982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9020079Z layer_outputs = layer_module( 2025-08-14T21:40:44.9020299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9020377Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9020624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9020704Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9020995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9021063Z return func(*args, **kwargs) 2025-08-14T21:40:44.9021304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9021383Z self_outputs = self.self( 2025-08-14T21:40:44.9021621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9021689Z return func(*args, **kwargs) 2025-08-14T21:40:44.9021936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:40:44.9022070Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:44.9022073Z 2025-08-14T21:40:44.9022214Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9022414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9022482Z return mod(**inputs) 2025-08-14T21:40:44.9022734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9022798Z outputs = self.bert( 2025-08-14T21:40:44.9023050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9023125Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9023365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9023445Z layer_outputs = layer_module( 2025-08-14T21:40:44.9023665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9023744Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9023998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9024077Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9024321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9024389Z return func(*args, **kwargs) 2025-08-14T21:40:44.9024630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:40:44.9024763Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:40:44.9025005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:40:44.9025088Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.9025101Z 2025-08-14T21:40:44.9025202Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9025400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9025473Z return mod(**inputs) 2025-08-14T21:40:44.9025717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9025782Z outputs = self.bert( 2025-08-14T21:40:44.9026034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9026127Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9026375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9026455Z layer_outputs = layer_module( 2025-08-14T21:40:44.9026662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9026743Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9026987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.9027070Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.9027319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.9027394Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.9027664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.9027777Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.9028007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:40:44.9028123Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.9028127Z 2025-08-14T21:40:44.9028225Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9028424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9028486Z return mod(**inputs) 2025-08-14T21:40:44.9028720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9028792Z outputs = self.bert( 2025-08-14T21:40:44.9029024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9029094Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9029332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9029401Z layer_outputs = layer_module( 2025-08-14T21:40:44.9029617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9029691Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9029923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.9030008Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.9030250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.9030325Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.9030592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.9030702Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.9030945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:40:44.9031055Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:40:44.9031257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:44.9031335Z return self.act(input) 2025-08-14T21:40:44.9031338Z 2025-08-14T21:40:44.9031439Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9031640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9031735Z return mod(**inputs) 2025-08-14T21:40:44.9031973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9032044Z outputs = self.bert( 2025-08-14T21:40:44.9032282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9032355Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9032596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9032681Z layer_outputs = layer_module( 2025-08-14T21:40:44.9032899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9032972Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9033205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.9033302Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.9033542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.9033624Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.9033887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:40:44.9034053Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:40:44.9034299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:40:44.9034379Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.9034382Z 2025-08-14T21:40:44.9034480Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9034679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9034744Z return mod(**inputs) 2025-08-14T21:40:44.9034992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9035057Z outputs = self.bert( 2025-08-14T21:40:44.9035294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9035376Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9035611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9035690Z layer_outputs = layer_module( 2025-08-14T21:40:44.9035901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9035977Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9036219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9036313Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9036538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9036610Z return func(*args, **kwargs) 2025-08-14T21:40:44.9036840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9036913Z self_outputs = self.self( 2025-08-14T21:40:44.9037139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9037207Z return func(*args, **kwargs) 2025-08-14T21:40:44.9037445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:40:44.9037789Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:40:44.9037835Z 2025-08-14T21:40:44.9037949Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9038149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9038215Z return mod(**inputs) 2025-08-14T21:40:44.9038472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9038538Z outputs = self.bert( 2025-08-14T21:40:44.9038795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9038910Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9039166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9039248Z layer_outputs = layer_module( 2025-08-14T21:40:44.9039474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9039558Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9039820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9039907Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9040212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9040295Z return func(*args, **kwargs) 2025-08-14T21:40:44.9040554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9040631Z self_outputs = self.self( 2025-08-14T21:40:44.9040865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9040933Z return func(*args, **kwargs) 2025-08-14T21:40:44.9041184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:40:44.9041254Z self.key(current_states) 2025-08-14T21:40:44.9041257Z 2025-08-14T21:40:44.9041359Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9041562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9041631Z return mod(**inputs) 2025-08-14T21:40:44.9041891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9041963Z outputs = self.bert( 2025-08-14T21:40:44.9042221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9042309Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9042570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9042654Z layer_outputs = layer_module( 2025-08-14T21:40:44.9042882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9042962Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9043229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9043315Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9043568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9043648Z return func(*args, **kwargs) 2025-08-14T21:40:44.9043906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9043985Z self_outputs = self.self( 2025-08-14T21:40:44.9044262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9044335Z return func(*args, **kwargs) 2025-08-14T21:40:44.9044602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:40:44.9044676Z self.value(current_states) 2025-08-14T21:40:44.9044681Z 2025-08-14T21:40:44.9044769Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.9044884Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9045110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9045188Z return mod(**inputs) 2025-08-14T21:40:44.9045526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9045601Z outputs = self.bert( 2025-08-14T21:40:44.9045869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9045953Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9046216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9046300Z layer_outputs = layer_module( 2025-08-14T21:40:44.9046586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9046673Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9046919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9046999Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9047245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9047318Z return func(*args, **kwargs) 2025-08-14T21:40:44.9047573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9047643Z self_outputs = self.self( 2025-08-14T21:40:44.9047877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9047957Z return func(*args, **kwargs) 2025-08-14T21:40:44.9048200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:40:44.9048334Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:44.9048338Z 2025-08-14T21:40:44.9048450Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9048646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9048720Z return mod(**inputs) 2025-08-14T21:40:44.9048970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9049034Z outputs = self.bert( 2025-08-14T21:40:44.9049286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9049359Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9049602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9049683Z layer_outputs = layer_module( 2025-08-14T21:40:44.9049903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9049987Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9050227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9050305Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9050570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9050640Z return func(*args, **kwargs) 2025-08-14T21:40:44.9050891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:40:44.9051021Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:40:44.9051262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:40:44.9051373Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.9051376Z 2025-08-14T21:40:44.9051476Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9051673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9051746Z return mod(**inputs) 2025-08-14T21:40:44.9051993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9052065Z outputs = self.bert( 2025-08-14T21:40:44.9052309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9052380Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9052657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9052729Z layer_outputs = layer_module( 2025-08-14T21:40:44.9052954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9053033Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9053274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.9053365Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.9053618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.9053693Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.9053972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.9054091Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.9054338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:40:44.9054419Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.9054423Z 2025-08-14T21:40:44.9054521Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9054723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9054788Z return mod(**inputs) 2025-08-14T21:40:44.9055038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9055109Z outputs = self.bert( 2025-08-14T21:40:44.9055345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9055423Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9055660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9055730Z layer_outputs = layer_module( 2025-08-14T21:40:44.9055955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9056029Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9056263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.9056360Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.9056602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.9056681Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.9056941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.9057055Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.9057317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:40:44.9057423Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:40:44.9057631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:44.9057697Z return self.act(input) 2025-08-14T21:40:44.9057702Z 2025-08-14T21:40:44.9057799Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9057995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9058057Z return mod(**inputs) 2025-08-14T21:40:44.9058296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9058389Z outputs = self.bert( 2025-08-14T21:40:44.9058622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9058700Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9058929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9058997Z layer_outputs = layer_module( 2025-08-14T21:40:44.9059210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9059287Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9059524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.9059601Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.9059845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.9059923Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.9060183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:40:44.9060309Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:40:44.9060549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:40:44.9060625Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.9060630Z 2025-08-14T21:40:44.9060733Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9060917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9060977Z return mod(**inputs) 2025-08-14T21:40:44.9061224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9061288Z outputs = self.bert( 2025-08-14T21:40:44.9061522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9061593Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9061819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9061893Z layer_outputs = layer_module( 2025-08-14T21:40:44.9062099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9062192Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9062429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9062505Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9062739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9062807Z return func(*args, **kwargs) 2025-08-14T21:40:44.9063051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9063125Z self_outputs = self.self( 2025-08-14T21:40:44.9063346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9063412Z return func(*args, **kwargs) 2025-08-14T21:40:44.9063647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:40:44.9063840Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:40:44.9063843Z 2025-08-14T21:40:44.9063944Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9064309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9064378Z return mod(**inputs) 2025-08-14T21:40:44.9064631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9064696Z outputs = self.bert( 2025-08-14T21:40:44.9064943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9065014Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9065251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9065328Z layer_outputs = layer_module( 2025-08-14T21:40:44.9065540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9065618Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9065872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9065951Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9066181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9066246Z return func(*args, **kwargs) 2025-08-14T21:40:44.9066474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9066550Z self_outputs = self.self( 2025-08-14T21:40:44.9066776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9066849Z return func(*args, **kwargs) 2025-08-14T21:40:44.9067079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:40:44.9067148Z self.key(current_states) 2025-08-14T21:40:44.9067154Z 2025-08-14T21:40:44.9067260Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9067446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9067506Z return mod(**inputs) 2025-08-14T21:40:44.9067749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9067812Z outputs = self.bert( 2025-08-14T21:40:44.9068051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9068138Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9068369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9068444Z layer_outputs = layer_module( 2025-08-14T21:40:44.9068657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9068731Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9068986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9069061Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9069293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9069358Z return func(*args, **kwargs) 2025-08-14T21:40:44.9069588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9069660Z self_outputs = self.self( 2025-08-14T21:40:44.9069885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9069955Z return func(*args, **kwargs) 2025-08-14T21:40:44.9070216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:40:44.9070288Z self.value(current_states) 2025-08-14T21:40:44.9070291Z 2025-08-14T21:40:44.9070374Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.9070473Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9070663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9070733Z return mod(**inputs) 2025-08-14T21:40:44.9070973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9071043Z outputs = self.bert( 2025-08-14T21:40:44.9071283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9071355Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9071603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9071672Z layer_outputs = layer_module( 2025-08-14T21:40:44.9071885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9071967Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9072264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9072350Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9072574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9072640Z return func(*args, **kwargs) 2025-08-14T21:40:44.9072877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9072943Z self_outputs = self.self( 2025-08-14T21:40:44.9073173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9073246Z return func(*args, **kwargs) 2025-08-14T21:40:44.9073474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:40:44.9073608Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:44.9073611Z 2025-08-14T21:40:44.9073709Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9073922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9073994Z return mod(**inputs) 2025-08-14T21:40:44.9074238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9074307Z outputs = self.bert( 2025-08-14T21:40:44.9074553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9074640Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9074893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9074962Z layer_outputs = layer_module( 2025-08-14T21:40:44.9075177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9075259Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9075504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9075586Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9075816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9075911Z return func(*args, **kwargs) 2025-08-14T21:40:44.9076147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:40:44.9076267Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:40:44.9076502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:40:44.9076582Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.9076585Z 2025-08-14T21:40:44.9076685Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9076883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9076946Z return mod(**inputs) 2025-08-14T21:40:44.9077184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9077255Z outputs = self.bert( 2025-08-14T21:40:44.9077494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9077574Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9077805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9077876Z layer_outputs = layer_module( 2025-08-14T21:40:44.9078090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9078168Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9078398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.9078490Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.9078736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.9078820Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.9079082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.9079196Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.9079437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:40:44.9079516Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.9079547Z 2025-08-14T21:40:44.9079655Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9079849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9079912Z return mod(**inputs) 2025-08-14T21:40:44.9080156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9080220Z outputs = self.bert( 2025-08-14T21:40:44.9080461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9080554Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9080789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9080863Z layer_outputs = layer_module( 2025-08-14T21:40:44.9081073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9081147Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9081392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.9081472Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.9081751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.9081831Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.9082093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.9082213Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.9082445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:40:44.9082551Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:40:44.9082759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:44.9082827Z return self.act(input) 2025-08-14T21:40:44.9082831Z 2025-08-14T21:40:44.9082932Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9083119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9083186Z return mod(**inputs) 2025-08-14T21:40:44.9083433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9083498Z outputs = self.bert( 2025-08-14T21:40:44.9083733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9083810Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9084048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9084126Z layer_outputs = layer_module( 2025-08-14T21:40:44.9084341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9084417Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9084667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.9084750Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.9085012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.9085088Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.9085438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:40:44.9085595Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:40:44.9085891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:40:44.9085978Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.9085990Z 2025-08-14T21:40:44.9086100Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9086320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9086401Z return mod(**inputs) 2025-08-14T21:40:44.9086688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9086760Z outputs = self.bert( 2025-08-14T21:40:44.9087034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9087115Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9087393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9087463Z layer_outputs = layer_module( 2025-08-14T21:40:44.9087673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9087756Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9088022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9088102Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9088348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9088416Z return func(*args, **kwargs) 2025-08-14T21:40:44.9088658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9088727Z self_outputs = self.self( 2025-08-14T21:40:44.9088956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9089031Z return func(*args, **kwargs) 2025-08-14T21:40:44.9089265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:40:44.9089467Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:40:44.9089477Z 2025-08-14T21:40:44.9089577Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9089773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9089842Z return mod(**inputs) 2025-08-14T21:40:44.9090084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9090150Z outputs = self.bert( 2025-08-14T21:40:44.9090400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9090470Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9090712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9090782Z layer_outputs = layer_module( 2025-08-14T21:40:44.9090993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9091078Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9091315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9091392Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9091633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9091719Z return func(*args, **kwargs) 2025-08-14T21:40:44.9091970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9092036Z self_outputs = self.self( 2025-08-14T21:40:44.9092274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9092352Z return func(*args, **kwargs) 2025-08-14T21:40:44.9092593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:40:44.9092682Z self.key(current_states) 2025-08-14T21:40:44.9092692Z 2025-08-14T21:40:44.9092789Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9092978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9093048Z return mod(**inputs) 2025-08-14T21:40:44.9093288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9093351Z outputs = self.bert( 2025-08-14T21:40:44.9093596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9093667Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9093982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9094055Z layer_outputs = layer_module( 2025-08-14T21:40:44.9094268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9094353Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9094591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9094672Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9094914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9094980Z return func(*args, **kwargs) 2025-08-14T21:40:44.9095226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9095296Z self_outputs = self.self( 2025-08-14T21:40:44.9095534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9095608Z return func(*args, **kwargs) 2025-08-14T21:40:44.9095855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:40:44.9095923Z self.value(current_states) 2025-08-14T21:40:44.9095935Z 2025-08-14T21:40:44.9096012Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.9096111Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9096310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9096372Z return mod(**inputs) 2025-08-14T21:40:44.9096608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9096678Z outputs = self.bert( 2025-08-14T21:40:44.9096918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9096991Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9097231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9097301Z layer_outputs = layer_module( 2025-08-14T21:40:44.9097515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9097611Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9097849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9097934Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9098162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9098238Z return func(*args, **kwargs) 2025-08-14T21:40:44.9098470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9098556Z self_outputs = self.self( 2025-08-14T21:40:44.9098797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9098863Z return func(*args, **kwargs) 2025-08-14T21:40:44.9099105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:40:44.9099238Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:44.9099241Z 2025-08-14T21:40:44.9099337Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9099527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9099589Z return mod(**inputs) 2025-08-14T21:40:44.9099862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9099935Z outputs = self.bert( 2025-08-14T21:40:44.9100171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9100242Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9100478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9100547Z layer_outputs = layer_module( 2025-08-14T21:40:44.9100761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9100835Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9101062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9101149Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9101375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9101448Z return func(*args, **kwargs) 2025-08-14T21:40:44.9101677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:40:44.9101795Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:40:44.9102033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:40:44.9102112Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.9102116Z 2025-08-14T21:40:44.9102210Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9102404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9102466Z return mod(**inputs) 2025-08-14T21:40:44.9102707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9102772Z outputs = self.bert( 2025-08-14T21:40:44.9103002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9103080Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9103310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9103404Z layer_outputs = layer_module( 2025-08-14T21:40:44.9103610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9103686Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9103926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.9104011Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.9104261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.9104365Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.9104636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.9104761Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.9105001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:40:44.9105083Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.9105086Z 2025-08-14T21:40:44.9105195Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9105392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9105495Z return mod(**inputs) 2025-08-14T21:40:44.9105736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9105802Z outputs = self.bert( 2025-08-14T21:40:44.9106050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9106124Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9106364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9106444Z layer_outputs = layer_module( 2025-08-14T21:40:44.9106657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9106739Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9106981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.9107064Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.9107332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.9107408Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.9107672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.9107795Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.9108031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:40:44.9108144Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:40:44.9108346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:44.9108415Z return self.act(input) 2025-08-14T21:40:44.9108421Z 2025-08-14T21:40:44.9108528Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9108717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9108787Z return mod(**inputs) 2025-08-14T21:40:44.9109025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9109088Z outputs = self.bert( 2025-08-14T21:40:44.9109330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9109428Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9109668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9109746Z layer_outputs = layer_module( 2025-08-14T21:40:44.9109959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9110041Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9110293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.9110374Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.9110626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.9110700Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.9110963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:40:44.9111097Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:40:44.9111332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:40:44.9111446Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.9111450Z 2025-08-14T21:40:44.9111551Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9111742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9111813Z return mod(**inputs) 2025-08-14T21:40:44.9112055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9112125Z outputs = self.bert( 2025-08-14T21:40:44.9112364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9112436Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9112677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9112746Z layer_outputs = layer_module( 2025-08-14T21:40:44.9112956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9113043Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9113276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9113362Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9113592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9113662Z return func(*args, **kwargs) 2025-08-14T21:40:44.9113900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9113967Z self_outputs = self.self( 2025-08-14T21:40:44.9114202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9114272Z return func(*args, **kwargs) 2025-08-14T21:40:44.9114508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:40:44.9114714Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:40:44.9114717Z 2025-08-14T21:40:44.9114817Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9115009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9115135Z return mod(**inputs) 2025-08-14T21:40:44.9115377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9115447Z outputs = self.bert( 2025-08-14T21:40:44.9115687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9115762Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9116003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9116091Z layer_outputs = layer_module( 2025-08-14T21:40:44.9116303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9116389Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9116627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9116716Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9116946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9117013Z return func(*args, **kwargs) 2025-08-14T21:40:44.9117286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9117357Z self_outputs = self.self( 2025-08-14T21:40:44.9117594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9117663Z return func(*args, **kwargs) 2025-08-14T21:40:44.9117895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:40:44.9117972Z self.key(current_states) 2025-08-14T21:40:44.9117976Z 2025-08-14T21:40:44.9118075Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9118266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9118339Z return mod(**inputs) 2025-08-14T21:40:44.9118580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9118661Z outputs = self.bert( 2025-08-14T21:40:44.9118900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9118971Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9119211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9119281Z layer_outputs = layer_module( 2025-08-14T21:40:44.9119487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9119569Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9119799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9119883Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9120109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9120177Z return func(*args, **kwargs) 2025-08-14T21:40:44.9120424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9120495Z self_outputs = self.self( 2025-08-14T21:40:44.9120734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9120802Z return func(*args, **kwargs) 2025-08-14T21:40:44.9121041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:40:44.9121143Z self.value(current_states) 2025-08-14T21:40:44.9121147Z 2025-08-14T21:40:44.9121227Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.9121328Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9121535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9121602Z return mod(**inputs) 2025-08-14T21:40:44.9121856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9121937Z outputs = self.bert( 2025-08-14T21:40:44.9122189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9122270Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9122520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9122592Z layer_outputs = layer_module( 2025-08-14T21:40:44.9122822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9122899Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9123154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9123272Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9123509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9123586Z return func(*args, **kwargs) 2025-08-14T21:40:44.9123825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9123894Z self_outputs = self.self( 2025-08-14T21:40:44.9124135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9124205Z return func(*args, **kwargs) 2025-08-14T21:40:44.9124450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:40:44.9124580Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:44.9124583Z 2025-08-14T21:40:44.9124687Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9124889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9124955Z return mod(**inputs) 2025-08-14T21:40:44.9125209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9125280Z outputs = self.bert( 2025-08-14T21:40:44.9125619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9125716Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9125979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9126056Z layer_outputs = layer_module( 2025-08-14T21:40:44.9126301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9126393Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9126661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9126748Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9126995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9127085Z return func(*args, **kwargs) 2025-08-14T21:40:44.9127325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:40:44.9127486Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:40:44.9127727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:40:44.9127809Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.9127813Z 2025-08-14T21:40:44.9127922Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9128120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9128205Z return mod(**inputs) 2025-08-14T21:40:44.9128459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9128525Z outputs = self.bert( 2025-08-14T21:40:44.9128775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9128849Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9129089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9129167Z layer_outputs = layer_module( 2025-08-14T21:40:44.9129414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9129494Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9129744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.9129829Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.9130095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.9130171Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.9130447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.9130575Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.9130817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:40:44.9130906Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.9130912Z 2025-08-14T21:40:44.9131016Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9131214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9131287Z return mod(**inputs) 2025-08-14T21:40:44.9131553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9131623Z outputs = self.bert( 2025-08-14T21:40:44.9131899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9131977Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9132240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9132314Z layer_outputs = layer_module( 2025-08-14T21:40:44.9132546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9132636Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9132894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.9132989Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.9133261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.9133377Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.9133658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.9133776Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.9134022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:40:44.9134144Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:40:44.9134353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:44.9134445Z return self.act(input) 2025-08-14T21:40:44.9134449Z 2025-08-14T21:40:44.9134552Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9134769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9134845Z return mod(**inputs) 2025-08-14T21:40:44.9135106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9135177Z outputs = self.bert( 2025-08-14T21:40:44.9135450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9135526Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9135822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9135900Z layer_outputs = layer_module( 2025-08-14T21:40:44.9136132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9136222Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9136479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.9136573Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.9136842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.9136919Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.9137209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:40:44.9137351Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:40:44.9137731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:40:44.9137842Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.9137847Z 2025-08-14T21:40:44.9137957Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9138175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9138248Z return mod(**inputs) 2025-08-14T21:40:44.9138523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9138602Z outputs = self.bert( 2025-08-14T21:40:44.9138874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9138958Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9139220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9139297Z layer_outputs = layer_module( 2025-08-14T21:40:44.9139537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9139618Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9139875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9140015Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9140267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9140348Z return func(*args, **kwargs) 2025-08-14T21:40:44.9140600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9140671Z self_outputs = self.self( 2025-08-14T21:40:44.9140915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9141012Z return func(*args, **kwargs) 2025-08-14T21:40:44.9141251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:40:44.9141461Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:40:44.9141466Z 2025-08-14T21:40:44.9141568Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9141771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9141836Z return mod(**inputs) 2025-08-14T21:40:44.9142081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9142199Z outputs = self.bert( 2025-08-14T21:40:44.9142446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9142527Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9142766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9142835Z layer_outputs = layer_module( 2025-08-14T21:40:44.9143056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9143134Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9143376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9143462Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9143699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9143774Z return func(*args, **kwargs) 2025-08-14T21:40:44.9144018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9144087Z self_outputs = self.self( 2025-08-14T21:40:44.9144330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9144399Z return func(*args, **kwargs) 2025-08-14T21:40:44.9144639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:40:44.9144717Z self.key(current_states) 2025-08-14T21:40:44.9144721Z 2025-08-14T21:40:44.9144821Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9145023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9145090Z return mod(**inputs) 2025-08-14T21:40:44.9145334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9145407Z outputs = self.bert( 2025-08-14T21:40:44.9145650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9145732Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9145972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9146063Z layer_outputs = layer_module( 2025-08-14T21:40:44.9146289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9146366Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9146605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9146700Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9146936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9147043Z return func(*args, **kwargs) 2025-08-14T21:40:44.9147286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9147702Z self_outputs = self.self( 2025-08-14T21:40:44.9148057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9148459Z return func(*args, **kwargs) 2025-08-14T21:40:44.9148831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:40:44.9149247Z self.value(current_states) 2025-08-14T21:40:44.9149366Z 2025-08-14T21:40:44.9149444Z cudagraph partition due to non gpu ops 2025-08-14T21:40:44.9149712Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9150080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9150387Z return mod(**inputs) 2025-08-14T21:40:44.9150733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9151093Z outputs = self.bert( 2025-08-14T21:40:44.9151437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9151804Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9152167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9152598Z layer_outputs = layer_module( 2025-08-14T21:40:44.9152929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9153278Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9153651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9154029Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9154401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9154774Z return func(*args, **kwargs) 2025-08-14T21:40:44.9155126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:40:44.9155486Z self_outputs = self.self( 2025-08-14T21:40:44.9155822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9156175Z return func(*args, **kwargs) 2025-08-14T21:40:44.9156527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:40:44.9156940Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:40:44.9157129Z 2025-08-14T21:40:44.9157229Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9157574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9157884Z return mod(**inputs) 2025-08-14T21:40:44.9158217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9158603Z outputs = self.bert( 2025-08-14T21:40:44.9158947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9159308Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9159675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9160044Z layer_outputs = layer_module( 2025-08-14T21:40:44.9160400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9160746Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9161123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:40:44.9161506Z self_attention_outputs = self.attention( 2025-08-14T21:40:44.9161874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:40:44.9162237Z return func(*args, **kwargs) 2025-08-14T21:40:44.9162598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:40:44.9163024Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:40:44.9163471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:40:44.9163862Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.9164005Z 2025-08-14T21:40:44.9164104Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9164452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9164761Z return mod(**inputs) 2025-08-14T21:40:44.9165110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9165674Z outputs = self.bert( 2025-08-14T21:40:44.9166051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9166479Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9166881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9167273Z layer_outputs = layer_module( 2025-08-14T21:40:44.9167622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9167972Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9168371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.9168797Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.9169234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.9169665Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.9170107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.9170595Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.9171056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:40:44.9171482Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.9171630Z 2025-08-14T21:40:44.9171750Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9172132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9172484Z return mod(**inputs) 2025-08-14T21:40:44.9172901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9173306Z outputs = self.bert( 2025-08-14T21:40:44.9173693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9174111Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9174484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9174846Z layer_outputs = layer_module( 2025-08-14T21:40:44.9175167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9175507Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9175855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.9176224Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.9176596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.9176969Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.9177343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:40:44.9177799Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:40:44.9178199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:40:44.9178591Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:40:44.9178941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:40:44.9179263Z return self.act(input) 2025-08-14T21:40:44.9179367Z 2025-08-14T21:40:44.9179475Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9179805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9180110Z return mod(**inputs) 2025-08-14T21:40:44.9180447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1308, in forward 2025-08-14T21:40:44.9180814Z outputs = self.bert( 2025-08-14T21:40:44.9181164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:40:44.9181523Z encoder_outputs = self.encoder( 2025-08-14T21:40:44.9181878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:40:44.9182237Z layer_outputs = layer_module( 2025-08-14T21:40:44.9182575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:40:44.9182942Z return super().__call__(*args, **kwargs) 2025-08-14T21:40:44.9183324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:40:44.9183702Z layer_output = apply_chunking_to_forward( 2025-08-14T21:40:44.9184096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:40:44.9184487Z return forward_fn(*input_tensors) 2025-08-14T21:40:44.9184896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:40:44.9185344Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:40:44.9185763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:40:44.9186139Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.9186293Z 2025-08-14T21:40:44.9186402Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9186742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9187058Z return mod(**inputs) 2025-08-14T21:40:44.9187412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1323, in forward 2025-08-14T21:40:44.9187813Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:40:44.9188215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 780, in forward 2025-08-14T21:40:44.9188657Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:40:44.9189078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 769, in forward 2025-08-14T21:40:44.9189476Z hidden_states = self.transform(hidden_states) 2025-08-14T21:40:44.9189882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 745, in forward 2025-08-14T21:40:44.9190277Z hidden_states = self.dense(hidden_states) 2025-08-14T21:40:44.9190415Z 2025-08-14T21:40:44.9190522Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9190884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9191259Z return mod(**inputs) 2025-08-14T21:40:44.9191621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1323, in forward 2025-08-14T21:40:44.9192020Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:40:44.9192417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 780, in forward 2025-08-14T21:40:44.9192852Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:40:44.9193285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 770, in forward 2025-08-14T21:40:44.9193691Z hidden_states = self.decoder(hidden_states) 2025-08-14T21:40:44.9193848Z 2025-08-14T21:40:44.9193955Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:40:44.9194328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:40:44.9194663Z return mod(**inputs) 2025-08-14T21:40:44.9195040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1328, in forward 2025-08-14T21:40:44.9195569Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:40:44.9195822Z 2025-08-14T21:40:53.4098442Z Compilation time (from dynamo_timed): 14.666243096 2025-08-14T21:40:53.4175582Z pass 2025-08-14T21:40:53.4176083Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:40:53.4180139Z TIMING: _recursive_pre_grad_passes:0.0076 _recursive_joint_graph_passes:0.62937 _recursive_post_grad_passes:0.07931 async_compile.wait:0.74051 code_gen:7.30344 inductor_compile:8.42885 backend_compile:11.56614 gc:0.00044 entire_frame_compile:14.66624 total_wall_time:14.66624 2025-08-14T21:40:53.4181077Z STATS: call_* op count: 289 | FakeTensorMode.__torch_dispatch__:12337 | FakeTensor.__torch_dispatch__:4686 | ProxyTorchDispatchMode.__torch_dispatch__:4495 2025-08-14T21:40:53.4181606Z Dynamo produced 1 graphs covering 289 ops with 0 graph breaks (0 unique) 2025-08-14T21:40:58.4671516Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:40:58.4677688Z from pkg_resources import resource_filename 2025-08-14T21:40:59.0391530Z 2025-08-14T21:41:00.2077849Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:41:00.2078280Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:41:00.2096340Z cpu eval BertForQuestionAnswering 2025-08-14T21:41:00.6553582Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:41:00.8523500Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:41:01.0811118Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:41:08.5846303Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.5846698Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.5846931Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.5847164Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.5847448Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.5847733Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.5848293Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.5848613Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.5848944Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.5850064Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.5850568Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.5864499Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.5864842Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.5865616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.5865999Z return mod(**inputs) 2025-08-14T21:41:08.5866429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.5866847Z outputs = self.bert( 2025-08-14T21:41:08.5867232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.5867659Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.5868062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.5868474Z layer_outputs = layer_module( 2025-08-14T21:41:08.5868846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.5869242Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.5869660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.5870093Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.5870513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.5870910Z return func(*args, **kwargs) 2025-08-14T21:41:08.5871296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.5871837Z self_outputs = self.self( 2025-08-14T21:41:08.5872410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.5873063Z return func(*args, **kwargs) 2025-08-14T21:41:08.5873542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:08.5874444Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:08.5874818Z 2025-08-14T21:41:08.5874987Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.5875486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.5875830Z return mod(**inputs) 2025-08-14T21:41:08.5876201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.5876663Z outputs = self.bert( 2025-08-14T21:41:08.5877036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.5877438Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.5877828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.5878231Z layer_outputs = layer_module( 2025-08-14T21:41:08.5878606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.5879041Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.5879456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.5879867Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.5880271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.5880691Z return func(*args, **kwargs) 2025-08-14T21:41:08.5881207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.5881622Z self_outputs = self.self( 2025-08-14T21:41:08.5882060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.5882456Z return func(*args, **kwargs) 2025-08-14T21:41:08.5882853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:08.5883266Z self.key(current_states) 2025-08-14T21:41:08.5883391Z 2025-08-14T21:41:08.5883505Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.5883901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.5884260Z return mod(**inputs) 2025-08-14T21:41:08.5884658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.5885267Z outputs = self.bert( 2025-08-14T21:41:08.5885661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.5886086Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.5886493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.5886907Z layer_outputs = layer_module( 2025-08-14T21:41:08.5887283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.5887673Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.5888081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.5888505Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.5888919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.5889336Z return func(*args, **kwargs) 2025-08-14T21:41:08.5889729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.5890136Z self_outputs = self.self( 2025-08-14T21:41:08.5890521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.5890922Z return func(*args, **kwargs) 2025-08-14T21:41:08.5891324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:08.5891736Z self.value(current_states) 2025-08-14T21:41:08.5891894Z 2025-08-14T21:41:08.5891993Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.5892251Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.5892642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.5892996Z return mod(**inputs) 2025-08-14T21:41:08.5893372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.5893778Z outputs = self.bert( 2025-08-14T21:41:08.5894159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.5894612Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.5895010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.5895431Z layer_outputs = layer_module( 2025-08-14T21:41:08.5895790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.5896161Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.5896560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.5896965Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.5897392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.5897776Z return func(*args, **kwargs) 2025-08-14T21:41:08.5898159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.5898550Z self_outputs = self.self( 2025-08-14T21:41:08.5898914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.5899301Z return func(*args, **kwargs) 2025-08-14T21:41:08.5899683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:08.5900142Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:08.5900338Z 2025-08-14T21:41:08.5900448Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.5900829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.5901165Z return mod(**inputs) 2025-08-14T21:41:08.5901538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.5901921Z outputs = self.bert( 2025-08-14T21:41:08.5902287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.5902689Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.5903071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.5903470Z layer_outputs = layer_module( 2025-08-14T21:41:08.5903830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.5904205Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.5904599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.5905003Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.5905397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.5905784Z return func(*args, **kwargs) 2025-08-14T21:41:08.5906141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:08.5906600Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:08.5907045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:08.5907426Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.5907573Z 2025-08-14T21:41:08.5907676Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.5908038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.5908370Z return mod(**inputs) 2025-08-14T21:41:08.5908745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.5909159Z outputs = self.bert( 2025-08-14T21:41:08.5909533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.5909930Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.5910323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.5910768Z layer_outputs = layer_module( 2025-08-14T21:41:08.5911108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.5911519Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.5911980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.5912376Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.5912772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.5913171Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.5913584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.5914049Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.5914471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:08.5914868Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.5915006Z 2025-08-14T21:41:08.5915118Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.5915478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.5915798Z return mod(**inputs) 2025-08-14T21:41:08.5916159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.5916536Z outputs = self.bert( 2025-08-14T21:41:08.5916879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.5917263Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.5917638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.5918019Z layer_outputs = layer_module( 2025-08-14T21:41:08.5918357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.5918716Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.5919102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.5919489Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.5919890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.5920283Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.5920711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.5921198Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.5921628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:08.5922042Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:08.5922423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:08.5922754Z return self.act(input) 2025-08-14T21:41:08.5922869Z 2025-08-14T21:41:08.5923021Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.5923378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.5923696Z return mod(**inputs) 2025-08-14T21:41:08.5924059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.5924454Z outputs = self.bert( 2025-08-14T21:41:08.5924824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.5925338Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.5925742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.5926149Z layer_outputs = layer_module( 2025-08-14T21:41:08.5926551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.5926935Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.5927336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.5927752Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.5928157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.5928552Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.5928958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:08.5929420Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:08.5929849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:08.5930238Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.5930374Z 2025-08-14T21:41:08.5930487Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.5930834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.5931162Z return mod(**inputs) 2025-08-14T21:41:08.5931524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.5931922Z outputs = self.bert( 2025-08-14T21:41:08.5932285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.5932686Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.5933065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.5933449Z layer_outputs = layer_module( 2025-08-14T21:41:08.5933784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.5934142Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.5934526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.5934923Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.5935321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.5935708Z return func(*args, **kwargs) 2025-08-14T21:41:08.5936070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.5936435Z self_outputs = self.self( 2025-08-14T21:41:08.5936797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.5937163Z return func(*args, **kwargs) 2025-08-14T21:41:08.5937534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:08.5938289Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:08.5938565Z 2025-08-14T21:41:08.5938671Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.5939036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.5939352Z return mod(**inputs) 2025-08-14T21:41:08.5939710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.5940084Z outputs = self.bert( 2025-08-14T21:41:08.5940521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.5940903Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.5941279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.5941658Z layer_outputs = layer_module( 2025-08-14T21:41:08.5941996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.5942355Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.5942739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.5943129Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.5943505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.5943876Z return func(*args, **kwargs) 2025-08-14T21:41:08.5944250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.5944622Z self_outputs = self.self( 2025-08-14T21:41:08.5944984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.5945355Z return func(*args, **kwargs) 2025-08-14T21:41:08.5945717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:08.5946087Z self.key(current_states) 2025-08-14T21:41:08.5946208Z 2025-08-14T21:41:08.5946599Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.5946958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.5947279Z return mod(**inputs) 2025-08-14T21:41:08.5947626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.5948004Z outputs = self.bert( 2025-08-14T21:41:08.5948360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.5948739Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.5949113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.5949492Z layer_outputs = layer_module( 2025-08-14T21:41:08.5949834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.5950216Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.5950592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.5950974Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.5951352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.5951711Z return func(*args, **kwargs) 2025-08-14T21:41:08.5952099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.5952454Z self_outputs = self.self( 2025-08-14T21:41:08.5952783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.5953133Z return func(*args, **kwargs) 2025-08-14T21:41:08.5953480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:08.5953842Z self.value(current_states) 2025-08-14T21:41:08.5953953Z 2025-08-14T21:41:08.5954031Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.5954256Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.5954627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.5954934Z return mod(**inputs) 2025-08-14T21:41:08.5955271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.5955624Z outputs = self.bert( 2025-08-14T21:41:08.5955953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.5956308Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.5956667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.5957035Z layer_outputs = layer_module( 2025-08-14T21:41:08.5957361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.5957708Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.5958085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.5958448Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.5958797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.5959146Z return func(*args, **kwargs) 2025-08-14T21:41:08.5959494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.5959854Z self_outputs = self.self( 2025-08-14T21:41:08.5960199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.5960555Z return func(*args, **kwargs) 2025-08-14T21:41:08.5960910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:08.5961330Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:08.5961514Z 2025-08-14T21:41:08.5961614Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.5961962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.5962279Z return mod(**inputs) 2025-08-14T21:41:08.5962619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.5962986Z outputs = self.bert( 2025-08-14T21:41:08.5963349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.5963717Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.5964080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.5964444Z layer_outputs = layer_module( 2025-08-14T21:41:08.5964790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.5965317Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.5965770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.5966254Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.5966611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.5966968Z return func(*args, **kwargs) 2025-08-14T21:41:08.5967318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:08.5967764Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:08.5968186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:08.5968607Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.5968747Z 2025-08-14T21:41:08.5968857Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.5969213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.5969534Z return mod(**inputs) 2025-08-14T21:41:08.5969880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.5970242Z outputs = self.bert( 2025-08-14T21:41:08.5970578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.5970947Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.5971311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.5971685Z layer_outputs = layer_module( 2025-08-14T21:41:08.5972028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.5972467Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.5972841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.5973216Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.5973607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.5973998Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.5974396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.5974837Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.5975259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:08.5975650Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.5975787Z 2025-08-14T21:41:08.5975902Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.5976265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.5976594Z return mod(**inputs) 2025-08-14T21:41:08.5976957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.5977358Z outputs = self.bert( 2025-08-14T21:41:08.5977711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.5978094Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.5978467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.5978835Z layer_outputs = layer_module( 2025-08-14T21:41:08.5979185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.5979551Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.5979914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.5980294Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.5980681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.5981063Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.5981454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.5981890Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.5983132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:08.5983552Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:08.5983918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:08.5984247Z return self.act(input) 2025-08-14T21:41:08.5984356Z 2025-08-14T21:41:08.5984463Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.5984807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.5985128Z return mod(**inputs) 2025-08-14T21:41:08.5985477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.5985839Z outputs = self.bert( 2025-08-14T21:41:08.5986170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.5986540Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.5986908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.5987257Z layer_outputs = layer_module( 2025-08-14T21:41:08.5987577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.5987912Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.5988267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.5988629Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.5989005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.5989381Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.5989769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:08.5990220Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:08.5990643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:08.5991022Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.5991156Z 2025-08-14T21:41:08.5991256Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.5991624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.5991993Z return mod(**inputs) 2025-08-14T21:41:08.5992332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.5992694Z outputs = self.bert( 2025-08-14T21:41:08.5993030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.5993401Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.5993761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.5994152Z layer_outputs = layer_module( 2025-08-14T21:41:08.5994481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.5994831Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.5995207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.5995576Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.5995948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.5996312Z return func(*args, **kwargs) 2025-08-14T21:41:08.5996696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.5997061Z self_outputs = self.self( 2025-08-14T21:41:08.5997413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.5997773Z return func(*args, **kwargs) 2025-08-14T21:41:08.5998124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:08.5998640Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:08.5998911Z 2025-08-14T21:41:08.5999014Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.5999373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.5999692Z return mod(**inputs) 2025-08-14T21:41:08.6000052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6000430Z outputs = self.bert( 2025-08-14T21:41:08.6000785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6001159Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6001552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6001951Z layer_outputs = layer_module( 2025-08-14T21:41:08.6002290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6002674Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6003080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6003501Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6003899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6004295Z return func(*args, **kwargs) 2025-08-14T21:41:08.6004680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6005214Z self_outputs = self.self( 2025-08-14T21:41:08.6005619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6006056Z return func(*args, **kwargs) 2025-08-14T21:41:08.6006460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:08.6006860Z self.key(current_states) 2025-08-14T21:41:08.6006991Z 2025-08-14T21:41:08.6007101Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6007488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6007837Z return mod(**inputs) 2025-08-14T21:41:08.6008226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6008624Z outputs = self.bert( 2025-08-14T21:41:08.6009000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6009400Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6009797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6010203Z layer_outputs = layer_module( 2025-08-14T21:41:08.6010566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6010938Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6011367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6011745Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6012106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6012474Z return func(*args, **kwargs) 2025-08-14T21:41:08.6012831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6013200Z self_outputs = self.self( 2025-08-14T21:41:08.6013541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6013898Z return func(*args, **kwargs) 2025-08-14T21:41:08.6014251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:08.6014614Z self.value(current_states) 2025-08-14T21:41:08.6014738Z 2025-08-14T21:41:08.6014819Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.6015052Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6015397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6015707Z return mod(**inputs) 2025-08-14T21:41:08.6016054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6016418Z outputs = self.bert( 2025-08-14T21:41:08.6016752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6017126Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6017485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6017851Z layer_outputs = layer_module( 2025-08-14T21:41:08.6018176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6018528Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6018900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6019278Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6019638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6020014Z return func(*args, **kwargs) 2025-08-14T21:41:08.6020370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6020728Z self_outputs = self.self( 2025-08-14T21:41:08.6021076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6021437Z return func(*args, **kwargs) 2025-08-14T21:41:08.6021790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:08.6022223Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:08.6022409Z 2025-08-14T21:41:08.6022514Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6022877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6023183Z return mod(**inputs) 2025-08-14T21:41:08.6023526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6023888Z outputs = self.bert( 2025-08-14T21:41:08.6024227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6024587Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6024991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6025364Z layer_outputs = layer_module( 2025-08-14T21:41:08.6025688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6026035Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6026409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6026795Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6027160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6027530Z return func(*args, **kwargs) 2025-08-14T21:41:08.6027891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:08.6028319Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:08.6028735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:08.6029124Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6029261Z 2025-08-14T21:41:08.6029372Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6029718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6030044Z return mod(**inputs) 2025-08-14T21:41:08.6030398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6030773Z outputs = self.bert( 2025-08-14T21:41:08.6031114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6031496Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6031872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6032246Z layer_outputs = layer_module( 2025-08-14T21:41:08.6032582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6032939Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6033314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6033717Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6034111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6034499Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6034903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.6035347Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.6035798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:08.6036184Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6036321Z 2025-08-14T21:41:08.6036427Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6036786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6037115Z return mod(**inputs) 2025-08-14T21:41:08.6037473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6037987Z outputs = self.bert( 2025-08-14T21:41:08.6038349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6038804Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6039171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6039550Z layer_outputs = layer_module( 2025-08-14T21:41:08.6039894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6040254Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6040629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6041020Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6041420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6041814Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6042218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.6042698Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.6043145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:08.6043588Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:08.6043992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:08.6044355Z return self.act(input) 2025-08-14T21:41:08.6044472Z 2025-08-14T21:41:08.6044587Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6044972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6045384Z return mod(**inputs) 2025-08-14T21:41:08.6045791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6046212Z outputs = self.bert( 2025-08-14T21:41:08.6046607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6047034Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6047434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6047842Z layer_outputs = layer_module( 2025-08-14T21:41:08.6048208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6048626Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6049044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6049469Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6049903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6050331Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6050778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:08.6051285Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:08.6051794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:08.6052223Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6052366Z 2025-08-14T21:41:08.6052478Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6052865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6053217Z return mod(**inputs) 2025-08-14T21:41:08.6053647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6054036Z outputs = self.bert( 2025-08-14T21:41:08.6054406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6054804Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6055193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6055587Z layer_outputs = layer_module( 2025-08-14T21:41:08.6055948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6056336Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6056728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6057110Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6057487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6057850Z return func(*args, **kwargs) 2025-08-14T21:41:08.6058268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6058641Z self_outputs = self.self( 2025-08-14T21:41:08.6058996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6059356Z return func(*args, **kwargs) 2025-08-14T21:41:08.6059715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:08.6060225Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:08.6060483Z 2025-08-14T21:41:08.6060593Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6060944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6061265Z return mod(**inputs) 2025-08-14T21:41:08.6061618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6061991Z outputs = self.bert( 2025-08-14T21:41:08.6062343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6062741Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6063112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6063477Z layer_outputs = layer_module( 2025-08-14T21:41:08.6063819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6064178Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6064558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6064960Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6065333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6065698Z return func(*args, **kwargs) 2025-08-14T21:41:08.6066051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6066426Z self_outputs = self.self( 2025-08-14T21:41:08.6066780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6067145Z return func(*args, **kwargs) 2025-08-14T21:41:08.6067509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:08.6067940Z self.key(current_states) 2025-08-14T21:41:08.6068062Z 2025-08-14T21:41:08.6068178Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6068559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6068877Z return mod(**inputs) 2025-08-14T21:41:08.6069228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6069594Z outputs = self.bert( 2025-08-14T21:41:08.6069940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6070320Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6070687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6071056Z layer_outputs = layer_module( 2025-08-14T21:41:08.6071397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6071752Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6072131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6072508Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6072881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6073253Z return func(*args, **kwargs) 2025-08-14T21:41:08.6073610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6073985Z self_outputs = self.self( 2025-08-14T21:41:08.6074328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6074692Z return func(*args, **kwargs) 2025-08-14T21:41:08.6075040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:08.6075409Z self.value(current_states) 2025-08-14T21:41:08.6075522Z 2025-08-14T21:41:08.6075610Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.6075834Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6076178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6076514Z return mod(**inputs) 2025-08-14T21:41:08.6076859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6077213Z outputs = self.bert( 2025-08-14T21:41:08.6077555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6077928Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6078284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6078671Z layer_outputs = layer_module( 2025-08-14T21:41:08.6079002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6079349Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6079707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6080099Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6080489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6080879Z return func(*args, **kwargs) 2025-08-14T21:41:08.6081255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6081694Z self_outputs = self.self( 2025-08-14T21:41:08.6082061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6082442Z return func(*args, **kwargs) 2025-08-14T21:41:08.6082833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:08.6083304Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:08.6083501Z 2025-08-14T21:41:08.6083618Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6083993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6084335Z return mod(**inputs) 2025-08-14T21:41:08.6084717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6085189Z outputs = self.bert( 2025-08-14T21:41:08.6085589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6086028Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6086426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6086816Z layer_outputs = layer_module( 2025-08-14T21:41:08.6087153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6087508Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6087884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6088260Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6088635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6089002Z return func(*args, **kwargs) 2025-08-14T21:41:08.6089363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:08.6089801Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:08.6090240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:08.6090635Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6090797Z 2025-08-14T21:41:08.6090900Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6091260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6091595Z return mod(**inputs) 2025-08-14T21:41:08.6091939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6092327Z outputs = self.bert( 2025-08-14T21:41:08.6092684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6093087Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6093448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6093822Z layer_outputs = layer_module( 2025-08-14T21:41:08.6094164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6094522Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6094891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6095283Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6095706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6096095Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6096503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.6096954Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.6097372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:08.6097752Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6097895Z 2025-08-14T21:41:08.6097998Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6098353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6098667Z return mod(**inputs) 2025-08-14T21:41:08.6099025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6099397Z outputs = self.bert( 2025-08-14T21:41:08.6099748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6100122Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6100491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6100867Z layer_outputs = layer_module( 2025-08-14T21:41:08.6101207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6101560Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6101942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6102330Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6102725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6103114Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6103519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.6103975Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.6104376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:08.6104801Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:08.6105167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:08.6105489Z return self.act(input) 2025-08-14T21:41:08.6105603Z 2025-08-14T21:41:08.6105703Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6106056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6106370Z return mod(**inputs) 2025-08-14T21:41:08.6106729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6107096Z outputs = self.bert( 2025-08-14T21:41:08.6107446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6107818Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6108175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6108538Z layer_outputs = layer_module( 2025-08-14T21:41:08.6108871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6109213Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6109621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6110016Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6110419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6110804Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6111214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:08.6111686Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:08.6112113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:08.6112510Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6112653Z 2025-08-14T21:41:08.6112759Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6113169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6113481Z return mod(**inputs) 2025-08-14T21:41:08.6113830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6114195Z outputs = self.bert( 2025-08-14T21:41:08.6114541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6114911Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6115278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6115649Z layer_outputs = layer_module( 2025-08-14T21:41:08.6115979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6116334Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6116711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6117094Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6117456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6117818Z return func(*args, **kwargs) 2025-08-14T21:41:08.6118177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6118564Z self_outputs = self.self( 2025-08-14T21:41:08.6118920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6119285Z return func(*args, **kwargs) 2025-08-14T21:41:08.6119645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:08.6120162Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:08.6120451Z 2025-08-14T21:41:08.6120560Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6120941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6121292Z return mod(**inputs) 2025-08-14T21:41:08.6121661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6122060Z outputs = self.bert( 2025-08-14T21:41:08.6122432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6122828Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6123221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6123649Z layer_outputs = layer_module( 2025-08-14T21:41:08.6124013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6124391Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6124812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6125337Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6125940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6126538Z return func(*args, **kwargs) 2025-08-14T21:41:08.6127169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6127603Z self_outputs = self.self( 2025-08-14T21:41:08.6127993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6128403Z return func(*args, **kwargs) 2025-08-14T21:41:08.6128802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:08.6129214Z self.key(current_states) 2025-08-14T21:41:08.6129340Z 2025-08-14T21:41:08.6129457Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6129852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6130206Z return mod(**inputs) 2025-08-14T21:41:08.6130671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6131301Z outputs = self.bert( 2025-08-14T21:41:08.6131690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6132109Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6132513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6132937Z layer_outputs = layer_module( 2025-08-14T21:41:08.6133311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6133691Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6134113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6134579Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6134993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6135389Z return func(*args, **kwargs) 2025-08-14T21:41:08.6135786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6136210Z self_outputs = self.self( 2025-08-14T21:41:08.6136589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6137026Z return func(*args, **kwargs) 2025-08-14T21:41:08.6137421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:08.6138098Z self.value(current_states) 2025-08-14T21:41:08.6138231Z 2025-08-14T21:41:08.6138327Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.6138592Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6138986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6139332Z return mod(**inputs) 2025-08-14T21:41:08.6139726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6140209Z outputs = self.bert( 2025-08-14T21:41:08.6140596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6141006Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6141406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6141814Z layer_outputs = layer_module( 2025-08-14T21:41:08.6142182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6142564Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6142975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6143396Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6143805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6144207Z return func(*args, **kwargs) 2025-08-14T21:41:08.6144604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6145009Z self_outputs = self.self( 2025-08-14T21:41:08.6145388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6145788Z return func(*args, **kwargs) 2025-08-14T21:41:08.6146183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:08.6146648Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:08.6146856Z 2025-08-14T21:41:08.6146967Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6147357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6147706Z return mod(**inputs) 2025-08-14T21:41:08.6148083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6148489Z outputs = self.bert( 2025-08-14T21:41:08.6148872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6149266Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6149656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6150087Z layer_outputs = layer_module( 2025-08-14T21:41:08.6150456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6150833Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6151246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6151667Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6152094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6152476Z return func(*args, **kwargs) 2025-08-14T21:41:08.6152859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:08.6153316Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:08.6153760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:08.6154171Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6154321Z 2025-08-14T21:41:08.6154430Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6154805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6155169Z return mod(**inputs) 2025-08-14T21:41:08.6155549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6155951Z outputs = self.bert( 2025-08-14T21:41:08.6156327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6156746Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6157151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6157573Z layer_outputs = layer_module( 2025-08-14T21:41:08.6157894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6158234Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6158595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6158962Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6159337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6159710Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6160095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.6160524Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.6160951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:08.6161339Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6161474Z 2025-08-14T21:41:08.6161581Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6161932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6162254Z return mod(**inputs) 2025-08-14T21:41:08.6162611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6162983Z outputs = self.bert( 2025-08-14T21:41:08.6163326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6163710Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6164094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6164461Z layer_outputs = layer_module( 2025-08-14T21:41:08.6164802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6165264Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6165688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6166121Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6166548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6166942Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6167344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.6167806Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.6168219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:08.6168628Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:08.6168998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:08.6169368Z return self.act(input) 2025-08-14T21:41:08.6169479Z 2025-08-14T21:41:08.6169588Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6169937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6170255Z return mod(**inputs) 2025-08-14T21:41:08.6170611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6170984Z outputs = self.bert( 2025-08-14T21:41:08.6171334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6171721Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6172095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6172482Z layer_outputs = layer_module( 2025-08-14T21:41:08.6172801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6173141Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6173505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6173868Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6174251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6174637Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6175037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:08.6175472Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:08.6175886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:08.6176254Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6176385Z 2025-08-14T21:41:08.6176491Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6176822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6177129Z return mod(**inputs) 2025-08-14T21:41:08.6177475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6177847Z outputs = self.bert( 2025-08-14T21:41:08.6178196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6178559Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6178916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6179276Z layer_outputs = layer_module( 2025-08-14T21:41:08.6179610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6179975Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6180342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6180721Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6181089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6181450Z return func(*args, **kwargs) 2025-08-14T21:41:08.6181810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6182185Z self_outputs = self.self( 2025-08-14T21:41:08.6182573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6182927Z return func(*args, **kwargs) 2025-08-14T21:41:08.6183282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:08.6183788Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:08.6184046Z 2025-08-14T21:41:08.6184155Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6184552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6184906Z return mod(**inputs) 2025-08-14T21:41:08.6185282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6185644Z outputs = self.bert( 2025-08-14T21:41:08.6185979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6186410Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6186779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6187144Z layer_outputs = layer_module( 2025-08-14T21:41:08.6187493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6187836Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6188201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6188568Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6188934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6189293Z return func(*args, **kwargs) 2025-08-14T21:41:08.6189643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6190017Z self_outputs = self.self( 2025-08-14T21:41:08.6190373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6190739Z return func(*args, **kwargs) 2025-08-14T21:41:08.6191097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:08.6191473Z self.key(current_states) 2025-08-14T21:41:08.6191622Z 2025-08-14T21:41:08.6191731Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6192089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6192403Z return mod(**inputs) 2025-08-14T21:41:08.6192771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6193139Z outputs = self.bert( 2025-08-14T21:41:08.6193475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6193864Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6194227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6194591Z layer_outputs = layer_module( 2025-08-14T21:41:08.6194923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6195293Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6195690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6196097Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6196528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6196920Z return func(*args, **kwargs) 2025-08-14T21:41:08.6197302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6197687Z self_outputs = self.self( 2025-08-14T21:41:08.6198056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6198443Z return func(*args, **kwargs) 2025-08-14T21:41:08.6198818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:08.6199214Z self.value(current_states) 2025-08-14T21:41:08.6199343Z 2025-08-14T21:41:08.6199428Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.6199681Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6200056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6200394Z return mod(**inputs) 2025-08-14T21:41:08.6200763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6201149Z outputs = self.bert( 2025-08-14T21:41:08.6201518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6201917Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6202308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6202695Z layer_outputs = layer_module( 2025-08-14T21:41:08.6203054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6203441Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6203842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6204242Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6204642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6205115Z return func(*args, **kwargs) 2025-08-14T21:41:08.6205520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6205929Z self_outputs = self.self( 2025-08-14T21:41:08.6206349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6206744Z return func(*args, **kwargs) 2025-08-14T21:41:08.6207129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:08.6207589Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:08.6207786Z 2025-08-14T21:41:08.6207905Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6208298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6208640Z return mod(**inputs) 2025-08-14T21:41:08.6209027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6209429Z outputs = self.bert( 2025-08-14T21:41:08.6209795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6210199Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6210592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6210966Z layer_outputs = layer_module( 2025-08-14T21:41:08.6211336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6211706Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6212075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6212444Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6212808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6213165Z return func(*args, **kwargs) 2025-08-14T21:41:08.6213524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:08.6213939Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:08.6214360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:08.6214743Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6214877Z 2025-08-14T21:41:08.6214977Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6215332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6215649Z return mod(**inputs) 2025-08-14T21:41:08.6216001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6216360Z outputs = self.bert( 2025-08-14T21:41:08.6216711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6217097Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6217464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6217842Z layer_outputs = layer_module( 2025-08-14T21:41:08.6218192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6218553Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6218933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6219318Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6219710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6220127Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6220522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.6220971Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.6221394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:08.6221764Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6221904Z 2025-08-14T21:41:08.6222004Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6222374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6222692Z return mod(**inputs) 2025-08-14T21:41:08.6223030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6223392Z outputs = self.bert( 2025-08-14T21:41:08.6223735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6224104Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6224459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6224823Z layer_outputs = layer_module( 2025-08-14T21:41:08.6225184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6225528Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6225896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6226274Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6226657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6227036Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6227432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.6227875Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.6228288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:08.6228701Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:08.6229077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:08.6229411Z return self.act(input) 2025-08-14T21:41:08.6229520Z 2025-08-14T21:41:08.6229623Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6229981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6230304Z return mod(**inputs) 2025-08-14T21:41:08.6230697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6231060Z outputs = self.bert( 2025-08-14T21:41:08.6231409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6231792Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6232158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6232537Z layer_outputs = layer_module( 2025-08-14T21:41:08.6232880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6233236Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6233607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6234020Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6234418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6234807Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6235217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:08.6235680Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:08.6236135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:08.6236520Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6236663Z 2025-08-14T21:41:08.6236767Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6237124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6237448Z return mod(**inputs) 2025-08-14T21:41:08.6238109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6238510Z outputs = self.bert( 2025-08-14T21:41:08.6238871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6239340Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6239721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6240104Z layer_outputs = layer_module( 2025-08-14T21:41:08.6240465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6240834Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6241235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6241641Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6242032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6242424Z return func(*args, **kwargs) 2025-08-14T21:41:08.6242816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6243210Z self_outputs = self.self( 2025-08-14T21:41:08.6243594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6243990Z return func(*args, **kwargs) 2025-08-14T21:41:08.6244373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:08.6244916Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:08.6245260Z 2025-08-14T21:41:08.6245375Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6245764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6246137Z return mod(**inputs) 2025-08-14T21:41:08.6246486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6246915Z outputs = self.bert( 2025-08-14T21:41:08.6247307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6247737Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6248131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6248538Z layer_outputs = layer_module( 2025-08-14T21:41:08.6248949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6249341Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6249753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6250170Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6250583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6250686Z return func(*args, **kwargs) 2025-08-14T21:41:08.6250952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6251039Z self_outputs = self.self( 2025-08-14T21:41:08.6251307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6251395Z return func(*args, **kwargs) 2025-08-14T21:41:08.6251660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:08.6251736Z self.key(current_states) 2025-08-14T21:41:08.6251740Z 2025-08-14T21:41:08.6251859Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6252117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6252190Z return mod(**inputs) 2025-08-14T21:41:08.6252476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6252549Z outputs = self.bert( 2025-08-14T21:41:08.6252831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6252911Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6253172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6253260Z layer_outputs = layer_module( 2025-08-14T21:41:08.6253496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6253582Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6253852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6253939Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6254187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6254254Z return func(*args, **kwargs) 2025-08-14T21:41:08.6254481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6254554Z self_outputs = self.self( 2025-08-14T21:41:08.6254777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6254848Z return func(*args, **kwargs) 2025-08-14T21:41:08.6255074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:08.6255143Z self.value(current_states) 2025-08-14T21:41:08.6255147Z 2025-08-14T21:41:08.6255232Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.6255330Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6255520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6255590Z return mod(**inputs) 2025-08-14T21:41:08.6255818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6255888Z outputs = self.bert( 2025-08-14T21:41:08.6256134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6256204Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6256435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6256501Z layer_outputs = layer_module( 2025-08-14T21:41:08.6256708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6256788Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6257032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6257114Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6257336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6257403Z return func(*args, **kwargs) 2025-08-14T21:41:08.6257638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6257705Z self_outputs = self.self( 2025-08-14T21:41:08.6257926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6257999Z return func(*args, **kwargs) 2025-08-14T21:41:08.6258251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:08.6258385Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:08.6258388Z 2025-08-14T21:41:08.6258485Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6258670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6258739Z return mod(**inputs) 2025-08-14T21:41:08.6258973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6259043Z outputs = self.bert( 2025-08-14T21:41:08.6259276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6259348Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6259585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6259653Z layer_outputs = layer_module( 2025-08-14T21:41:08.6259861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6259943Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6260170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6260252Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6260472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6260537Z return func(*args, **kwargs) 2025-08-14T21:41:08.6260770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:08.6260892Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:08.6261122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:08.6261210Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6261213Z 2025-08-14T21:41:08.6261308Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6261502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6261563Z return mod(**inputs) 2025-08-14T21:41:08.6261824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6261896Z outputs = self.bert( 2025-08-14T21:41:08.6262137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6262216Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6262461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6262578Z layer_outputs = layer_module( 2025-08-14T21:41:08.6262799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6262875Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6263118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6263207Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6263454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6263542Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6263800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.6263938Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.6264173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:08.6264253Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6264257Z 2025-08-14T21:41:08.6264359Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6264545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6264608Z return mod(**inputs) 2025-08-14T21:41:08.6264851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6264913Z outputs = self.bert( 2025-08-14T21:41:08.6265149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6265229Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6265464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6265543Z layer_outputs = layer_module( 2025-08-14T21:41:08.6265753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6265827Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6266068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6266149Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6266397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6266477Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6266746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.6266866Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.6267112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:08.6267217Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:08.6267424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:08.6267490Z return self.act(input) 2025-08-14T21:41:08.6267509Z 2025-08-14T21:41:08.6267617Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6267809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6267872Z return mod(**inputs) 2025-08-14T21:41:08.6268118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6268183Z outputs = self.bert( 2025-08-14T21:41:08.6268421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6268513Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6268747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6268823Z layer_outputs = layer_module( 2025-08-14T21:41:08.6269034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6269109Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6269348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6269429Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6269725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6269799Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6270062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:08.6270202Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:08.6270434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:08.6270511Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6270523Z 2025-08-14T21:41:08.6270621Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6270810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6270879Z return mod(**inputs) 2025-08-14T21:41:08.6271124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6271188Z outputs = self.bert( 2025-08-14T21:41:08.6271441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6271512Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6271749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6271816Z layer_outputs = layer_module( 2025-08-14T21:41:08.6272026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6272108Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6272339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6272416Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6272657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6272723Z return func(*args, **kwargs) 2025-08-14T21:41:08.6272962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6273027Z self_outputs = self.self( 2025-08-14T21:41:08.6273251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6273323Z return func(*args, **kwargs) 2025-08-14T21:41:08.6273573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:08.6273771Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:08.6273782Z 2025-08-14T21:41:08.6273880Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6274073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6274145Z return mod(**inputs) 2025-08-14T21:41:08.6274409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6274470Z outputs = self.bert( 2025-08-14T21:41:08.6274707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6274777Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6275010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6275078Z layer_outputs = layer_module( 2025-08-14T21:41:08.6275280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6275360Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6275616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6275695Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6275928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6275993Z return func(*args, **kwargs) 2025-08-14T21:41:08.6276225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6276292Z self_outputs = self.self( 2025-08-14T21:41:08.6276514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6276587Z return func(*args, **kwargs) 2025-08-14T21:41:08.6276812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:08.6276882Z self.key(current_states) 2025-08-14T21:41:08.6276892Z 2025-08-14T21:41:08.6276990Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6277177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6277245Z return mod(**inputs) 2025-08-14T21:41:08.6277474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6277534Z outputs = self.bert( 2025-08-14T21:41:08.6277773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6277842Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6278076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6278142Z layer_outputs = layer_module( 2025-08-14T21:41:08.6278349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6278431Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6278657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6278734Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6278966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6279047Z return func(*args, **kwargs) 2025-08-14T21:41:08.6279282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6279347Z self_outputs = self.self( 2025-08-14T21:41:08.6279571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6279647Z return func(*args, **kwargs) 2025-08-14T21:41:08.6279887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:08.6279977Z self.value(current_states) 2025-08-14T21:41:08.6279988Z 2025-08-14T21:41:08.6280070Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.6280171Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6280382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6280445Z return mod(**inputs) 2025-08-14T21:41:08.6280687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6280760Z outputs = self.bert( 2025-08-14T21:41:08.6281005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6281080Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6281355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6281429Z layer_outputs = layer_module( 2025-08-14T21:41:08.6281657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6281733Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6281978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6282079Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6282310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6282384Z return func(*args, **kwargs) 2025-08-14T21:41:08.6282625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6282696Z self_outputs = self.self( 2025-08-14T21:41:08.6282942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6283013Z return func(*args, **kwargs) 2025-08-14T21:41:08.6283257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:08.6283397Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:08.6283400Z 2025-08-14T21:41:08.6283504Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6283708Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6283772Z return mod(**inputs) 2025-08-14T21:41:08.6284018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6284090Z outputs = self.bert( 2025-08-14T21:41:08.6284338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6284412Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6284661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6284731Z layer_outputs = layer_module( 2025-08-14T21:41:08.6284958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6285121Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6285369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6285457Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6285692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6285779Z return func(*args, **kwargs) 2025-08-14T21:41:08.6286042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:08.6286202Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:08.6286473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:08.6286565Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6286569Z 2025-08-14T21:41:08.6286681Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6286904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6286976Z return mod(**inputs) 2025-08-14T21:41:08.6287258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6287329Z outputs = self.bert( 2025-08-14T21:41:08.6287619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6287719Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6287962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6288039Z layer_outputs = layer_module( 2025-08-14T21:41:08.6288256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6288334Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6288582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6288666Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6288923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6289009Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6289282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.6289409Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.6289650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:08.6289732Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6289737Z 2025-08-14T21:41:08.6289844Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6290041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6290113Z return mod(**inputs) 2025-08-14T21:41:08.6290373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6290443Z outputs = self.bert( 2025-08-14T21:41:08.6290707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6290786Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6291040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6291121Z layer_outputs = layer_module( 2025-08-14T21:41:08.6291349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6291458Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6291712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6291802Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6292082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6292162Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6292481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.6292612Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.6292865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:08.6292991Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:08.6293211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:08.6293284Z return self.act(input) 2025-08-14T21:41:08.6293289Z 2025-08-14T21:41:08.6293404Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6293641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6293718Z return mod(**inputs) 2025-08-14T21:41:08.6293979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6294049Z outputs = self.bert( 2025-08-14T21:41:08.6294312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6294389Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6294644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6294729Z layer_outputs = layer_module( 2025-08-14T21:41:08.6294958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6295045Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6295303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6295386Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6295650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6295724Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6295996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:08.6296136Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:08.6296376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:08.6296466Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6296470Z 2025-08-14T21:41:08.6296571Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6296768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6296843Z return mod(**inputs) 2025-08-14T21:41:08.6297091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6297167Z outputs = self.bert( 2025-08-14T21:41:08.6297408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6297482Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6297744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6297815Z layer_outputs = layer_module( 2025-08-14T21:41:08.6298030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6298114Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6298353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6298455Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6298695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6298766Z return func(*args, **kwargs) 2025-08-14T21:41:08.6299015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6299087Z self_outputs = self.self( 2025-08-14T21:41:08.6299328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6299397Z return func(*args, **kwargs) 2025-08-14T21:41:08.6299639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:08.6299875Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:08.6299879Z 2025-08-14T21:41:08.6299985Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6300181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6300253Z return mod(**inputs) 2025-08-14T21:41:08.6300497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6300575Z outputs = self.bert( 2025-08-14T21:41:08.6300832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6300909Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6301173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6301248Z layer_outputs = layer_module( 2025-08-14T21:41:08.6301477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6301567Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6301818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6301911Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6302158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6302233Z return func(*args, **kwargs) 2025-08-14T21:41:08.6302496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6302569Z self_outputs = self.self( 2025-08-14T21:41:08.6302825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6302900Z return func(*args, **kwargs) 2025-08-14T21:41:08.6303155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:08.6303238Z self.key(current_states) 2025-08-14T21:41:08.6303242Z 2025-08-14T21:41:08.6303347Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6303557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6303645Z return mod(**inputs) 2025-08-14T21:41:08.6303909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6303982Z outputs = self.bert( 2025-08-14T21:41:08.6304225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6304297Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6304551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6304641Z layer_outputs = layer_module( 2025-08-14T21:41:08.6304858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6304945Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6305187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6305276Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6305512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6305579Z return func(*args, **kwargs) 2025-08-14T21:41:08.6305827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6305927Z self_outputs = self.self( 2025-08-14T21:41:08.6306172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6306241Z return func(*args, **kwargs) 2025-08-14T21:41:08.6306478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:08.6306556Z self.value(current_states) 2025-08-14T21:41:08.6306560Z 2025-08-14T21:41:08.6306640Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.6306742Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6306946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6307009Z return mod(**inputs) 2025-08-14T21:41:08.6307261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6307326Z outputs = self.bert( 2025-08-14T21:41:08.6307575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6307656Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6307948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6308022Z layer_outputs = layer_module( 2025-08-14T21:41:08.6308257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6308340Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6308608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6308692Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6308948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6309031Z return func(*args, **kwargs) 2025-08-14T21:41:08.6309286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6309359Z self_outputs = self.self( 2025-08-14T21:41:08.6309622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6309704Z return func(*args, **kwargs) 2025-08-14T21:41:08.6309951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:08.6310103Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:08.6310107Z 2025-08-14T21:41:08.6310208Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6310412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6310477Z return mod(**inputs) 2025-08-14T21:41:08.6310733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6310813Z outputs = self.bert( 2025-08-14T21:41:08.6311066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6311148Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6311406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6311481Z layer_outputs = layer_module( 2025-08-14T21:41:08.6311721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6311799Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6312078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6312198Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6312441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6312520Z return func(*args, **kwargs) 2025-08-14T21:41:08.6312785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:08.6312914Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:08.6313172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:08.6313255Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6313259Z 2025-08-14T21:41:08.6313369Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6313571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6313641Z return mod(**inputs) 2025-08-14T21:41:08.6313902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6313970Z outputs = self.bert( 2025-08-14T21:41:08.6314229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6314302Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6314568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6314649Z layer_outputs = layer_module( 2025-08-14T21:41:08.6314870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6314948Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6315223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6315307Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6315588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6315666Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6315952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.6316081Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.6316379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:08.6316468Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6316472Z 2025-08-14T21:41:08.6316575Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6316781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6316854Z return mod(**inputs) 2025-08-14T21:41:08.6317124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6317210Z outputs = self.bert( 2025-08-14T21:41:08.6317502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6317575Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6317823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6317896Z layer_outputs = layer_module( 2025-08-14T21:41:08.6318119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6318200Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6318479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6318562Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6318824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6318901Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6319182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.6319299Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.6319540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:08.6319659Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:08.6319868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:08.6319946Z return self.act(input) 2025-08-14T21:41:08.6319949Z 2025-08-14T21:41:08.6320050Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6320248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6320318Z return mod(**inputs) 2025-08-14T21:41:08.6320563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6320627Z outputs = self.bert( 2025-08-14T21:41:08.6320879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6320951Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6321201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6321273Z layer_outputs = layer_module( 2025-08-14T21:41:08.6321492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6321577Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6321818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6321906Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6322158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6322249Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6322530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:08.6322665Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:08.6322908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:08.6322999Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6323003Z 2025-08-14T21:41:08.6323121Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6323327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6323392Z return mod(**inputs) 2025-08-14T21:41:08.6323634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6323712Z outputs = self.bert( 2025-08-14T21:41:08.6323955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6324036Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6324275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6324345Z layer_outputs = layer_module( 2025-08-14T21:41:08.6324599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6324678Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6324941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6325108Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6325374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6325463Z return func(*args, **kwargs) 2025-08-14T21:41:08.6325724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6325799Z self_outputs = self.self( 2025-08-14T21:41:08.6326063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6326143Z return func(*args, **kwargs) 2025-08-14T21:41:08.6326413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:08.6326638Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:08.6326642Z 2025-08-14T21:41:08.6326749Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6326965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6327038Z return mod(**inputs) 2025-08-14T21:41:08.6327297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6327375Z outputs = self.bert( 2025-08-14T21:41:08.6327633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6327721Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6327975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6328053Z layer_outputs = layer_module( 2025-08-14T21:41:08.6328289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6328372Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6328625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6328742Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6328988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6329067Z return func(*args, **kwargs) 2025-08-14T21:41:08.6329321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6329394Z self_outputs = self.self( 2025-08-14T21:41:08.6329649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6329741Z return func(*args, **kwargs) 2025-08-14T21:41:08.6329997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:08.6330087Z self.key(current_states) 2025-08-14T21:41:08.6330093Z 2025-08-14T21:41:08.6330199Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6330419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6330486Z return mod(**inputs) 2025-08-14T21:41:08.6330745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6330819Z outputs = self.bert( 2025-08-14T21:41:08.6331108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6331195Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6331451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6331526Z layer_outputs = layer_module( 2025-08-14T21:41:08.6331764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6331847Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6332100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6332193Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6332444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6332527Z return func(*args, **kwargs) 2025-08-14T21:41:08.6332782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6332856Z self_outputs = self.self( 2025-08-14T21:41:08.6333113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6333186Z return func(*args, **kwargs) 2025-08-14T21:41:08.6333440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:08.6333525Z self.value(current_states) 2025-08-14T21:41:08.6333529Z 2025-08-14T21:41:08.6333614Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.6333727Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6333936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6334006Z return mod(**inputs) 2025-08-14T21:41:08.6334277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6334346Z outputs = self.bert( 2025-08-14T21:41:08.6334612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6334690Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6334945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6335044Z layer_outputs = layer_module( 2025-08-14T21:41:08.6335277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6335358Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6335627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6335711Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6335963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6336052Z return func(*args, **kwargs) 2025-08-14T21:41:08.6336311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6336390Z self_outputs = self.self( 2025-08-14T21:41:08.6336648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6336720Z return func(*args, **kwargs) 2025-08-14T21:41:08.6336988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:08.6337125Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:08.6337129Z 2025-08-14T21:41:08.6337281Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6337491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6337560Z return mod(**inputs) 2025-08-14T21:41:08.6338100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6338182Z outputs = self.bert( 2025-08-14T21:41:08.6338455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6338539Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6338795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6338877Z layer_outputs = layer_module( 2025-08-14T21:41:08.6339109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6339192Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6339457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6339544Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6339800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6339873Z return func(*args, **kwargs) 2025-08-14T21:41:08.6340129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:08.6340271Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:08.6340526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:08.6340612Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6340624Z 2025-08-14T21:41:08.6340733Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6340942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6341021Z return mod(**inputs) 2025-08-14T21:41:08.6341281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6341350Z outputs = self.bert( 2025-08-14T21:41:08.6341619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6341750Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6342016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6342090Z layer_outputs = layer_module( 2025-08-14T21:41:08.6342320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6342408Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6342689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6342777Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6343062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6343143Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6343445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.6343582Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.6343826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:08.6343959Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6343963Z 2025-08-14T21:41:08.6344066Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6344269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6344337Z return mod(**inputs) 2025-08-14T21:41:08.6344581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6344653Z outputs = self.bert( 2025-08-14T21:41:08.6344899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6344972Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6345219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6345290Z layer_outputs = layer_module( 2025-08-14T21:41:08.6345519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6345594Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6345837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6345925Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6346182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6346258Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6346537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.6346656Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.6346910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:08.6347025Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:08.6347239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:08.6347320Z return self.act(input) 2025-08-14T21:41:08.6347324Z 2025-08-14T21:41:08.6347426Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6347633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6347701Z return mod(**inputs) 2025-08-14T21:41:08.6347967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6348040Z outputs = self.bert( 2025-08-14T21:41:08.6348293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6348369Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6348623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6348712Z layer_outputs = layer_module( 2025-08-14T21:41:08.6348943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6349023Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6349272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6349365Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6349626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6349702Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6349988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:08.6350151Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:08.6350411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:08.6350497Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6350500Z 2025-08-14T21:41:08.6350604Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6350815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6350882Z return mod(**inputs) 2025-08-14T21:41:08.6351144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6351209Z outputs = self.bert( 2025-08-14T21:41:08.6351460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6351546Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6351794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6351868Z layer_outputs = layer_module( 2025-08-14T21:41:08.6352096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6352174Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6352433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6352516Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6352760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6352838Z return func(*args, **kwargs) 2025-08-14T21:41:08.6353089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6353165Z self_outputs = self.self( 2025-08-14T21:41:08.6353408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6353479Z return func(*args, **kwargs) 2025-08-14T21:41:08.6353733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:08.6353942Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:08.6353965Z 2025-08-14T21:41:08.6354069Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6354277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6354343Z return mod(**inputs) 2025-08-14T21:41:08.6354605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6354672Z outputs = self.bert( 2025-08-14T21:41:08.6354916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6355014Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6355263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6355340Z layer_outputs = layer_module( 2025-08-14T21:41:08.6355553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6355629Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6355876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6355954Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6356216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6356294Z return func(*args, **kwargs) 2025-08-14T21:41:08.6356536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6356611Z self_outputs = self.self( 2025-08-14T21:41:08.6356843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6356910Z return func(*args, **kwargs) 2025-08-14T21:41:08.6357158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:08.6357225Z self.key(current_states) 2025-08-14T21:41:08.6357229Z 2025-08-14T21:41:08.6357326Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6357527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6357592Z return mod(**inputs) 2025-08-14T21:41:08.6357842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6357907Z outputs = self.bert( 2025-08-14T21:41:08.6358147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6358225Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6358463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6358532Z layer_outputs = layer_module( 2025-08-14T21:41:08.6358756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6358831Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6359079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6359157Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6359392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6359469Z return func(*args, **kwargs) 2025-08-14T21:41:08.6359708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6359781Z self_outputs = self.self( 2025-08-14T21:41:08.6360056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6360125Z return func(*args, **kwargs) 2025-08-14T21:41:08.6360376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:08.6360453Z self.value(current_states) 2025-08-14T21:41:08.6360456Z 2025-08-14T21:41:08.6360544Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.6360659Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6360885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6360960Z return mod(**inputs) 2025-08-14T21:41:08.6361215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6361283Z outputs = self.bert( 2025-08-14T21:41:08.6361546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6361624Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6361876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6361959Z layer_outputs = layer_module( 2025-08-14T21:41:08.6362230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6362327Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6362569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6362646Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6362889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6362962Z return func(*args, **kwargs) 2025-08-14T21:41:08.6363221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6363294Z self_outputs = self.self( 2025-08-14T21:41:08.6363539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6363619Z return func(*args, **kwargs) 2025-08-14T21:41:08.6363873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:08.6364012Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:08.6364015Z 2025-08-14T21:41:08.6364129Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6364335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6364413Z return mod(**inputs) 2025-08-14T21:41:08.6364667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6364739Z outputs = self.bert( 2025-08-14T21:41:08.6365072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6365162Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6365430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6365517Z layer_outputs = layer_module( 2025-08-14T21:41:08.6365758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6365850Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6366112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6366204Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6366468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6366535Z return func(*args, **kwargs) 2025-08-14T21:41:08.6366778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:08.6366899Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:08.6367137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:08.6367244Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6367248Z 2025-08-14T21:41:08.6367346Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6367536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6367608Z return mod(**inputs) 2025-08-14T21:41:08.6367846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6367916Z outputs = self.bert( 2025-08-14T21:41:08.6368151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6368222Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6368569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6368646Z layer_outputs = layer_module( 2025-08-14T21:41:08.6368865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6368949Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6369190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6369278Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6369534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6369607Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6369889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.6370007Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.6370257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:08.6370339Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6370343Z 2025-08-14T21:41:08.6370443Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6370647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6370711Z return mod(**inputs) 2025-08-14T21:41:08.6370956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6371029Z outputs = self.bert( 2025-08-14T21:41:08.6371273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6371353Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6371598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6371668Z layer_outputs = layer_module( 2025-08-14T21:41:08.6371891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6371967Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6372219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6372318Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6372571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6372654Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6372923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.6373042Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.6373288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:08.6374280Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:08.6374494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:08.6374563Z return self.act(input) 2025-08-14T21:41:08.6374568Z 2025-08-14T21:41:08.6374667Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6374865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6374930Z return mod(**inputs) 2025-08-14T21:41:08.6375171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6375241Z outputs = self.bert( 2025-08-14T21:41:08.6375508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6375590Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6375828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6375900Z layer_outputs = layer_module( 2025-08-14T21:41:08.6376130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6376209Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6376463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6376547Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6376805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6376889Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6377165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:08.6377305Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:08.6377555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:08.6377633Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6377638Z 2025-08-14T21:41:08.6377741Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6377933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6377997Z return mod(**inputs) 2025-08-14T21:41:08.6378246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6378313Z outputs = self.bert( 2025-08-14T21:41:08.6378567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6378642Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6378888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6378966Z layer_outputs = layer_module( 2025-08-14T21:41:08.6379185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6379279Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6379533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6379613Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6379862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6379932Z return func(*args, **kwargs) 2025-08-14T21:41:08.6380190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6380267Z self_outputs = self.self( 2025-08-14T21:41:08.6380505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6380573Z return func(*args, **kwargs) 2025-08-14T21:41:08.6380823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 374, in forward 2025-08-14T21:41:08.6381023Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:41:08.6381027Z 2025-08-14T21:41:08.6381134Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6381360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6381426Z return mod(**inputs) 2025-08-14T21:41:08.6381686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6381750Z outputs = self.bert( 2025-08-14T21:41:08.6382009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6382082Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6382323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6382401Z layer_outputs = layer_module( 2025-08-14T21:41:08.6382618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6382694Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6382944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6383023Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6383266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6383334Z return func(*args, **kwargs) 2025-08-14T21:41:08.6383576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6383652Z self_outputs = self.self( 2025-08-14T21:41:08.6383887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6383953Z return func(*args, **kwargs) 2025-08-14T21:41:08.6384202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 402, in forward 2025-08-14T21:41:08.6384270Z self.key(current_states) 2025-08-14T21:41:08.6384277Z 2025-08-14T21:41:08.6384385Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6384583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6384649Z return mod(**inputs) 2025-08-14T21:41:08.6384906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6384969Z outputs = self.bert( 2025-08-14T21:41:08.6385218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6385308Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6385550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6385627Z layer_outputs = layer_module( 2025-08-14T21:41:08.6385847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6385923Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6386194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6386274Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6386519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6386586Z return func(*args, **kwargs) 2025-08-14T21:41:08.6386827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6386902Z self_outputs = self.self( 2025-08-14T21:41:08.6387134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6387202Z return func(*args, **kwargs) 2025-08-14T21:41:08.6387481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 407, in forward 2025-08-14T21:41:08.6387556Z self.value(current_states) 2025-08-14T21:41:08.6387559Z 2025-08-14T21:41:08.6387648Z cudagraph partition due to non gpu ops 2025-08-14T21:41:08.6387749Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6387946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6388019Z return mod(**inputs) 2025-08-14T21:41:08.6388268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6388342Z outputs = self.bert( 2025-08-14T21:41:08.6388584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6388657Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6388906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6388977Z layer_outputs = layer_module( 2025-08-14T21:41:08.6389196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6389280Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6389520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6389609Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6389857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6389929Z return func(*args, **kwargs) 2025-08-14T21:41:08.6390190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 514, in forward 2025-08-14T21:41:08.6390263Z self_outputs = self.self( 2025-08-14T21:41:08.6390512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6390593Z return func(*args, **kwargs) 2025-08-14T21:41:08.6390860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 438, in forward 2025-08-14T21:41:08.6391016Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:08.6391020Z 2025-08-14T21:41:08.6391125Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6391351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6391427Z return mod(**inputs) 2025-08-14T21:41:08.6391684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6391756Z outputs = self.bert( 2025-08-14T21:41:08.6392002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6392075Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6392336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6392406Z layer_outputs = layer_module( 2025-08-14T21:41:08.6392621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6392705Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6392959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 584, in forward 2025-08-14T21:41:08.6393043Z self_attention_outputs = self.attention( 2025-08-14T21:41:08.6393270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:41:08.6393336Z return func(*args, **kwargs) 2025-08-14T21:41:08.6393602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 524, in forward 2025-08-14T21:41:08.6393727Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:41:08.6393962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 461, in forward 2025-08-14T21:41:08.6394049Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6394052Z 2025-08-14T21:41:08.6394149Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6394350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6394414Z return mod(**inputs) 2025-08-14T21:41:08.6394651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6394722Z outputs = self.bert( 2025-08-14T21:41:08.6394969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6395050Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6395286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6395355Z layer_outputs = layer_module( 2025-08-14T21:41:08.6395575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6395650Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6395884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6395974Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6396220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6396304Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6396566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.6396681Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.6396921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 539, in forward 2025-08-14T21:41:08.6396999Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6397019Z 2025-08-14T21:41:08.6397124Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6397318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6397380Z return mod(**inputs) 2025-08-14T21:41:08.6397626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6397690Z outputs = self.bert( 2025-08-14T21:41:08.6397932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6398026Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6398262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6398335Z layer_outputs = layer_module( 2025-08-14T21:41:08.6398542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6398618Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6398857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6398936Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6399183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6399291Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6399558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 622, in feed_forward_chunk 2025-08-14T21:41:08.6399680Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:41:08.6399919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 540, in forward 2025-08-14T21:41:08.6400027Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:41:08.6400241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:08.6400308Z return self.act(input) 2025-08-14T21:41:08.6400311Z 2025-08-14T21:41:08.6400414Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6400608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6400673Z return mod(**inputs) 2025-08-14T21:41:08.6400918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1767, in forward 2025-08-14T21:41:08.6400983Z outputs = self.bert( 2025-08-14T21:41:08.6401251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1028, in forward 2025-08-14T21:41:08.6401338Z encoder_outputs = self.encoder( 2025-08-14T21:41:08.6401595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 675, in forward 2025-08-14T21:41:08.6401688Z layer_outputs = layer_module( 2025-08-14T21:41:08.6401904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:08.6401981Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:08.6402232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 614, in forward 2025-08-14T21:41:08.6402316Z layer_output = apply_chunking_to_forward( 2025-08-14T21:41:08.6402583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:41:08.6402658Z return forward_fn(*input_tensors) 2025-08-14T21:41:08.6402930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 623, in feed_forward_chunk 2025-08-14T21:41:08.6403071Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:41:08.6403334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 552, in forward 2025-08-14T21:41:08.6403416Z hidden_states = self.dense(hidden_states) 2025-08-14T21:41:08.6403419Z 2025-08-14T21:41:08.6403527Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6403726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6403799Z return mod(**inputs) 2025-08-14T21:41:08.6404042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1781, in forward 2025-08-14T21:41:08.6404140Z logits = self.qa_outputs(sequence_output) 2025-08-14T21:41:08.6404143Z 2025-08-14T21:41:08.6404249Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6404442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6404517Z return mod(**inputs) 2025-08-14T21:41:08.6404760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1799, in forward 2025-08-14T21:41:08.6404863Z start_loss = loss_fct(start_logits, start_positions) 2025-08-14T21:41:08.6404867Z 2025-08-14T21:41:08.6404971Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:08.6405398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:08.6405474Z return mod(**inputs) 2025-08-14T21:41:08.6405749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/bert/modeling_bert.py", line 1800, in forward 2025-08-14T21:41:08.6405847Z end_loss = loss_fct(end_logits, end_positions) 2025-08-14T21:41:08.6405852Z 2025-08-14T21:41:16.2350451Z Compilation time (from dynamo_timed): 13.903596338 2025-08-14T21:41:16.2355824Z pass 2025-08-14T21:41:16.2360324Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:41:16.2361151Z TIMING: _recursive_pre_grad_passes:0.00694 _recursive_joint_graph_passes:0.36817 _recursive_post_grad_passes:0.08837 async_compile.wait:0.00215 code_gen:6.56023 inductor_compile:7.7504 backend_compile:10.8843 gc:0.00015 entire_frame_compile:13.9036 total_wall_time:13.9036 2025-08-14T21:41:16.2362188Z STATS: call_* op count: 296 | FakeTensorMode.__torch_dispatch__:12371 | FakeTensor.__torch_dispatch__:4710 | ProxyTorchDispatchMode.__torch_dispatch__:4531 2025-08-14T21:41:16.2362688Z Dynamo produced 1 graphs covering 296 ops with 0 graph breaks (0 unique) 2025-08-14T21:41:21.3308778Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:41:21.3310895Z from pkg_resources import resource_filename 2025-08-14T21:41:21.9671920Z 2025-08-14T21:41:41.0608762Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:41:41.0609312Z loading model: 0it [00:19, ?it/s] 2025-08-14T21:41:41.0640293Z cpu eval BlenderbotForCausalLM 2025-08-14T21:41:41.2630579Z Compilation time (from dynamo_timed): 0 2025-08-14T21:41:41.2630889Z pass_due_to_skip 2025-08-14T21:41:41.2638711Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:41:41.2639054Z TIMING: total_wall_time:0 2025-08-14T21:41:41.2639282Z STATS: call_* op count: 0 2025-08-14T21:41:41.2639536Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-08-14T21:41:45.9128018Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:41:45.9129323Z from pkg_resources import resource_filename 2025-08-14T21:41:46.4914942Z 2025-08-14T21:41:47.3488698Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:41:47.3488997Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:41:47.3494935Z cpu eval BlenderbotSmallForCausalLM 2025-08-14T21:41:47.5135262Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:41:47.5658454Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:41:47.6149984Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:41:53.2898150Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.2904288Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.2908428Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.2912947Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.2919685Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.2922529Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.2922773Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.2923384Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.2923706Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.2924754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.2925168Z return mod(**inputs) 2025-08-14T21:41:53.2925780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.2926366Z outputs = self.model.decoder( 2025-08-14T21:41:53.2926858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.2927361Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.2927760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.2928169Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.2928661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.2929177Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.2929702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:41:53.2930285Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:53.2930510Z 2025-08-14T21:41:53.2930643Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.2931047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.2931444Z return mod(**inputs) 2025-08-14T21:41:53.2931909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.2932394Z outputs = self.model.decoder( 2025-08-14T21:41:53.2932878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.2933370Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.2933763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.2934165Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.2934664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.2935171Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.2935756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:41:53.2936242Z key_states = self.k_proj(current_states) 2025-08-14T21:41:53.2936392Z 2025-08-14T21:41:53.2936503Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.2936915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.2937261Z return mod(**inputs) 2025-08-14T21:41:53.2938080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.2938746Z outputs = self.model.decoder( 2025-08-14T21:41:53.2939216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.2939681Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.2940052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.2940434Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.2940901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.2941382Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.2941935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:41:53.2942426Z value_states = self.v_proj(current_states) 2025-08-14T21:41:53.2942575Z 2025-08-14T21:41:53.2942672Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.2942894Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.2943114Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.2943331Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.2943572Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.2943976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.2944335Z return mod(**inputs) 2025-08-14T21:41:53.2944781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.2945249Z outputs = self.model.decoder( 2025-08-14T21:41:53.2945709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.2946230Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.2946598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.2946974Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.2947433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.2947921Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.2948397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:41:53.2948885Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:53.2949354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:53.2949863Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:53.2950056Z 2025-08-14T21:41:53.2950165Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.2950543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.2950887Z return mod(**inputs) 2025-08-14T21:41:53.2951365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.2951823Z outputs = self.model.decoder( 2025-08-14T21:41:53.2952282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.2952744Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.2953107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.2953516Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.2953978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.2954470Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.2954986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:41:53.2955471Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:53.2955941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:53.2956423Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:53.2956596Z 2025-08-14T21:41:53.2956748Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.2957127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.2957474Z return mod(**inputs) 2025-08-14T21:41:53.2957914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.2958373Z outputs = self.model.decoder( 2025-08-14T21:41:53.2958833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.2959298Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.2959666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.2960043Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.2960512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.2961008Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.2961492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:41:53.2961964Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:53.2962113Z 2025-08-14T21:41:53.2962221Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.2962602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.2962940Z return mod(**inputs) 2025-08-14T21:41:53.2963378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.2963840Z outputs = self.model.decoder( 2025-08-14T21:41:53.2964298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.2964927Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.2965313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.2965706Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.2966194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:41:53.2966740Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:53.2966928Z 2025-08-14T21:41:53.2967040Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.2967416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.2967755Z return mod(**inputs) 2025-08-14T21:41:53.2968207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.2968711Z outputs = self.model.decoder( 2025-08-14T21:41:53.2969177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.2969644Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.2970022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.2970415Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.2970896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:41:53.2971419Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:53.2971920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:53.2972303Z return self.act(input) 2025-08-14T21:41:53.2972425Z 2025-08-14T21:41:53.2972537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.2972935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.2973309Z return mod(**inputs) 2025-08-14T21:41:53.2973770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.2974258Z outputs = self.model.decoder( 2025-08-14T21:41:53.2974741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.2975235Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.2975612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.2976018Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.2976464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:41:53.2976922Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:53.2977058Z 2025-08-14T21:41:53.2977163Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.2977547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.2977892Z return mod(**inputs) 2025-08-14T21:41:53.2978340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.2978812Z outputs = self.model.decoder( 2025-08-14T21:41:53.2979282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.2979763Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.2980130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.2980514Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.2980991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.2981490Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.2981996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:41:53.2982516Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:53.2982727Z 2025-08-14T21:41:53.2982830Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.2983188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.2983504Z return mod(**inputs) 2025-08-14T21:41:53.2983944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.2984380Z outputs = self.model.decoder( 2025-08-14T21:41:53.2984806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.2985236Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.2985581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.2985937Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.2986365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.2986875Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.2987337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:41:53.2987787Z key_states = self.k_proj(current_states) 2025-08-14T21:41:53.2987926Z 2025-08-14T21:41:53.2988030Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.2988386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.2988716Z return mod(**inputs) 2025-08-14T21:41:53.2989132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.2989578Z outputs = self.model.decoder( 2025-08-14T21:41:53.2990046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.2990512Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.2990857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.2991218Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.2991663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.2992128Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.2992586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:41:53.2993049Z value_states = self.v_proj(current_states) 2025-08-14T21:41:53.2993189Z 2025-08-14T21:41:53.2993279Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.2993496Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.2993702Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.2993913Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.2994146Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.2994504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.2994834Z return mod(**inputs) 2025-08-14T21:41:53.2995252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.2995691Z outputs = self.model.decoder( 2025-08-14T21:41:53.2996154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.2996591Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.2996941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.2997293Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.2997731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.2998240Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.2998742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:41:53.2999235Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:53.2999713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:53.3000228Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:53.3000421Z 2025-08-14T21:41:53.3000536Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3000910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3001291Z return mod(**inputs) 2025-08-14T21:41:53.3001734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3002201Z outputs = self.model.decoder( 2025-08-14T21:41:53.3002660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3003121Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3003491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3003863Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3004334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3004972Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3005485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:41:53.3005982Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:53.3006446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:53.3006932Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:53.3007109Z 2025-08-14T21:41:53.3007219Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3007594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3007951Z return mod(**inputs) 2025-08-14T21:41:53.3008388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3008890Z outputs = self.model.decoder( 2025-08-14T21:41:53.3009357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3009833Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3010205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3010586Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3011060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3011618Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3012095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:41:53.3012571Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:53.3012723Z 2025-08-14T21:41:53.3012836Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3013216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3013581Z return mod(**inputs) 2025-08-14T21:41:53.3014019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3014472Z outputs = self.model.decoder( 2025-08-14T21:41:53.3014902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3015340Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3015688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3016048Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3016520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:41:53.3017010Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:53.3017191Z 2025-08-14T21:41:53.3017300Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3017680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3018017Z return mod(**inputs) 2025-08-14T21:41:53.3018459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3018928Z outputs = self.model.decoder( 2025-08-14T21:41:53.3019387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3019843Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3020213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3020591Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3021023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:41:53.3021504Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:53.3021886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:53.3022228Z return self.act(input) 2025-08-14T21:41:53.3022335Z 2025-08-14T21:41:53.3022436Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3022798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3023121Z return mod(**inputs) 2025-08-14T21:41:53.3023526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3023962Z outputs = self.model.decoder( 2025-08-14T21:41:53.3024391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3024826Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3025162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3025551Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3025994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:41:53.3026445Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:53.3026586Z 2025-08-14T21:41:53.3026688Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3027049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3027400Z return mod(**inputs) 2025-08-14T21:41:53.3027890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3028369Z outputs = self.model.decoder( 2025-08-14T21:41:53.3028802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3029252Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3029601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3030076Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3030530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3031055Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3031517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:41:53.3032035Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:53.3032238Z 2025-08-14T21:41:53.3032348Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3032721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3033039Z return mod(**inputs) 2025-08-14T21:41:53.3033447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3033883Z outputs = self.model.decoder( 2025-08-14T21:41:53.3034308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3034751Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3035092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3035453Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3035878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3036339Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3036796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:41:53.3037268Z key_states = self.k_proj(current_states) 2025-08-14T21:41:53.3037410Z 2025-08-14T21:41:53.3037519Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3038119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3038466Z return mod(**inputs) 2025-08-14T21:41:53.3038900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3039379Z outputs = self.model.decoder( 2025-08-14T21:41:53.3039853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3040410Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3040769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3041152Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3041614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3042101Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3042576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:41:53.3043095Z value_states = self.v_proj(current_states) 2025-08-14T21:41:53.3043242Z 2025-08-14T21:41:53.3043336Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3043553Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3043773Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3043992Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3044237Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3044762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3045152Z return mod(**inputs) 2025-08-14T21:41:53.3045618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3046162Z outputs = self.model.decoder( 2025-08-14T21:41:53.3046623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3047087Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3047455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3047831Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3048298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3048796Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3049269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:41:53.3049767Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:53.3050240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:53.3050747Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:53.3050946Z 2025-08-14T21:41:53.3051057Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3051454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3051817Z return mod(**inputs) 2025-08-14T21:41:53.3052258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3052710Z outputs = self.model.decoder( 2025-08-14T21:41:53.3053166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3053641Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3053983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3054343Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3054776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3055234Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3055729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:41:53.3056191Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:53.3056634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:53.3057093Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:53.3057257Z 2025-08-14T21:41:53.3057366Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3057776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3058120Z return mod(**inputs) 2025-08-14T21:41:53.3058563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3059002Z outputs = self.model.decoder( 2025-08-14T21:41:53.3059430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3059868Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3060207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3060569Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3061046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3061507Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3061957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:41:53.3062399Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:53.3062530Z 2025-08-14T21:41:53.3062641Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3062994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3063307Z return mod(**inputs) 2025-08-14T21:41:53.3063720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3064172Z outputs = self.model.decoder( 2025-08-14T21:41:53.3064620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3065084Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3065453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3065808Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3066241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:41:53.3066727Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:53.3066897Z 2025-08-14T21:41:53.3067008Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3067365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3067687Z return mod(**inputs) 2025-08-14T21:41:53.3068124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3068586Z outputs = self.model.decoder( 2025-08-14T21:41:53.3069002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3069440Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3069780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3070162Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3070592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:41:53.3071072Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:53.3071466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:53.3071799Z return self.act(input) 2025-08-14T21:41:53.3071932Z 2025-08-14T21:41:53.3072031Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3072380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3072696Z return mod(**inputs) 2025-08-14T21:41:53.3073091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3073518Z outputs = self.model.decoder( 2025-08-14T21:41:53.3073937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3074366Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3074733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3075100Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3075554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:41:53.3075988Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:53.3076131Z 2025-08-14T21:41:53.3076235Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3076586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3076915Z return mod(**inputs) 2025-08-14T21:41:53.3077333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3077804Z outputs = self.model.decoder( 2025-08-14T21:41:53.3078266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3078733Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3079106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3079467Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3079908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3080377Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3080876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:41:53.3081419Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:53.3081634Z 2025-08-14T21:41:53.3081751Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3082126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3082475Z return mod(**inputs) 2025-08-14T21:41:53.3082919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3083384Z outputs = self.model.decoder( 2025-08-14T21:41:53.3083836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3084349Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3084817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3085206Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3085689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3086188Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3086677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:41:53.3087153Z key_states = self.k_proj(current_states) 2025-08-14T21:41:53.3087291Z 2025-08-14T21:41:53.3087394Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3087744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3088075Z return mod(**inputs) 2025-08-14T21:41:53.3088500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3088966Z outputs = self.model.decoder( 2025-08-14T21:41:53.3089464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3089934Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3090303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3090685Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3091150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3091632Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3092116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:41:53.3092593Z value_states = self.v_proj(current_states) 2025-08-14T21:41:53.3092741Z 2025-08-14T21:41:53.3092833Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3093080Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3093304Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3093522Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3093758Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3094135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3094485Z return mod(**inputs) 2025-08-14T21:41:53.3094922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3095383Z outputs = self.model.decoder( 2025-08-14T21:41:53.3095837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3096300Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3096656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3097037Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3097506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3097968Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3098414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:41:53.3098881Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:53.3099335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:53.3099797Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:53.3099973Z 2025-08-14T21:41:53.3100071Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3100414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3100726Z return mod(**inputs) 2025-08-14T21:41:53.3101123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3101579Z outputs = self.model.decoder( 2025-08-14T21:41:53.3102004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3102435Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3102770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3103121Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3103552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3104047Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3104490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:41:53.3104947Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:53.3105397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:53.3105838Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:53.3105994Z 2025-08-14T21:41:53.3106094Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3106437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3106756Z return mod(**inputs) 2025-08-14T21:41:53.3107154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3107594Z outputs = self.model.decoder( 2025-08-14T21:41:53.3108016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3108444Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3108791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3109141Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3109566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3110017Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3110460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:41:53.3110900Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:53.3111032Z 2025-08-14T21:41:53.3111139Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3111482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3111796Z return mod(**inputs) 2025-08-14T21:41:53.3112190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3112607Z outputs = self.model.decoder( 2025-08-14T21:41:53.3113037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3113459Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3113786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3114125Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3114547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:41:53.3115038Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:53.3115205Z 2025-08-14T21:41:53.3115312Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3115649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3115965Z return mod(**inputs) 2025-08-14T21:41:53.3116365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3116783Z outputs = self.model.decoder( 2025-08-14T21:41:53.3117188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3117603Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3117976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3118321Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3118750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:41:53.3119224Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:53.3119616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:53.3119962Z return self.act(input) 2025-08-14T21:41:53.3120082Z 2025-08-14T21:41:53.3120186Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3120557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3120911Z return mod(**inputs) 2025-08-14T21:41:53.3121349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3121794Z outputs = self.model.decoder( 2025-08-14T21:41:53.3122233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3122671Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3123044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3123432Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3123904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:41:53.3124373Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:53.3124612Z 2025-08-14T21:41:53.3124729Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3125124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3125477Z return mod(**inputs) 2025-08-14T21:41:53.3125912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3126353Z outputs = self.model.decoder( 2025-08-14T21:41:53.3126782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3127258Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3127630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3128015Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3128486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3128998Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3129459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:41:53.3129998Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:53.3130201Z 2025-08-14T21:41:53.3130313Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3130658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3130981Z return mod(**inputs) 2025-08-14T21:41:53.3131404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3131870Z outputs = self.model.decoder( 2025-08-14T21:41:53.3133138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3133623Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3133975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3134329Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3134769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3135230Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3135689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:41:53.3136126Z key_states = self.k_proj(current_states) 2025-08-14T21:41:53.3136265Z 2025-08-14T21:41:53.3136368Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3136730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3137047Z return mod(**inputs) 2025-08-14T21:41:53.3137460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3138076Z outputs = self.model.decoder( 2025-08-14T21:41:53.3138518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3138961Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3139309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3139677Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3140122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3140585Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3141049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:41:53.3141509Z value_states = self.v_proj(current_states) 2025-08-14T21:41:53.3141650Z 2025-08-14T21:41:53.3141739Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3141951Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3142165Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3142445Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3142668Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3143027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3143356Z return mod(**inputs) 2025-08-14T21:41:53.3143771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3144210Z outputs = self.model.decoder( 2025-08-14T21:41:53.3144713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3145155Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3145515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3145892Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3146359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3146847Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3147319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:41:53.3147893Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:53.3148363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:53.3148860Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:53.3149061Z 2025-08-14T21:41:53.3149169Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3149540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3149887Z return mod(**inputs) 2025-08-14T21:41:53.3150315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3150788Z outputs = self.model.decoder( 2025-08-14T21:41:53.3151250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3151720Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3152079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3152464Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3152924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3153399Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3153883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:41:53.3154367Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:53.3154825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:53.3155302Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:53.3155476Z 2025-08-14T21:41:53.3155582Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3155959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3156300Z return mod(**inputs) 2025-08-14T21:41:53.3156725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3157228Z outputs = self.model.decoder( 2025-08-14T21:41:53.3157695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3158158Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3158534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3158928Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3159401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3159925Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3160434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:41:53.3160930Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:53.3161078Z 2025-08-14T21:41:53.3161197Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3161578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3161935Z return mod(**inputs) 2025-08-14T21:41:53.3162387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3162893Z outputs = self.model.decoder( 2025-08-14T21:41:53.3163370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3163852Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3164231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3164692Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3165174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:41:53.3165711Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:53.3165897Z 2025-08-14T21:41:53.3166015Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3166397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3166755Z return mod(**inputs) 2025-08-14T21:41:53.3167204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3167674Z outputs = self.model.decoder( 2025-08-14T21:41:53.3168146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3168628Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3169004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3169393Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3169865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:41:53.3170396Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:53.3170823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:53.3171186Z return self.act(input) 2025-08-14T21:41:53.3171310Z 2025-08-14T21:41:53.3171418Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3171801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3172144Z return mod(**inputs) 2025-08-14T21:41:53.3172593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3173099Z outputs = self.model.decoder( 2025-08-14T21:41:53.3173563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3174030Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3174406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3174794Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3175302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:41:53.3175764Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:53.3175912Z 2025-08-14T21:41:53.3176020Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3176394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3176729Z return mod(**inputs) 2025-08-14T21:41:53.3177165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3177630Z outputs = self.model.decoder( 2025-08-14T21:41:53.3178119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3178577Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3178943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3179322Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3179774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3180264Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3180748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:41:53.3181295Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:53.3181508Z 2025-08-14T21:41:53.3181619Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3181999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3182342Z return mod(**inputs) 2025-08-14T21:41:53.3182780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3183239Z outputs = self.model.decoder( 2025-08-14T21:41:53.3183699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3184166Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3184534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3184887Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3185329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3185790Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3186238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:41:53.3186693Z key_states = self.k_proj(current_states) 2025-08-14T21:41:53.3186835Z 2025-08-14T21:41:53.3186938Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3187293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3187636Z return mod(**inputs) 2025-08-14T21:41:53.3188070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3188533Z outputs = self.model.decoder( 2025-08-14T21:41:53.3188987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3189444Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3189828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3190206Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3190657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3191147Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3191691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:41:53.3192169Z value_states = self.v_proj(current_states) 2025-08-14T21:41:53.3192316Z 2025-08-14T21:41:53.3192403Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3192634Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3192919Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3193140Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3193399Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3193783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3194136Z return mod(**inputs) 2025-08-14T21:41:53.3194578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3195053Z outputs = self.model.decoder( 2025-08-14T21:41:53.3195516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3195994Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3196365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3196760Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3197232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3197732Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3198213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:41:53.3198698Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:53.3199174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:53.3199687Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:53.3199893Z 2025-08-14T21:41:53.3200002Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3200391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3200744Z return mod(**inputs) 2025-08-14T21:41:53.3201194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3201667Z outputs = self.model.decoder( 2025-08-14T21:41:53.3202131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3202628Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3203010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3203400Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3203873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3204371Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3204949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:41:53.3205495Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:53.3205972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:53.3206463Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:53.3206649Z 2025-08-14T21:41:53.3206761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3207151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3207536Z return mod(**inputs) 2025-08-14T21:41:53.3208046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3208534Z outputs = self.model.decoder( 2025-08-14T21:41:53.3209030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3209520Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3209913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3210317Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3210814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3211328Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3211841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:41:53.3212346Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:53.3212495Z 2025-08-14T21:41:53.3212617Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3213008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3213376Z return mod(**inputs) 2025-08-14T21:41:53.3213845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3214335Z outputs = self.model.decoder( 2025-08-14T21:41:53.3214827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3215316Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3215706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3216104Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3216600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:41:53.3217145Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:53.3217338Z 2025-08-14T21:41:53.3217461Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3217849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3218237Z return mod(**inputs) 2025-08-14T21:41:53.3218673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3219130Z outputs = self.model.decoder( 2025-08-14T21:41:53.3219584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3220049Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3220415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3220805Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3221265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:41:53.3221769Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:53.3222172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:53.3222530Z return self.act(input) 2025-08-14T21:41:53.3222650Z 2025-08-14T21:41:53.3222758Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3223135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3223470Z return mod(**inputs) 2025-08-14T21:41:53.3223936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3224407Z outputs = self.model.decoder( 2025-08-14T21:41:53.3224859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3225315Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3225678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3226055Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3226510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:41:53.3226981Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:53.3227131Z 2025-08-14T21:41:53.3227241Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3227613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3227946Z return mod(**inputs) 2025-08-14T21:41:53.3228377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3228839Z outputs = self.model.decoder( 2025-08-14T21:41:53.3229281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3229747Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3230108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3230488Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3230943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3231430Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3231910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:41:53.3232445Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:53.3232658Z 2025-08-14T21:41:53.3232766Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3233169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3233511Z return mod(**inputs) 2025-08-14T21:41:53.3233948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3234407Z outputs = self.model.decoder( 2025-08-14T21:41:53.3234869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3235357Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3235711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3236092Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3236558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3237070Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3237544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:41:53.3238206Z key_states = self.k_proj(current_states) 2025-08-14T21:41:53.3238354Z 2025-08-14T21:41:53.3238474Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3238935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3239287Z return mod(**inputs) 2025-08-14T21:41:53.3239736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3240217Z outputs = self.model.decoder( 2025-08-14T21:41:53.3240678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3241161Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3241538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3241929Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3242406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3242912Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3243416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:41:53.3243911Z value_states = self.v_proj(current_states) 2025-08-14T21:41:53.3244062Z 2025-08-14T21:41:53.3244151Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3244382Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3244679Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3244902Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3245153Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3245539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3245881Z return mod(**inputs) 2025-08-14T21:41:53.3246341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3246821Z outputs = self.model.decoder( 2025-08-14T21:41:53.3247297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3247768Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3248151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3248588Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3249072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3249584Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3250090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:41:53.3250599Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:53.3251128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:53.3251643Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:53.3251851Z 2025-08-14T21:41:53.3251964Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3252349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3252715Z return mod(**inputs) 2025-08-14T21:41:53.3253172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3253659Z outputs = self.model.decoder( 2025-08-14T21:41:53.3254175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3254658Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3255037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3255439Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3255919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3256418Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3256907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:41:53.3257396Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:53.3257852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:53.3258341Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:53.3258514Z 2025-08-14T21:41:53.3258622Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3259006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3259355Z return mod(**inputs) 2025-08-14T21:41:53.3259787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3260252Z outputs = self.model.decoder( 2025-08-14T21:41:53.3260703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3261168Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3261544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3261922Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3262377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3262866Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3263356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:41:53.3263824Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:53.3263995Z 2025-08-14T21:41:53.3264103Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3264468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3264796Z return mod(**inputs) 2025-08-14T21:41:53.3265211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3265644Z outputs = self.model.decoder( 2025-08-14T21:41:53.3266084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3266552Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3266893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3267254Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3267727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:41:53.3268246Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:53.3268426Z 2025-08-14T21:41:53.3268533Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3268949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3269295Z return mod(**inputs) 2025-08-14T21:41:53.3269711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3270140Z outputs = self.model.decoder( 2025-08-14T21:41:53.3270568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3271002Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3271338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3271702Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3272157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:41:53.3272665Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:53.3273067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:53.3273432Z return self.act(input) 2025-08-14T21:41:53.3273539Z 2025-08-14T21:41:53.3273646Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3274001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3274318Z return mod(**inputs) 2025-08-14T21:41:53.3274729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3275168Z outputs = self.model.decoder( 2025-08-14T21:41:53.3275587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3276021Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3276366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3276723Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3277153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:41:53.3277620Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:53.3277762Z 2025-08-14T21:41:53.3277878Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3278277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3278611Z return mod(**inputs) 2025-08-14T21:41:53.3279048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3279512Z outputs = self.model.decoder( 2025-08-14T21:41:53.3279959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3280444Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3280810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3281192Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3281645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3282136Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3282616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:41:53.3283154Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:41:53.3283369Z 2025-08-14T21:41:53.3283511Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3283890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3284236Z return mod(**inputs) 2025-08-14T21:41:53.3299912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3300569Z outputs = self.model.decoder( 2025-08-14T21:41:53.3301159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3301658Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3302035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3302429Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3302912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3303413Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3303906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:41:53.3304381Z key_states = self.k_proj(current_states) 2025-08-14T21:41:53.3304527Z 2025-08-14T21:41:53.3304656Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3305048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3305396Z return mod(**inputs) 2025-08-14T21:41:53.3305844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3306321Z outputs = self.model.decoder( 2025-08-14T21:41:53.3306782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3307254Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3307628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3308014Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3308472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3309081Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3309565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:41:53.3310047Z value_states = self.v_proj(current_states) 2025-08-14T21:41:53.3310197Z 2025-08-14T21:41:53.3310286Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3310524Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3310744Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3310992Z cudagraph partition due to non gpu ops 2025-08-14T21:41:53.3311238Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3311621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3311964Z return mod(**inputs) 2025-08-14T21:41:53.3312418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3312889Z outputs = self.model.decoder( 2025-08-14T21:41:53.3313340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3313804Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3314238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3314616Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3315081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3315566Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3316051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:41:53.3316540Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:53.3317006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:41:53.3317533Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:41:53.3317733Z 2025-08-14T21:41:53.3317855Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3318237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3318595Z return mod(**inputs) 2025-08-14T21:41:53.3319049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3319523Z outputs = self.model.decoder( 2025-08-14T21:41:53.3319989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3320468Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3320845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3321225Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3321701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3322197Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3322694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:41:53.3323184Z attn_output, attn_weights = attention_interface( 2025-08-14T21:41:53.3323659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:41:53.3324195Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:41:53.3324366Z 2025-08-14T21:41:53.3324582Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3324977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3325332Z return mod(**inputs) 2025-08-14T21:41:53.3325793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3326272Z outputs = self.model.decoder( 2025-08-14T21:41:53.3326769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3327252Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3327632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3328026Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3328505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:41:53.3329010Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:41:53.3329511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:41:53.3330027Z attn_output = self.out_proj(attn_output) 2025-08-14T21:41:53.3330187Z 2025-08-14T21:41:53.3330304Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3330699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3331052Z return mod(**inputs) 2025-08-14T21:41:53.3331507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3331988Z outputs = self.model.decoder( 2025-08-14T21:41:53.3332458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3332932Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3333304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3333686Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3334146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:41:53.3334652Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:53.3334845Z 2025-08-14T21:41:53.3334954Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3335329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3335666Z return mod(**inputs) 2025-08-14T21:41:53.3336103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3336570Z outputs = self.model.decoder( 2025-08-14T21:41:53.3337025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3337485Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3338028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3338427Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3338886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:41:53.3339406Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:41:53.3339901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:41:53.3340266Z return self.act(input) 2025-08-14T21:41:53.3340381Z 2025-08-14T21:41:53.3340489Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3340863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3341210Z return mod(**inputs) 2025-08-14T21:41:53.3341650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1512, in forward 2025-08-14T21:41:53.3342143Z outputs = self.model.decoder( 2025-08-14T21:41:53.3342603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:41:53.3343068Z layer_outputs = decoder_layer( 2025-08-14T21:41:53.3343430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:41:53.3343816Z return super().__call__(*args, **kwargs) 2025-08-14T21:41:53.3344285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:41:53.3344763Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:41:53.3344909Z 2025-08-14T21:41:53.3345074Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3345458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3345799Z return mod(**inputs) 2025-08-14T21:41:53.3346234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1528, in forward 2025-08-14T21:41:53.3346694Z logits = self.lm_head(outputs[0]) 2025-08-14T21:41:53.3346838Z 2025-08-14T21:41:53.3346947Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:41:53.3347321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:41:53.3347653Z return mod(**inputs) 2025-08-14T21:41:53.3348092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1534, in forward 2025-08-14T21:41:53.3348637Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:41:53.3348843Z 2025-08-14T21:42:00.8222021Z Compilation time (from dynamo_timed): 12.05949685 2025-08-14T21:42:00.8249731Z pass 2025-08-14T21:42:00.8250291Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:42:00.8255398Z TIMING: _recursive_pre_grad_passes:0.00618 _recursive_joint_graph_passes:0.29408 _recursive_post_grad_passes:0.34507 async_compile.wait:0.74612 code_gen:7.21233 inductor_compile:8.36656 backend_compile:10.49501 gc:0.00162 entire_frame_compile:12.0595 total_wall_time:12.0595 2025-08-14T21:42:00.8256479Z STATS: call_* op count: 252 | FakeTensorMode.__torch_dispatch__:9096 | FakeTensor.__torch_dispatch__:3327 | ProxyTorchDispatchMode.__torch_dispatch__:3279 2025-08-14T21:42:00.8257030Z Dynamo produced 1 graphs covering 252 ops with 0 graph breaks (0 unique) 2025-08-14T21:42:05.9822246Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:42:05.9823173Z from pkg_resources import resource_filename 2025-08-14T21:42:06.6115825Z 2025-08-14T21:42:07.8795012Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:42:07.8795484Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:42:07.8816974Z cpu eval BlenderbotSmallForConditionalGeneration 2025-08-14T21:42:08.1634650Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:42:08.2661487Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:42:08.3657066Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:42:20.3763955Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3764356Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3764595Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3765162Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3765386Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3765611Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3765831Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3766045Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3766301Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3766740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3767086Z return mod(**inputs) 2025-08-14T21:42:20.3767566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3768052Z outputs = self.model( 2025-08-14T21:42:20.3768655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3769161Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3769643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3770119Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3770502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3770891Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3771369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3771861Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3772449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.3773000Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.3773243Z 2025-08-14T21:42:20.3773358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3773754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3774118Z return mod(**inputs) 2025-08-14T21:42:20.3774560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3775032Z outputs = self.model( 2025-08-14T21:42:20.3775482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3775951Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3776405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3776870Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3777242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3777631Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3778097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3778691Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3779169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.3779629Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.3779779Z 2025-08-14T21:42:20.3779888Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3780283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3780646Z return mod(**inputs) 2025-08-14T21:42:20.3781123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3781584Z outputs = self.model( 2025-08-14T21:42:20.3782018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3782485Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3782930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3783388Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3783761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3784170Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3784636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3785131Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3785612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.3786089Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.3786248Z 2025-08-14T21:42:20.3786333Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3786559Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3786777Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3786985Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3787233Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3787622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3787962Z return mod(**inputs) 2025-08-14T21:42:20.3788409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3788882Z outputs = self.model( 2025-08-14T21:42:20.3789319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3789795Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3790263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3790721Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3791074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3791460Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3791932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3792433Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3792933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.3793433Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.3793910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.3794436Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.3794635Z 2025-08-14T21:42:20.3794745Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3795137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3795490Z return mod(**inputs) 2025-08-14T21:42:20.3795920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3796399Z outputs = self.model( 2025-08-14T21:42:20.3796843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3797321Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3797789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3798272Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3798651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3799040Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3799551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3800051Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3800540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.3801049Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.3801520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.3802031Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.3802207Z 2025-08-14T21:42:20.3802328Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3802706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3803057Z return mod(**inputs) 2025-08-14T21:42:20.3803515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3804133Z outputs = self.model( 2025-08-14T21:42:20.3804595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3805094Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3805564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3806059Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3806442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3806848Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3807339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3807834Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3808347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.3808842Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.3808992Z 2025-08-14T21:42:20.3809112Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3809519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3809891Z return mod(**inputs) 2025-08-14T21:42:20.3810345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3810817Z outputs = self.model( 2025-08-14T21:42:20.3811265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3811749Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3812240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3812702Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3813080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3813471Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3813946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:42:20.3814475Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.3814674Z 2025-08-14T21:42:20.3814787Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3815206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3815561Z return mod(**inputs) 2025-08-14T21:42:20.3816006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3816480Z outputs = self.model( 2025-08-14T21:42:20.3816925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3817406Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3817883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3818381Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3818749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3819139Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3819613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:42:20.3820122Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.3820528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:20.3820897Z return self.act(input) 2025-08-14T21:42:20.3821019Z 2025-08-14T21:42:20.3821127Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3821506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3821850Z return mod(**inputs) 2025-08-14T21:42:20.3822285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3822748Z outputs = self.model( 2025-08-14T21:42:20.3823196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3823681Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3824145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3824615Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3824987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3825398Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3825856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-14T21:42:20.3826339Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:42:20.3826484Z 2025-08-14T21:42:20.3826593Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3826974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3827345Z return mod(**inputs) 2025-08-14T21:42:20.3827786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3828239Z outputs = self.model( 2025-08-14T21:42:20.3828681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3829148Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3829750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3830216Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3830628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3831017Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3831482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3831984Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3832483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.3833041Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.3833265Z 2025-08-14T21:42:20.3833375Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3833771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3834120Z return mod(**inputs) 2025-08-14T21:42:20.3834560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3835015Z outputs = self.model( 2025-08-14T21:42:20.3835452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3835925Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3836377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3836845Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3837207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3837589Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3838256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3838749Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3839250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.3839719Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.3839859Z 2025-08-14T21:42:20.3839975Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3840350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3840765Z return mod(**inputs) 2025-08-14T21:42:20.3841224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3841706Z outputs = self.model( 2025-08-14T21:42:20.3842151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3842640Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3843135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3843610Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3844091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3844492Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3844975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3845479Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3845962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.3846509Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.3846666Z 2025-08-14T21:42:20.3846767Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3846993Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3847220Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3847444Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3847694Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3848075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3848447Z return mod(**inputs) 2025-08-14T21:42:20.3848904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3849372Z outputs = self.model( 2025-08-14T21:42:20.3849831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3850367Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3850840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3851310Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3851687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3852079Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3852556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3853039Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3853520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.3854008Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.3854465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.3854987Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.3855196Z 2025-08-14T21:42:20.3855309Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3855707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3856087Z return mod(**inputs) 2025-08-14T21:42:20.3856532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3856995Z outputs = self.model( 2025-08-14T21:42:20.3857436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3857899Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3858354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3858840Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3859200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3859585Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3860052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3860539Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3861009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.3861494Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.3861991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.3862474Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.3862645Z 2025-08-14T21:42:20.3862753Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3863133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3863476Z return mod(**inputs) 2025-08-14T21:42:20.3863918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3864390Z outputs = self.model( 2025-08-14T21:42:20.3864832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3865310Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3865769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3866246Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3866621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3867009Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3867471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3867964Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3868450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.3868919Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.3869071Z 2025-08-14T21:42:20.3869182Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3869566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3869913Z return mod(**inputs) 2025-08-14T21:42:20.3870352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3870821Z outputs = self.model( 2025-08-14T21:42:20.3871265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3871752Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3872197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3872657Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3873028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3873400Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3874360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:42:20.3874876Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.3875058Z 2025-08-14T21:42:20.3875175Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3875549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3875897Z return mod(**inputs) 2025-08-14T21:42:20.3876334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3876793Z outputs = self.model( 2025-08-14T21:42:20.3877256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3877727Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3878182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3878639Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3879001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3879380Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3879838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:42:20.3880345Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.3880756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:20.3881114Z return self.act(input) 2025-08-14T21:42:20.3881230Z 2025-08-14T21:42:20.3881347Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3881719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3882058Z return mod(**inputs) 2025-08-14T21:42:20.3882493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3882944Z outputs = self.model( 2025-08-14T21:42:20.3883380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3883964Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3884438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3884920Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3885302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3885709Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3886179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-14T21:42:20.3886648Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:42:20.3886803Z 2025-08-14T21:42:20.3886948Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3887337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3887691Z return mod(**inputs) 2025-08-14T21:42:20.3888160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3888636Z outputs = self.model( 2025-08-14T21:42:20.3889087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3889578Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3890045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3890515Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3890892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3891275Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3891749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3892237Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3892770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.3893333Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.3893561Z 2025-08-14T21:42:20.3893672Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3894055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3894418Z return mod(**inputs) 2025-08-14T21:42:20.3894867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3895337Z outputs = self.model( 2025-08-14T21:42:20.3895783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3896256Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3896727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3897198Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3897555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3897915Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3898348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3898799Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3899246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.3899712Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.3899859Z 2025-08-14T21:42:20.3899971Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3900346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3900679Z return mod(**inputs) 2025-08-14T21:42:20.3901116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3901574Z outputs = self.model( 2025-08-14T21:42:20.3901999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3902493Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3902950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3903410Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3903774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3904151Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3904630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3905110Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3905575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.3906051Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.3906197Z 2025-08-14T21:42:20.3906290Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3906510Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3906733Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3906950Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3907195Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3907591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3907936Z return mod(**inputs) 2025-08-14T21:42:20.3908373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3908825Z outputs = self.model( 2025-08-14T21:42:20.3909261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3909730Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3910188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3910638Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3910989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3911351Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3911787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3912254Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3912727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.3913212Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.3913666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.3914168Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.3914368Z 2025-08-14T21:42:20.3914478Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3914859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3915196Z return mod(**inputs) 2025-08-14T21:42:20.3915637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3916100Z outputs = self.model( 2025-08-14T21:42:20.3916540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3917035Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3917491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3917951Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3918308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3918689Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3919148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3919641Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3920112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.3920598Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.3921063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.3921542Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.3921712Z 2025-08-14T21:42:20.3921820Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3922235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3922582Z return mod(**inputs) 2025-08-14T21:42:20.3923008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3923469Z outputs = self.model( 2025-08-14T21:42:20.3924032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3924518Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3924976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3925449Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3925821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3926206Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3926671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3927159Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3927641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.3928085Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.3928231Z 2025-08-14T21:42:20.3928335Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3928692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3929020Z return mod(**inputs) 2025-08-14T21:42:20.3929438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3929877Z outputs = self.model( 2025-08-14T21:42:20.3930300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3930746Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3931175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3931615Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3931974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3932385Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3932848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:42:20.3933352Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.3933536Z 2025-08-14T21:42:20.3933655Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3934018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3934363Z return mod(**inputs) 2025-08-14T21:42:20.3934777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3935218Z outputs = self.model( 2025-08-14T21:42:20.3935627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3936071Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3936509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3936940Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3937341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3937847Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3938301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:42:20.3938777Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.3939165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:20.3939510Z return self.act(input) 2025-08-14T21:42:20.3939619Z 2025-08-14T21:42:20.3939731Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3940085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3940408Z return mod(**inputs) 2025-08-14T21:42:20.3940823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3941251Z outputs = self.model( 2025-08-14T21:42:20.3941667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3942101Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3942525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3942952Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3943299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3943657Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3944100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-14T21:42:20.3944517Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:42:20.3944652Z 2025-08-14T21:42:20.3944749Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3945091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3945394Z return mod(**inputs) 2025-08-14T21:42:20.3945785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3946256Z outputs = self.model( 2025-08-14T21:42:20.3946645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3947052Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3947470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3947896Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3948226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3948612Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3949139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3949581Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3950012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.3950521Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.3950730Z 2025-08-14T21:42:20.3950832Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3951183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3951558Z return mod(**inputs) 2025-08-14T21:42:20.3951961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3952381Z outputs = self.model( 2025-08-14T21:42:20.3952785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3953198Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3953616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3954042Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3954372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3954736Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3955176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3955634Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3956079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.3956522Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.3956657Z 2025-08-14T21:42:20.3956769Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3957127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3957444Z return mod(**inputs) 2025-08-14T21:42:20.3957857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3958290Z outputs = self.model( 2025-08-14T21:42:20.3958697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3959140Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3959565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3959997Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3960333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3960715Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3961148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3961628Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3962098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.3962576Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.3962743Z 2025-08-14T21:42:20.3962838Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3963054Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3963271Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3963489Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.3963810Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3964250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3964609Z return mod(**inputs) 2025-08-14T21:42:20.3965069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3965500Z outputs = self.model( 2025-08-14T21:42:20.3965965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3966411Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3966846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3967277Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3967621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3967984Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3968414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3968869Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3969320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.3969778Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.3970208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.3970683Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.3970870Z 2025-08-14T21:42:20.3970973Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3971331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3971652Z return mod(**inputs) 2025-08-14T21:42:20.3972065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3972503Z outputs = self.model( 2025-08-14T21:42:20.3972934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3973392Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3973821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3974253Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3974591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3974980Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3975416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3975869Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3976300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.3976742Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.3977163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.3977629Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.3977785Z 2025-08-14T21:42:20.3977886Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3978233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3978548Z return mod(**inputs) 2025-08-14T21:42:20.3978939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3979362Z outputs = self.model( 2025-08-14T21:42:20.3979797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3980228Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3980648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3981076Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3981418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3981774Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3982203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.3982655Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.3983100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.3983538Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.3983681Z 2025-08-14T21:42:20.3983785Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3984146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3984471Z return mod(**inputs) 2025-08-14T21:42:20.3984873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3985304Z outputs = self.model( 2025-08-14T21:42:20.3985712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3986153Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3986588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3987022Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3987361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3987711Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3988142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:42:20.3988621Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.3988789Z 2025-08-14T21:42:20.3988917Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3989252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3989567Z return mod(**inputs) 2025-08-14T21:42:20.3989965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3990390Z outputs = self.model( 2025-08-14T21:42:20.3990785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3991234Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3991650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3992067Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3992408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3992758Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3993179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:42:20.3993634Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.3994037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:20.3994375Z return self.act(input) 2025-08-14T21:42:20.3994479Z 2025-08-14T21:42:20.3994594Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3994926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.3995230Z return mod(**inputs) 2025-08-14T21:42:20.3995626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.3996032Z outputs = self.model( 2025-08-14T21:42:20.3996422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.3996836Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.3997241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.3997642Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.3997969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.3998306Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.3998711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-14T21:42:20.3999145Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:42:20.3999282Z 2025-08-14T21:42:20.3999382Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.3999729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4000037Z return mod(**inputs) 2025-08-14T21:42:20.4000441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4000860Z outputs = self.model( 2025-08-14T21:42:20.4001265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4001700Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4002125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4002584Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4002923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4003300Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4003866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4004374Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4004820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.4005387Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.4005601Z 2025-08-14T21:42:20.4005706Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4006071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4006396Z return mod(**inputs) 2025-08-14T21:42:20.4006810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4007244Z outputs = self.model( 2025-08-14T21:42:20.4007635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4008143Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4008557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4008980Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4009312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4009666Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4010104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4010556Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4010998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.4011439Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.4011573Z 2025-08-14T21:42:20.4011686Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4012036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4012364Z return mod(**inputs) 2025-08-14T21:42:20.4012771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4013199Z outputs = self.model( 2025-08-14T21:42:20.4013602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4014036Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4014455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4014881Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4015220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4015583Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4016026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4016477Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4016934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.4017385Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.4017519Z 2025-08-14T21:42:20.4017607Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4017808Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4018014Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4018214Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4018433Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4018779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4019116Z return mod(**inputs) 2025-08-14T21:42:20.4019525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4019941Z outputs = self.model( 2025-08-14T21:42:20.4020346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4020781Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4021199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4021617Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4021991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4022345Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4022759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4023199Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4023634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4024079Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4024497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.4024959Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.4025137Z 2025-08-14T21:42:20.4025247Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4025592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4025897Z return mod(**inputs) 2025-08-14T21:42:20.4026299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4026719Z outputs = self.model( 2025-08-14T21:42:20.4027112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4027539Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4027955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4028375Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4028715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4029059Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4029474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4029898Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4030317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4030773Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4031203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.4031642Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.4031809Z 2025-08-14T21:42:20.4031912Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4032267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4032599Z return mod(**inputs) 2025-08-14T21:42:20.4033001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4033409Z outputs = self.model( 2025-08-14T21:42:20.4033798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4034212Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4034610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4035019Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4035377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4035717Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4036134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4036577Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4037015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.4037447Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.4037585Z 2025-08-14T21:42:20.4037815Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4038177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4038500Z return mod(**inputs) 2025-08-14T21:42:20.4038911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4039342Z outputs = self.model( 2025-08-14T21:42:20.4039755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4040187Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4040622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4041059Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4041402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4041756Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4042193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:42:20.4042675Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4042847Z 2025-08-14T21:42:20.4042956Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4043303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4043622Z return mod(**inputs) 2025-08-14T21:42:20.4044164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4044689Z outputs = self.model( 2025-08-14T21:42:20.4045132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4045583Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4046014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4046505Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4046876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4047302Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4047768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:42:20.4048276Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4048683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:20.4049044Z return self.act(input) 2025-08-14T21:42:20.4049160Z 2025-08-14T21:42:20.4049268Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4049649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4049999Z return mod(**inputs) 2025-08-14T21:42:20.4050482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4050941Z outputs = self.model( 2025-08-14T21:42:20.4051384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4051854Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4052277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4052703Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4053045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4053409Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4053825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-14T21:42:20.4054250Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:42:20.4054392Z 2025-08-14T21:42:20.4054491Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4054835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4055146Z return mod(**inputs) 2025-08-14T21:42:20.4055543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4055957Z outputs = self.model( 2025-08-14T21:42:20.4056344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4056767Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4057182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4057596Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4057933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4058288Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4058723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4059205Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4059645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.4060139Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.4060333Z 2025-08-14T21:42:20.4060439Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4060781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4061100Z return mod(**inputs) 2025-08-14T21:42:20.4061507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4061936Z outputs = self.model( 2025-08-14T21:42:20.4062341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4062778Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4063212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4063631Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4063957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4064335Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4064764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4065208Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4065639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.4066070Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.4066201Z 2025-08-14T21:42:20.4066310Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4066649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4066962Z return mod(**inputs) 2025-08-14T21:42:20.4067365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4067785Z outputs = self.model( 2025-08-14T21:42:20.4068181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4068610Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4069026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4069451Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4069784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4070129Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4070550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4070989Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4071439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.4071887Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.4072024Z 2025-08-14T21:42:20.4072116Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4072333Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4072554Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4072780Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4073027Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4073392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4073708Z return mod(**inputs) 2025-08-14T21:42:20.4074113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4074533Z outputs = self.model( 2025-08-14T21:42:20.4074936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4075385Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4075794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4076219Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4076558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4076905Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4077320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4077760Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4078312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4078763Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4079180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.4079645Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.4079822Z 2025-08-14T21:42:20.4079935Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4080281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4080591Z return mod(**inputs) 2025-08-14T21:42:20.4080993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4081418Z outputs = self.model( 2025-08-14T21:42:20.4081827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4082271Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4082701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4083135Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4083475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4083950Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4084395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4084866Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4085338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4085803Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4086242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.4086686Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.4086856Z 2025-08-14T21:42:20.4086961Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4087344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4087671Z return mod(**inputs) 2025-08-14T21:42:20.4088081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4088517Z outputs = self.model( 2025-08-14T21:42:20.4088931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4089386Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4089804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4090236Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4090579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4090934Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4091367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4091814Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4092295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.4092734Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.4092879Z 2025-08-14T21:42:20.4092981Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4093348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4093678Z return mod(**inputs) 2025-08-14T21:42:20.4094087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4094525Z outputs = self.model( 2025-08-14T21:42:20.4094942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4095377Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4095811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4096254Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4096604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4096961Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4097402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:42:20.4097892Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4098064Z 2025-08-14T21:42:20.4098175Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4098558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4098887Z return mod(**inputs) 2025-08-14T21:42:20.4099303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4099757Z outputs = self.model( 2025-08-14T21:42:20.4100177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4100623Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4101047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4101488Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4101828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4102224Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4102649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:42:20.4103132Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4103505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:20.4103862Z return self.act(input) 2025-08-14T21:42:20.4103969Z 2025-08-14T21:42:20.4104069Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4104417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4104735Z return mod(**inputs) 2025-08-14T21:42:20.4105163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4105588Z outputs = self.model( 2025-08-14T21:42:20.4105997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4106434Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4106898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4107336Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4107675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4108040Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4108472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-14T21:42:20.4108917Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:42:20.4109053Z 2025-08-14T21:42:20.4109162Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4109518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4109836Z return mod(**inputs) 2025-08-14T21:42:20.4110251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4110685Z outputs = self.model( 2025-08-14T21:42:20.4111090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4111532Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4111983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4112461Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4112820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4113209Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4113684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4114171Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4114651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.4115189Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.4115391Z 2025-08-14T21:42:20.4115505Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4115878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4116199Z return mod(**inputs) 2025-08-14T21:42:20.4116611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4117044Z outputs = self.model( 2025-08-14T21:42:20.4117448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4117909Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4118336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4118766Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4119101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4119463Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4119895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4120342Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4120820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.4121279Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.4121420Z 2025-08-14T21:42:20.4121534Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4121900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4122222Z return mod(**inputs) 2025-08-14T21:42:20.4122627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4123073Z outputs = self.model( 2025-08-14T21:42:20.4123500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4124097Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4124573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4125045Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4125417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4125798Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4126262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4126746Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4127222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.4127699Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.4127849Z 2025-08-14T21:42:20.4127940Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4128161Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4128386Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4128602Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4128845Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4129226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4129572Z return mod(**inputs) 2025-08-14T21:42:20.4130012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4130471Z outputs = self.model( 2025-08-14T21:42:20.4130882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4131331Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4131777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4132245Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4132613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4133014Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4133473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4133963Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4134457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4138302Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4138815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.4139367Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.4139567Z 2025-08-14T21:42:20.4139689Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4140069Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4140417Z return mod(**inputs) 2025-08-14T21:42:20.4140862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4141318Z outputs = self.model( 2025-08-14T21:42:20.4141757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4142266Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4142698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4143143Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4143495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4143860Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4144279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4144722Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4145160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4145609Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4146030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.4146472Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.4146627Z 2025-08-14T21:42:20.4146733Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4147086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4147392Z return mod(**inputs) 2025-08-14T21:42:20.4147792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4148214Z outputs = self.model( 2025-08-14T21:42:20.4148608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4149075Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4149512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4149938Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4150270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4150655Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4151083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4151534Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4151975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.4152426Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.4152560Z 2025-08-14T21:42:20.4152753Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4153108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4153444Z return mod(**inputs) 2025-08-14T21:42:20.4153867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4154289Z outputs = self.model( 2025-08-14T21:42:20.4154686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4155115Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4155620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4156050Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4156380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4156782Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4157240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:42:20.4157709Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4157875Z 2025-08-14T21:42:20.4157978Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4158332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4158656Z return mod(**inputs) 2025-08-14T21:42:20.4159061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4159490Z outputs = self.model( 2025-08-14T21:42:20.4159890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4160317Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4160744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4161190Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4161559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4161935Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4162386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:42:20.4162919Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4163327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:20.4163691Z return self.act(input) 2025-08-14T21:42:20.4163931Z 2025-08-14T21:42:20.4164050Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4164449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4164811Z return mod(**inputs) 2025-08-14T21:42:20.4165242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4165734Z outputs = self.model( 2025-08-14T21:42:20.4166181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4166615Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4167033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4167512Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4167853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4168197Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4168633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-14T21:42:20.4169071Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:42:20.4169203Z 2025-08-14T21:42:20.4169310Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4169658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4169965Z return mod(**inputs) 2025-08-14T21:42:20.4170378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4170802Z outputs = self.model( 2025-08-14T21:42:20.4171196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4171619Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4172036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4172457Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4172791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4173146Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4173580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4174031Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4174469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.4174964Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.4175160Z 2025-08-14T21:42:20.4175268Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4175606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4175924Z return mod(**inputs) 2025-08-14T21:42:20.4176324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4176745Z outputs = self.model( 2025-08-14T21:42:20.4177140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4177593Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4178001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4178408Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4178730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4179072Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4179506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4179926Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4180356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.4180778Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.4180908Z 2025-08-14T21:42:20.4181015Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4181380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4181695Z return mod(**inputs) 2025-08-14T21:42:20.4182111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4182536Z outputs = self.model( 2025-08-14T21:42:20.4182928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4183357Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4183763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4184168Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4184506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4184853Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4185279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4185708Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4186144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.4186582Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.4186718Z 2025-08-14T21:42:20.4186806Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4187012Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4187215Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4187415Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4187636Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4187987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4188304Z return mod(**inputs) 2025-08-14T21:42:20.4188703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4189126Z outputs = self.model( 2025-08-14T21:42:20.4189532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4189966Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4190379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4190828Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4191175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4191533Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4191961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4192411Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4192863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4193348Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4193793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.4194276Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.4194460Z 2025-08-14T21:42:20.4194572Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4194947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4195284Z return mod(**inputs) 2025-08-14T21:42:20.4195718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4196168Z outputs = self.model( 2025-08-14T21:42:20.4196581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4197035Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4197467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4197901Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4198251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4198617Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4199064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4199514Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4199967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4200427Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4200868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.4201315Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.4201484Z 2025-08-14T21:42:20.4201586Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4201945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4202276Z return mod(**inputs) 2025-08-14T21:42:20.4202685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4203144Z outputs = self.model( 2025-08-14T21:42:20.4203582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4204192Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4204658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4205121Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4205538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4205916Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4206376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 296, in forward 2025-08-14T21:42:20.4206837Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:42:20.4207284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.4207755Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.4207901Z 2025-08-14T21:42:20.4208005Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4208368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4208694Z return mod(**inputs) 2025-08-14T21:42:20.4209112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4209553Z outputs = self.model( 2025-08-14T21:42:20.4210000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4210444Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4210895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4211346Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4211692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4212051Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4212484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:42:20.4212970Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4213139Z 2025-08-14T21:42:20.4213249Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4213617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4213945Z return mod(**inputs) 2025-08-14T21:42:20.4214361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4214801Z outputs = self.model( 2025-08-14T21:42:20.4215222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4215677Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4216103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4216551Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4216920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4217310Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4217761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 307, in forward 2025-08-14T21:42:20.4218279Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4218678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:20.4219016Z return self.act(input) 2025-08-14T21:42:20.4219126Z 2025-08-14T21:42:20.4219229Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4219582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4219947Z return mod(**inputs) 2025-08-14T21:42:20.4220354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4220796Z outputs = self.model( 2025-08-14T21:42:20.4221245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1195, in forward 2025-08-14T21:42:20.4221714Z encoder_outputs = self.encoder( 2025-08-14T21:42:20.4222163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 812, in forward 2025-08-14T21:42:20.4222666Z layer_outputs = encoder_layer( 2025-08-14T21:42:20.4223012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4223389Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4223844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 309, in forward 2025-08-14T21:42:20.4224343Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:42:20.4224488Z 2025-08-14T21:42:20.4224605Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4224990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4225344Z return mod(**inputs) 2025-08-14T21:42:20.4225787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4226251Z outputs = self.model( 2025-08-14T21:42:20.4226692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4227169Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4227641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4228117Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4228483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4228861Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4229322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4229810Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4230297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.4230838Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.4231058Z 2025-08-14T21:42:20.4231175Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4231548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4231903Z return mod(**inputs) 2025-08-14T21:42:20.4232350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4232810Z outputs = self.model( 2025-08-14T21:42:20.4233243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4233722Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4234156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4234585Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4236040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4236404Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4236855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4237320Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4237956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.4238563Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.4238707Z 2025-08-14T21:42:20.4238825Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4239197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4239544Z return mod(**inputs) 2025-08-14T21:42:20.4239989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4240443Z outputs = self.model( 2025-08-14T21:42:20.4240931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4241407Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4241896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4242352Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4242718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4243096Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4243559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4244193Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4244687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.4245163Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.4245313Z 2025-08-14T21:42:20.4245402Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4245631Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4245840Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4246049Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4246275Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4246637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4246961Z return mod(**inputs) 2025-08-14T21:42:20.4247367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4247803Z outputs = self.model( 2025-08-14T21:42:20.4248216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4248654Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4249078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4249515Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4249861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4250220Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4250654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4251170Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4251629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4252089Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4252515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.4252978Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.4253178Z 2025-08-14T21:42:20.4253286Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4253624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4253944Z return mod(**inputs) 2025-08-14T21:42:20.4254348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4254772Z outputs = self.model( 2025-08-14T21:42:20.4255191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4255624Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4256059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4256480Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4256818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4257167Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4257596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4258041Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4258488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4258935Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4259364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.4259796Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.4259961Z 2025-08-14T21:42:20.4260062Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4260408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4260717Z return mod(**inputs) 2025-08-14T21:42:20.4261116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4261539Z outputs = self.model( 2025-08-14T21:42:20.4261939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4262358Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4262780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4263213Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4263560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4263909Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4264348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4264808Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4265276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.4265726Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.4265864Z 2025-08-14T21:42:20.4265964Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4266312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4266628Z return mod(**inputs) 2025-08-14T21:42:20.4267031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4267473Z outputs = self.model( 2025-08-14T21:42:20.4267873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4268291Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4268717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4269168Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4269501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4269853Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4270294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4270755Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4271201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.4271698Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.4271900Z 2025-08-14T21:42:20.4271999Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4272372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4272705Z return mod(**inputs) 2025-08-14T21:42:20.4273143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4273605Z outputs = self.model( 2025-08-14T21:42:20.4274023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4274451Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4274881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4275322Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4275653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4276001Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4276434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4276900Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4277361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.4277806Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.4277939Z 2025-08-14T21:42:20.4278051Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4278409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4278753Z return mod(**inputs) 2025-08-14T21:42:20.4279222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4279680Z outputs = self.model( 2025-08-14T21:42:20.4280117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4280596Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4281065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4281560Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4281917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4282298Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4282773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4283272Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4283897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.4284406Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.4284559Z 2025-08-14T21:42:20.4284682Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4284914Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4285151Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4285384Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4285630Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4286004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4286334Z return mod(**inputs) 2025-08-14T21:42:20.4286753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4287181Z outputs = self.model( 2025-08-14T21:42:20.4287597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4288046Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4288494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4288923Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4289269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4289651Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4290106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4290613Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4291114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4291607Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4292049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.4292525Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.4292716Z 2025-08-14T21:42:20.4292821Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4293181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4293502Z return mod(**inputs) 2025-08-14T21:42:20.4293973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4294449Z outputs = self.model( 2025-08-14T21:42:20.4294860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4295301Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4295782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4296280Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4296629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4296989Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4297426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4297890Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4298375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4298839Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4299294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.4299751Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.4299912Z 2025-08-14T21:42:20.4300015Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4300372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4300695Z return mod(**inputs) 2025-08-14T21:42:20.4301096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4301528Z outputs = self.model( 2025-08-14T21:42:20.4301939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4302380Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4302828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4303298Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4303666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4304039Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4304466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4304933Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4305399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.4305833Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.4305975Z 2025-08-14T21:42:20.4306079Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4306438Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4306761Z return mod(**inputs) 2025-08-14T21:42:20.4307171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4307603Z outputs = self.model( 2025-08-14T21:42:20.4308018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4308486Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4308910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4309347Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4309685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4310027Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4310453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:42:20.4310941Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4311107Z 2025-08-14T21:42:20.4311215Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4311555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4311879Z return mod(**inputs) 2025-08-14T21:42:20.4312310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4312744Z outputs = self.model( 2025-08-14T21:42:20.4313151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4313608Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4314029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4314449Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4314799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4315168Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4315599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:42:20.4316076Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4316462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:20.4316800Z return self.act(input) 2025-08-14T21:42:20.4316907Z 2025-08-14T21:42:20.4317018Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4317371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4317692Z return mod(**inputs) 2025-08-14T21:42:20.4318107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4318559Z outputs = self.model( 2025-08-14T21:42:20.4318993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4319459Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4319891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4320319Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4320672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4321067Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4321530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:42:20.4322019Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:42:20.4322169Z 2025-08-14T21:42:20.4322279Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4322687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4323035Z return mod(**inputs) 2025-08-14T21:42:20.4323472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4324224Z outputs = self.model( 2025-08-14T21:42:20.4324683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4325191Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4325680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4326167Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4326530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4326913Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4327403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4327894Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4328398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.4328946Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.4329171Z 2025-08-14T21:42:20.4329281Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4329658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4329990Z return mod(**inputs) 2025-08-14T21:42:20.4330430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4330900Z outputs = self.model( 2025-08-14T21:42:20.4331332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4331799Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4332266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4332730Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4333088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4333468Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4333930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4334421Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4334898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.4335367Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.4335507Z 2025-08-14T21:42:20.4335622Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4336002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4336338Z return mod(**inputs) 2025-08-14T21:42:20.4336778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4337240Z outputs = self.model( 2025-08-14T21:42:20.4337756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4338253Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4338692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4339139Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4339482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4339847Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4340291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4340797Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4341252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.4341699Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.4341838Z 2025-08-14T21:42:20.4341925Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4342132Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4342371Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4342584Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4342808Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4343194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4343524Z return mod(**inputs) 2025-08-14T21:42:20.4343937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4344360Z outputs = self.model( 2025-08-14T21:42:20.4344777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4345214Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4345641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4346072Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4346421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4346779Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4347209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4347673Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4348125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4348590Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4349023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.4349496Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.4349691Z 2025-08-14T21:42:20.4349791Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4350130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4350433Z return mod(**inputs) 2025-08-14T21:42:20.4350823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4351240Z outputs = self.model( 2025-08-14T21:42:20.4351629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4352054Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4352528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4352966Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4353308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4353666Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4354112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4354584Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4355021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4355475Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4355909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.4356361Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.4356518Z 2025-08-14T21:42:20.4356642Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4357001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4357337Z return mod(**inputs) 2025-08-14T21:42:20.4357746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4358184Z outputs = self.model( 2025-08-14T21:42:20.4358602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4359042Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4359473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4359915Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4360266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4360622Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4361063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4361526Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4361985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.4362426Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.4362569Z 2025-08-14T21:42:20.4362671Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4363027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4363356Z return mod(**inputs) 2025-08-14T21:42:20.4363854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4364354Z outputs = self.model( 2025-08-14T21:42:20.4364815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4365310Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4365774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4366228Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4366580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4366969Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4367409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4367874Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4368340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.4368842Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.4369076Z 2025-08-14T21:42:20.4369178Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4369538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4369861Z return mod(**inputs) 2025-08-14T21:42:20.4370269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4370703Z outputs = self.model( 2025-08-14T21:42:20.4371132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4371578Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4372069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4372509Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4372857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4373204Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4373639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4374106Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4374572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.4375005Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.4375147Z 2025-08-14T21:42:20.4375248Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4375600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4375915Z return mod(**inputs) 2025-08-14T21:42:20.4376323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4376751Z outputs = self.model( 2025-08-14T21:42:20.4377167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4377596Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4378021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4378451Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4378795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4379148Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4379580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4380043Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4380498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.4381030Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.4381173Z 2025-08-14T21:42:20.4381255Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4381466Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4381668Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4381876Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4382108Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4382476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4382831Z return mod(**inputs) 2025-08-14T21:42:20.4383293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4383750Z outputs = self.model( 2025-08-14T21:42:20.4384159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4384604Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4385088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4385521Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4385870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4386248Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4386688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4387146Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4387608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4388065Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4388506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.4388971Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.4389163Z 2025-08-14T21:42:20.4389266Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4389622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4389948Z return mod(**inputs) 2025-08-14T21:42:20.4390340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4390760Z outputs = self.model( 2025-08-14T21:42:20.4391052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4391136Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4391435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4391517Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4391737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4391818Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4392129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4392236Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4392533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4392635Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4392936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.4393050Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.4393054Z 2025-08-14T21:42:20.4393165Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4393357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4393428Z return mod(**inputs) 2025-08-14T21:42:20.4393720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4393814Z outputs = self.model( 2025-08-14T21:42:20.4394110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4394181Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4394489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4394578Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4394797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4394885Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4395207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4395324Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4395624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.4395704Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.4395708Z 2025-08-14T21:42:20.4395820Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4396017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4396090Z return mod(**inputs) 2025-08-14T21:42:20.4396392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4396460Z outputs = self.model( 2025-08-14T21:42:20.4396773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4396849Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4397153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4397233Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4397454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4397542Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4397849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:42:20.4397969Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4397973Z 2025-08-14T21:42:20.4398085Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4398283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4398359Z return mod(**inputs) 2025-08-14T21:42:20.4398661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4398729Z outputs = self.model( 2025-08-14T21:42:20.4399038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4399129Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4399442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4399513Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4399735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4399819Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4400135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:42:20.4400252Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4400470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:20.4400542Z return self.act(input) 2025-08-14T21:42:20.4400545Z 2025-08-14T21:42:20.4400653Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4400868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4400937Z return mod(**inputs) 2025-08-14T21:42:20.4401265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4401336Z outputs = self.model( 2025-08-14T21:42:20.4401661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4401746Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4402061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4402147Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4402377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4402460Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4402782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:42:20.4402868Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:42:20.4402872Z 2025-08-14T21:42:20.4402985Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4403192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4403260Z return mod(**inputs) 2025-08-14T21:42:20.4403589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4403662Z outputs = self.model( 2025-08-14T21:42:20.4404096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4404185Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4404506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4404591Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4404823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4404908Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4405228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4405329Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4405665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.4405815Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.4405820Z 2025-08-14T21:42:20.4405921Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4406125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4406191Z return mod(**inputs) 2025-08-14T21:42:20.4406499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4406594Z outputs = self.model( 2025-08-14T21:42:20.4406897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4406976Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4407278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4407368Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4407596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4407673Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4407997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4408099Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4408402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.4408490Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.4408493Z 2025-08-14T21:42:20.4408597Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4408798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4408862Z return mod(**inputs) 2025-08-14T21:42:20.4409165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4409239Z outputs = self.model( 2025-08-14T21:42:20.4409542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4409616Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4409922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4409992Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4410217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4410294Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4410596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4410700Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4411003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.4411102Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.4411106Z 2025-08-14T21:42:20.4411189Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4411271Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4411360Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4411442Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4411548Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4411786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4411854Z return mod(**inputs) 2025-08-14T21:42:20.4412182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4412252Z outputs = self.model( 2025-08-14T21:42:20.4412574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4412683Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4412989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4413059Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4413283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4413361Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4413689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4413788Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4414105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4414211Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4414498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.4414636Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.4414640Z 2025-08-14T21:42:20.4414741Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4414944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4415016Z return mod(**inputs) 2025-08-14T21:42:20.4415309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4415383Z outputs = self.model( 2025-08-14T21:42:20.4415677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4415747Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4416045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4416114Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4416331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4416419Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4416719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4416831Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4417123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4417215Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4417502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.4417606Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.4417610Z 2025-08-14T21:42:20.4417713Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4417904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4417985Z return mod(**inputs) 2025-08-14T21:42:20.4418295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4418361Z outputs = self.model( 2025-08-14T21:42:20.4418662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4418740Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4419052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4419128Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4419339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4419417Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4419723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4419829Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4420127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.4420220Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.4420224Z 2025-08-14T21:42:20.4420322Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4420518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4420580Z return mod(**inputs) 2025-08-14T21:42:20.4420866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4420939Z outputs = self.model( 2025-08-14T21:42:20.4421233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4421313Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4421606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4421675Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4421894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4421971Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4422273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4422378Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4422670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.4422832Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.4422836Z 2025-08-14T21:42:20.4422932Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4423120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4423182Z return mod(**inputs) 2025-08-14T21:42:20.4423475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4423545Z outputs = self.model( 2025-08-14T21:42:20.4423842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4423973Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4424279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4424349Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4424572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4424647Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4424945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4425071Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4425364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.4425448Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.4425454Z 2025-08-14T21:42:20.4425552Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4425742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4425833Z return mod(**inputs) 2025-08-14T21:42:20.4426132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4426213Z outputs = self.model( 2025-08-14T21:42:20.4426513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4426583Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4426882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4426952Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4427166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4427247Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4427539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4427649Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4427944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.4428027Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.4428031Z 2025-08-14T21:42:20.4428115Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4428191Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4428266Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4428345Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4428445Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4428644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4428708Z return mod(**inputs) 2025-08-14T21:42:20.4429002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4429077Z outputs = self.model( 2025-08-14T21:42:20.4429372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4429445Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4429746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4429816Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4430035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4430128Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4430418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4430529Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4430819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4430937Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4431219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.4431350Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.4431354Z 2025-08-14T21:42:20.4431462Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4431660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4431735Z return mod(**inputs) 2025-08-14T21:42:20.4432059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4432127Z outputs = self.model( 2025-08-14T21:42:20.4432452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4432527Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4432827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4432905Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4433121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4433223Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4433521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4433629Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4433934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4434029Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4434320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.4434426Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.4434430Z 2025-08-14T21:42:20.4434530Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4434733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4434798Z return mod(**inputs) 2025-08-14T21:42:20.4435103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4435177Z outputs = self.model( 2025-08-14T21:42:20.4435469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4435552Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4435844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4435913Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4436132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4436244Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4436553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4436657Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4436960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.4437050Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.4437071Z 2025-08-14T21:42:20.4437173Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4437377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4437441Z return mod(**inputs) 2025-08-14T21:42:20.4437869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4437959Z outputs = self.model( 2025-08-14T21:42:20.4438327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4438407Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4438765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4438842Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4439084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4439168Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4439484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:42:20.4439616Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4439622Z 2025-08-14T21:42:20.4439729Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4439954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4440023Z return mod(**inputs) 2025-08-14T21:42:20.4440350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4440428Z outputs = self.model( 2025-08-14T21:42:20.4440747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4440824Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4441150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4441227Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4441465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4441547Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4441869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:42:20.4442005Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4442227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:20.4442302Z return self.act(input) 2025-08-14T21:42:20.4442313Z 2025-08-14T21:42:20.4442420Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4442627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4442703Z return mod(**inputs) 2025-08-14T21:42:20.4443056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4443128Z outputs = self.model( 2025-08-14T21:42:20.4443483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4443561Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4444037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4444177Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4444416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4444510Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4444836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:42:20.4444926Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:42:20.4444939Z 2025-08-14T21:42:20.4445070Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4445292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4445373Z return mod(**inputs) 2025-08-14T21:42:20.4445735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4445811Z outputs = self.model( 2025-08-14T21:42:20.4446145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4446223Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4446547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4446623Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4446855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4446947Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4447269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4447382Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4447702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.4447859Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.4447863Z 2025-08-14T21:42:20.4447974Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4448183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4448252Z return mod(**inputs) 2025-08-14T21:42:20.4448588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4448657Z outputs = self.model( 2025-08-14T21:42:20.4448987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4449063Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4449390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4449474Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4449701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4449809Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4450127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4450230Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4450556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.4450639Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.4450661Z 2025-08-14T21:42:20.4450769Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4450985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4451052Z return mod(**inputs) 2025-08-14T21:42:20.4451381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4451452Z outputs = self.model( 2025-08-14T21:42:20.4451792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4451878Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4452215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4452300Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4452530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4452610Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4452934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4453036Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4453354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.4453455Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.4453459Z 2025-08-14T21:42:20.4453542Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4453632Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4453713Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4453791Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4453907Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4454113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4454181Z return mod(**inputs) 2025-08-14T21:42:20.4454510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4454582Z outputs = self.model( 2025-08-14T21:42:20.4454906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4454984Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4455303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4455385Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4455616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4455706Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4456020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4456121Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4456464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4456570Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4456881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.4457036Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.4457040Z 2025-08-14T21:42:20.4457150Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4457391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4457461Z return mod(**inputs) 2025-08-14T21:42:20.4457801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4457879Z outputs = self.model( 2025-08-14T21:42:20.4458198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4458299Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4458620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4458721Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4458961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4459044Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4459370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4459471Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4459792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4459900Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4460208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.4460315Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.4460325Z 2025-08-14T21:42:20.4460426Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4460624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4460697Z return mod(**inputs) 2025-08-14T21:42:20.4461001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4461066Z outputs = self.model( 2025-08-14T21:42:20.4461368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4461442Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4461754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4461825Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4462047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4462134Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4462437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4462535Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4462852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.4462950Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.4462954Z 2025-08-14T21:42:20.4463063Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4463256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4463320Z return mod(**inputs) 2025-08-14T21:42:20.4463627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4463711Z outputs = self.model( 2025-08-14T21:42:20.4464012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4464082Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4464371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4464451Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4464677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4464754Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4465070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4465174Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4465471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.4465616Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.4465619Z 2025-08-14T21:42:20.4465718Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4465916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4465981Z return mod(**inputs) 2025-08-14T21:42:20.4466280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4466348Z outputs = self.model( 2025-08-14T21:42:20.4466641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4466724Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4467016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4467094Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4467304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4467380Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4467678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4467779Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4468070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.4468154Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.4468158Z 2025-08-14T21:42:20.4468257Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4468453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4468516Z return mod(**inputs) 2025-08-14T21:42:20.4468809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4468900Z outputs = self.model( 2025-08-14T21:42:20.4469197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4469274Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4469571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4469639Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4469873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4469948Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4470238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4470350Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4470658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.4470750Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.4470753Z 2025-08-14T21:42:20.4470829Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4470923Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4471008Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4471081Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4471184Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4471381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4471444Z return mod(**inputs) 2025-08-14T21:42:20.4471748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4471815Z outputs = self.model( 2025-08-14T21:42:20.4472108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4472187Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4472480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4472559Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4472771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4472844Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4473145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4473245Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4473542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4473644Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4473923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.4474060Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.4474065Z 2025-08-14T21:42:20.4474163Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4474352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4474424Z return mod(**inputs) 2025-08-14T21:42:20.4474715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4474819Z outputs = self.model( 2025-08-14T21:42:20.4475124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4475196Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4475507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4475575Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4475796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4475887Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4476181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4476291Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4476585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4476694Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4476978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.4477096Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.4477101Z 2025-08-14T21:42:20.4477207Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4477399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4477462Z return mod(**inputs) 2025-08-14T21:42:20.4477761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4477827Z outputs = self.model( 2025-08-14T21:42:20.4478128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4478202Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4478496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4478576Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4478789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4478866Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4479173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4479277Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4479588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.4479670Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.4479676Z 2025-08-14T21:42:20.4479778Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4479984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4480049Z return mod(**inputs) 2025-08-14T21:42:20.4480360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4480428Z outputs = self.model( 2025-08-14T21:42:20.4480729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4480812Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4481148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4481222Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4481459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4481538Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4481857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:42:20.4482001Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4482006Z 2025-08-14T21:42:20.4482111Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4482327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4482396Z return mod(**inputs) 2025-08-14T21:42:20.4482724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4482793Z outputs = self.model( 2025-08-14T21:42:20.4483133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4483221Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4483560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4483646Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4484011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4484101Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4484428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:42:20.4484562Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4484792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:20.4484880Z return self.act(input) 2025-08-14T21:42:20.4484886Z 2025-08-14T21:42:20.4484997Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4485231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4485303Z return mod(**inputs) 2025-08-14T21:42:20.4485620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4485698Z outputs = self.model( 2025-08-14T21:42:20.4486017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4486092Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4486401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4486472Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4486695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4486773Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4487072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:42:20.4487164Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:42:20.4487167Z 2025-08-14T21:42:20.4487267Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4487470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4487564Z return mod(**inputs) 2025-08-14T21:42:20.4487873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4487949Z outputs = self.model( 2025-08-14T21:42:20.4488260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4488334Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4488651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4488740Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4488966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4489043Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4489349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4489483Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4489784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.4489958Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.4489963Z 2025-08-14T21:42:20.4490068Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4490272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4490347Z return mod(**inputs) 2025-08-14T21:42:20.4490656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4490734Z outputs = self.model( 2025-08-14T21:42:20.4491039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4491115Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4491426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4491500Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4491721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4491809Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4492111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4492218Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4492521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.4492602Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.4492608Z 2025-08-14T21:42:20.4492719Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4492918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4492995Z return mod(**inputs) 2025-08-14T21:42:20.4493300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4493372Z outputs = self.model( 2025-08-14T21:42:20.4493683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4493757Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4494078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4494155Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4494373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4494457Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4494756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4494871Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4495178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.4495261Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.4495264Z 2025-08-14T21:42:20.4495350Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4495430Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4495506Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4495588Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4495705Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4495903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4495976Z return mod(**inputs) 2025-08-14T21:42:20.4496299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4496377Z outputs = self.model( 2025-08-14T21:42:20.4496679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4496753Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4497068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4497140Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4497362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4497448Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4497749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4497853Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4498154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4498248Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4498542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.4498673Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.4498677Z 2025-08-14T21:42:20.4498785Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4498984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4499060Z return mod(**inputs) 2025-08-14T21:42:20.4499369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4499437Z outputs = self.model( 2025-08-14T21:42:20.4499746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4499823Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4500131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4500230Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4500446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4500520Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4500822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4500916Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4501241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4501335Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4501617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.4501732Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.4501736Z 2025-08-14T21:42:20.4501837Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4502059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4502128Z return mod(**inputs) 2025-08-14T21:42:20.4502446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4502524Z outputs = self.model( 2025-08-14T21:42:20.4502827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4502901Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4503216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4503286Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4503506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4503581Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4503873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4503975Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4504270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.4504357Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.4504361Z 2025-08-14T21:42:20.4504457Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4504649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4504722Z return mod(**inputs) 2025-08-14T21:42:20.4505018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4505087Z outputs = self.model( 2025-08-14T21:42:20.4505389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4505459Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4505767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4505837Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4506048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4506130Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4506437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4506549Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4506837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.4506981Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.4506985Z 2025-08-14T21:42:20.4507118Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4507310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4507382Z return mod(**inputs) 2025-08-14T21:42:20.4507677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4507743Z outputs = self.model( 2025-08-14T21:42:20.4508060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4508133Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4508431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4508521Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4508734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4508818Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4509108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4509209Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4509509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.4509588Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.4509592Z 2025-08-14T21:42:20.4509698Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4509893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4509957Z return mod(**inputs) 2025-08-14T21:42:20.4510258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4510327Z outputs = self.model( 2025-08-14T21:42:20.4510623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4510701Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4510993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4511070Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4511282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4511357Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4511660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4511764Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4512061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.4512143Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.4512147Z 2025-08-14T21:42:20.4512239Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4512322Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4512395Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4512468Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4512578Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4512771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4512845Z return mod(**inputs) 2025-08-14T21:42:20.4513154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4513239Z outputs = self.model( 2025-08-14T21:42:20.4513537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4513608Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4513899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4513975Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4514203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4514291Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4514605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4514710Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4515009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4515102Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4515385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.4515516Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.4515519Z 2025-08-14T21:42:20.4515619Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4515815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4515879Z return mod(**inputs) 2025-08-14T21:42:20.4516178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4516247Z outputs = self.model( 2025-08-14T21:42:20.4516538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4516616Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4516912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4516983Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4517203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4517279Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4517578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4517680Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4517972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4518073Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4518351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.4518483Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.4518487Z 2025-08-14T21:42:20.4518585Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4518780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4518854Z return mod(**inputs) 2025-08-14T21:42:20.4519154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4519242Z outputs = self.model( 2025-08-14T21:42:20.4519558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4519630Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4519943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4520016Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4520251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4520339Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4520679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4520795Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4521094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.4521180Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.4521183Z 2025-08-14T21:42:20.4521290Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4521488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4521552Z return mod(**inputs) 2025-08-14T21:42:20.4521861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4521929Z outputs = self.model( 2025-08-14T21:42:20.4522237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4522311Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4522612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4522690Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4522905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4522989Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4523289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:42:20.4523411Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4523415Z 2025-08-14T21:42:20.4523526Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4523821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4523915Z return mod(**inputs) 2025-08-14T21:42:20.4524254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4524328Z outputs = self.model( 2025-08-14T21:42:20.4524661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4524777Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4525114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4525198Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4525434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4525524Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4525846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:42:20.4525997Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4526227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:20.4526300Z return self.act(input) 2025-08-14T21:42:20.4526308Z 2025-08-14T21:42:20.4526423Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4526629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4526723Z return mod(**inputs) 2025-08-14T21:42:20.4527052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4527140Z outputs = self.model( 2025-08-14T21:42:20.4527460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4527548Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4527863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4527945Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4528174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4528258Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4528582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:42:20.4528666Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:42:20.4528670Z 2025-08-14T21:42:20.4528783Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4528989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4529058Z return mod(**inputs) 2025-08-14T21:42:20.4529384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4529454Z outputs = self.model( 2025-08-14T21:42:20.4529769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4529855Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4530173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4530255Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4530484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4530564Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4530886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4530991Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4531314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.4531506Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.4531510Z 2025-08-14T21:42:20.4531618Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4531830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4531899Z return mod(**inputs) 2025-08-14T21:42:20.4532220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4532320Z outputs = self.model( 2025-08-14T21:42:20.4532646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4532729Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4533051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4533128Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4533391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4533477Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4533829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4533938Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4534262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.4534355Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.4534359Z 2025-08-14T21:42:20.4534467Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4534678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4534756Z return mod(**inputs) 2025-08-14T21:42:20.4535090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4535169Z outputs = self.model( 2025-08-14T21:42:20.4535489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4535564Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4535892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4535965Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4536202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4536285Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4536600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4536713Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4537032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.4537121Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.4537131Z 2025-08-14T21:42:20.4537216Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4537298Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4537385Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4537465Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4537569Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4537943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4538046Z return mod(**inputs) 2025-08-14T21:42:20.4538373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4538454Z outputs = self.model( 2025-08-14T21:42:20.4538777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4538865Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4539231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4539307Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4539542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4539623Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4539949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4540077Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4540399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4540532Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4540842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.4540992Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.4540996Z 2025-08-14T21:42:20.4541104Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4541315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4541396Z return mod(**inputs) 2025-08-14T21:42:20.4541720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4541792Z outputs = self.model( 2025-08-14T21:42:20.4542121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4542200Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4542526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4542601Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4542832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4542922Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4543242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4543351Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4543672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4543772Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4544083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.4544198Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.4544202Z 2025-08-14T21:42:20.4544307Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4544526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4544624Z return mod(**inputs) 2025-08-14T21:42:20.4544959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4545029Z outputs = self.model( 2025-08-14T21:42:20.4545335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4545417Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4545729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4545823Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4546040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4546117Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4546413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4546507Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4546813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.4546902Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.4546906Z 2025-08-14T21:42:20.4547018Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4547219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4547286Z return mod(**inputs) 2025-08-14T21:42:20.4547588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4547665Z outputs = self.model( 2025-08-14T21:42:20.4547963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4548045Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4548347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4548418Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4548643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4548723Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4549023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4549137Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4549440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.4549592Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.4549596Z 2025-08-14T21:42:20.4549694Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4549882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4549953Z return mod(**inputs) 2025-08-14T21:42:20.4550245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4550319Z outputs = self.model( 2025-08-14T21:42:20.4550618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4550689Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4551002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4551096Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4551334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4551414Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4551736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4551855Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4552200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.4552277Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.4552289Z 2025-08-14T21:42:20.4552387Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4552579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4552653Z return mod(**inputs) 2025-08-14T21:42:20.4552968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4553038Z outputs = self.model( 2025-08-14T21:42:20.4553372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4553447Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4553759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4553827Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4554041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4554126Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4554418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4554518Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4554816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.4554900Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.4554905Z 2025-08-14T21:42:20.4554988Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4555062Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4555136Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4555218Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4555316Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4555509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4555580Z return mod(**inputs) 2025-08-14T21:42:20.4555874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4555945Z outputs = self.model( 2025-08-14T21:42:20.4556236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4556306Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4556605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4556674Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4556893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4556988Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4557284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4557393Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4557691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4557784Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4558088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.4558213Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.4558217Z 2025-08-14T21:42:20.4558322Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4558510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4558576Z return mod(**inputs) 2025-08-14T21:42:20.4558893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4558961Z outputs = self.model( 2025-08-14T21:42:20.4559278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4559353Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4559660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4559738Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4559957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4560033Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4560343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4560449Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4560756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4560852Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4561138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.4561256Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.4561260Z 2025-08-14T21:42:20.4561365Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4561581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4561651Z return mod(**inputs) 2025-08-14T21:42:20.4561974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4562052Z outputs = self.model( 2025-08-14T21:42:20.4562373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4562458Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4562778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4562854Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4563093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4563175Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4563520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4563638Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4564085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.4564193Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.4564197Z 2025-08-14T21:42:20.4564306Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4564537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4564617Z return mod(**inputs) 2025-08-14T21:42:20.4564956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4565036Z outputs = self.model( 2025-08-14T21:42:20.4565338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4565434Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4565747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4566773Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4567013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4567103Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4567405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:42:20.4567533Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4567537Z 2025-08-14T21:42:20.4567640Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4567837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4567915Z return mod(**inputs) 2025-08-14T21:42:20.4568218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4568295Z outputs = self.model( 2025-08-14T21:42:20.4568599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4568674Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4568983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4569054Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4569273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4569359Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4569663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:42:20.4569789Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4569998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:20.4570069Z return self.act(input) 2025-08-14T21:42:20.4570073Z 2025-08-14T21:42:20.4570186Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4570383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4570453Z return mod(**inputs) 2025-08-14T21:42:20.4570755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4570842Z outputs = self.model( 2025-08-14T21:42:20.4571160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4571232Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4571543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4571624Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4571859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4571942Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4572239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:42:20.4572321Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:42:20.4572325Z 2025-08-14T21:42:20.4572432Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4572641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4572716Z return mod(**inputs) 2025-08-14T21:42:20.4573038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4573109Z outputs = self.model( 2025-08-14T21:42:20.4573418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4573491Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4573795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4573868Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4574084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4574170Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4574469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4574567Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4574871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.4575021Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.4575024Z 2025-08-14T21:42:20.4575132Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4575323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4575390Z return mod(**inputs) 2025-08-14T21:42:20.4575701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4575768Z outputs = self.model( 2025-08-14T21:42:20.4576080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4576150Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4576441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4576520Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4576730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4576805Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4577126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4577222Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4577521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.4577599Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.4577603Z 2025-08-14T21:42:20.4577698Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4577919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4577981Z return mod(**inputs) 2025-08-14T21:42:20.4578282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4578347Z outputs = self.model( 2025-08-14T21:42:20.4578641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4578748Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4579041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4579126Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4579348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4579425Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4579721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4579813Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4580104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.4580195Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.4580198Z 2025-08-14T21:42:20.4580276Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4580359Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4580432Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4580506Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4580610Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4580800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4580863Z return mod(**inputs) 2025-08-14T21:42:20.4581165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4581233Z outputs = self.model( 2025-08-14T21:42:20.4581542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4581615Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4581921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4582000Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4582217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4582294Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4582598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4582692Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4582993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4583112Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4583397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.4583538Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.4583542Z 2025-08-14T21:42:20.4583642Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4583845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4583927Z return mod(**inputs) 2025-08-14T21:42:20.4584232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4584305Z outputs = self.model( 2025-08-14T21:42:20.4584611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4584691Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4584997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4585069Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4585310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4585390Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4585697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4585798Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4586087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4586188Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4586464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.4586568Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.4586571Z 2025-08-14T21:42:20.4586678Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4586868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4586945Z return mod(**inputs) 2025-08-14T21:42:20.4587236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4587302Z outputs = self.model( 2025-08-14T21:42:20.4587597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4587668Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4587961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4588040Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4588259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4588344Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4588644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4588739Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4589044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.4589142Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.4589145Z 2025-08-14T21:42:20.4589253Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4589451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4589515Z return mod(**inputs) 2025-08-14T21:42:20.4589826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4589891Z outputs = self.model( 2025-08-14T21:42:20.4590208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4590286Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4590585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4590664Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4590881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4590979Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4591291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4591411Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4591718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.4591868Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.4591872Z 2025-08-14T21:42:20.4591971Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4592175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4592245Z return mod(**inputs) 2025-08-14T21:42:20.4592555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4592622Z outputs = self.model( 2025-08-14T21:42:20.4592929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4593014Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4593349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4593423Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4593663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4593742Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4594066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4594178Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4594494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.4594588Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.4594592Z 2025-08-14T21:42:20.4594696Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4594916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4594985Z return mod(**inputs) 2025-08-14T21:42:20.4595305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4595406Z outputs = self.model( 2025-08-14T21:42:20.4595721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4595798Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4596122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4596198Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4596434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4596537Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4596855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4596976Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4597302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.4597399Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.4597421Z 2025-08-14T21:42:20.4597506Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4597588Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4597676Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4597773Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4597882Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4598102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4598170Z return mod(**inputs) 2025-08-14T21:42:20.4598498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4598567Z outputs = self.model( 2025-08-14T21:42:20.4598887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4598974Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4599294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4599368Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4599609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4599691Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4600018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4600128Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4600446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4600555Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4600857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.4601002Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.4601007Z 2025-08-14T21:42:20.4601116Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4601326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4601402Z return mod(**inputs) 2025-08-14T21:42:20.4601725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4601802Z outputs = self.model( 2025-08-14T21:42:20.4602148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4602224Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4602553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4602628Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4602859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4602977Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4603292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4603407Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4603819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4603962Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4604325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.4604442Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.4604447Z 2025-08-14T21:42:20.4604585Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4604801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4604876Z return mod(**inputs) 2025-08-14T21:42:20.4605216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4605289Z outputs = self.model( 2025-08-14T21:42:20.4605617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4605705Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4606033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4606119Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4606357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4606433Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4606738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4606840Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4607143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.4607224Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.4607227Z 2025-08-14T21:42:20.4607327Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4607529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4607595Z return mod(**inputs) 2025-08-14T21:42:20.4607893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4607967Z outputs = self.model( 2025-08-14T21:42:20.4608266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4608344Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4608641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4608731Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4608962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4609039Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4609353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:42:20.4609472Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4609496Z 2025-08-14T21:42:20.4609598Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4609802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4609868Z return mod(**inputs) 2025-08-14T21:42:20.4610169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4610244Z outputs = self.model( 2025-08-14T21:42:20.4610563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4610645Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4610966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4611042Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4611287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4611367Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4611695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:42:20.4611812Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4612022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:20.4612103Z return self.act(input) 2025-08-14T21:42:20.4612106Z 2025-08-14T21:42:20.4612209Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4612415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4612481Z return mod(**inputs) 2025-08-14T21:42:20.4612784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4612862Z outputs = self.model( 2025-08-14T21:42:20.4613163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4613234Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4613543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4613614Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4613838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4613914Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4614216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:42:20.4614308Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:42:20.4614312Z 2025-08-14T21:42:20.4614411Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4614616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4614683Z return mod(**inputs) 2025-08-14T21:42:20.4615005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4615079Z outputs = self.model( 2025-08-14T21:42:20.4615376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4615448Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4615757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4615845Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4616073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4616149Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4616451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4616556Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4616880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.4617038Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.4617041Z 2025-08-14T21:42:20.4617157Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4617355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4617430Z return mod(**inputs) 2025-08-14T21:42:20.4617734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4617799Z outputs = self.model( 2025-08-14T21:42:20.4618099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4618169Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4618465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4618533Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4618744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4618827Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4619117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4619218Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4619514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.4619595Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.4619598Z 2025-08-14T21:42:20.4619704Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4619900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4619964Z return mod(**inputs) 2025-08-14T21:42:20.4620271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4620339Z outputs = self.model( 2025-08-14T21:42:20.4620645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4620717Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4621016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4621118Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4621349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4621438Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4621754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4621855Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4622199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.4622288Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.4622291Z 2025-08-14T21:42:20.4622380Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4622463Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4622544Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4622630Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4622737Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4622989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4623066Z return mod(**inputs) 2025-08-14T21:42:20.4623399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4623469Z outputs = self.model( 2025-08-14T21:42:20.4623775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4623848Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4624157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4624229Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4624458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4624549Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4624870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4624979Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4625310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4625412Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4625726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.4625864Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.4625870Z 2025-08-14T21:42:20.4625983Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4626211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4626280Z return mod(**inputs) 2025-08-14T21:42:20.4626611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4626682Z outputs = self.model( 2025-08-14T21:42:20.4627009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4627091Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4627395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4627505Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4627723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4627802Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4628120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4628224Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4628559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4628679Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4628982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.4629103Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.4629108Z 2025-08-14T21:42:20.4629215Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4629445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4629524Z return mod(**inputs) 2025-08-14T21:42:20.4629844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4629938Z outputs = self.model( 2025-08-14T21:42:20.4630258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4630337Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4630670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4630744Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4630988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4631081Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4631411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 398, in forward 2025-08-14T21:42:20.4631520Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:42:20.4631865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.4631949Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.4631960Z 2025-08-14T21:42:20.4632064Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4632271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4632347Z return mod(**inputs) 2025-08-14T21:42:20.4632667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4632737Z outputs = self.model( 2025-08-14T21:42:20.4633070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4633147Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4633481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4633557Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4633785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4633874Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4634187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4634320Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4634650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 199, in forward 2025-08-14T21:42:20.4634808Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:42:20.4634814Z 2025-08-14T21:42:20.4634927Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4635155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4635223Z return mod(**inputs) 2025-08-14T21:42:20.4635553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4635622Z outputs = self.model( 2025-08-14T21:42:20.4635948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4636024Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4636357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4636442Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4636690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4636782Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4637100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4637208Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4637531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 218, in forward 2025-08-14T21:42:20.4637744Z key_states = self.k_proj(current_states) 2025-08-14T21:42:20.4637751Z 2025-08-14T21:42:20.4637869Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4638093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4638164Z return mod(**inputs) 2025-08-14T21:42:20.4638506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4638581Z outputs = self.model( 2025-08-14T21:42:20.4638909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4638997Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4639326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4639412Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4639650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4639735Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4640068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4640183Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4640516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 219, in forward 2025-08-14T21:42:20.4640617Z value_states = self.v_proj(current_states) 2025-08-14T21:42:20.4640620Z 2025-08-14T21:42:20.4640706Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4640799Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4640928Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4641009Z cudagraph partition due to non gpu ops 2025-08-14T21:42:20.4641129Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4641343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4641413Z return mod(**inputs) 2025-08-14T21:42:20.4641752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4641855Z outputs = self.model( 2025-08-14T21:42:20.4642200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4642280Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4642618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4642704Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4642973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4643067Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4643416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4643531Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4643987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4644100Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4644409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:42:20.4644563Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:20.4644568Z 2025-08-14T21:42:20.4644677Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4644899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4644971Z return mod(**inputs) 2025-08-14T21:42:20.4645302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4645396Z outputs = self.model( 2025-08-14T21:42:20.4645713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4645799Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4646116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4646194Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4646432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4646516Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4646838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4646950Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4647265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 237, in forward 2025-08-14T21:42:20.4647375Z attn_output, attn_weights = attention_interface( 2025-08-14T21:42:20.4647674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:42:20.4647810Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:42:20.4647821Z 2025-08-14T21:42:20.4647927Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4648137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4648218Z return mod(**inputs) 2025-08-14T21:42:20.4648543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4648613Z outputs = self.model( 2025-08-14T21:42:20.4648965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4649038Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4649344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4649417Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4649632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4649736Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4650034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 415, in forward 2025-08-14T21:42:20.4650162Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:42:20.4669994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 251, in forward 2025-08-14T21:42:20.4670251Z attn_output = self.out_proj(attn_output) 2025-08-14T21:42:20.4670259Z 2025-08-14T21:42:20.4670411Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4670651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4670734Z return mod(**inputs) 2025-08-14T21:42:20.4671116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4671199Z outputs = self.model( 2025-08-14T21:42:20.4671541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4671629Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4671962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4672055Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4672296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4672387Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4672720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:42:20.4672855Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4672862Z 2025-08-14T21:42:20.4672984Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4673197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4673270Z return mod(**inputs) 2025-08-14T21:42:20.4673605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4673683Z outputs = self.model( 2025-08-14T21:42:20.4674010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4674094Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4674516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4674607Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4674844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4674930Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4675262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 430, in forward 2025-08-14T21:42:20.4675429Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:42:20.4675667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:20.4675745Z return self.act(input) 2025-08-14T21:42:20.4675749Z 2025-08-14T21:42:20.4675860Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4676090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4676162Z return mod(**inputs) 2025-08-14T21:42:20.4676524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1375, in forward 2025-08-14T21:42:20.4676600Z outputs = self.model( 2025-08-14T21:42:20.4676949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1213, in forward 2025-08-14T21:42:20.4677042Z decoder_outputs = self.decoder( 2025-08-14T21:42:20.4677368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1057, in forward 2025-08-14T21:42:20.4677446Z layer_outputs = decoder_layer( 2025-08-14T21:42:20.4677689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:20.4677776Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:20.4678107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 432, in forward 2025-08-14T21:42:20.4678195Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:42:20.4678199Z 2025-08-14T21:42:20.4678310Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4678529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4678600Z return mod(**inputs) 2025-08-14T21:42:20.4678928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1393, in forward 2025-08-14T21:42:20.4679056Z lm_logits = self.lm_head(outputs[0]) + self.final_logits_bias 2025-08-14T21:42:20.4679060Z 2025-08-14T21:42:20.4679169Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:20.4679386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:20.4679454Z return mod(**inputs) 2025-08-14T21:42:20.4679787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/blenderbot_small/modeling_blenderbot_small.py", line 1398, in forward 2025-08-14T21:42:20.4679968Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:42:20.4679974Z 2025-08-14T21:42:30.1263839Z Compilation time (from dynamo_timed): 20.541826846 2025-08-14T21:42:30.1278333Z pass 2025-08-14T21:42:30.1279180Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:42:30.1280096Z TIMING: _recursive_pre_grad_passes:0.01093 _recursive_joint_graph_passes:0.58565 _recursive_post_grad_passes:0.12807 async_compile.wait:0.81785 code_gen:8.99622 inductor_compile:11.41319 backend_compile:16.66352 gc:0.00034 entire_frame_compile:20.54183 total_wall_time:20.54183 2025-08-14T21:42:30.1281406Z STATS: call_* op count: 652 | FakeTensorMode.__torch_dispatch__:22579 | FakeTensor.__torch_dispatch__:8019 | ProxyTorchDispatchMode.__torch_dispatch__:8304 2025-08-14T21:42:30.1281955Z Dynamo produced 1 graphs covering 652 ops with 0 graph breaks (0 unique) 2025-08-14T21:42:35.5927020Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:42:35.5928312Z from pkg_resources import resource_filename 2025-08-14T21:42:36.1886397Z 2025-08-14T21:42:37.6174963Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:42:37.6179736Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:42:37.6190780Z cpu eval CamemBert 2025-08-14T21:42:38.1521425Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:42:38.4001383Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:42:38.6764769Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:42:46.3615842Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3621632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3622118Z return mod(**inputs) 2025-08-14T21:42:46.3622576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3623044Z outputs = self.roberta( 2025-08-14T21:42:46.3623481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 886, in forward 2025-08-14T21:42:46.3623932Z embedding_output = self.embeddings( 2025-08-14T21:42:46.3624376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 90, in forward 2025-08-14T21:42:46.3624958Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-14T21:42:46.3625615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1590, in create_position_ids_from_input_ids 2025-08-14T21:42:46.3626149Z mask = input_ids.ne(padding_idx).int() 2025-08-14T21:42:46.3626300Z 2025-08-14T21:42:46.3626389Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.3626623Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.3626842Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.3627051Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.3627266Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.3627476Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.3627690Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.3627894Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.3628105Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.3628317Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.3628521Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.3628731Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.3628979Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3629363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3629746Z return mod(**inputs) 2025-08-14T21:42:46.3630161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3630595Z outputs = self.roberta( 2025-08-14T21:42:46.3630999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 886, in forward 2025-08-14T21:42:46.3631517Z embedding_output = self.embeddings( 2025-08-14T21:42:46.3631966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 90, in forward 2025-08-14T21:42:46.3632550Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-14T21:42:46.3633206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1591, in create_position_ids_from_input_ids 2025-08-14T21:42:46.3633889Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-14T21:42:46.3634142Z 2025-08-14T21:42:46.3634271Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3634658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3635012Z return mod(**inputs) 2025-08-14T21:42:46.3635431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3635862Z outputs = self.roberta( 2025-08-14T21:42:46.3636363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 886, in forward 2025-08-14T21:42:46.3636808Z embedding_output = self.embeddings( 2025-08-14T21:42:46.3637270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 90, in forward 2025-08-14T21:42:46.3638057Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-14T21:42:46.3638700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1591, in create_position_ids_from_input_ids 2025-08-14T21:42:46.3639347Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-14T21:42:46.3639630Z 2025-08-14T21:42:46.3639754Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3640167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3640527Z return mod(**inputs) 2025-08-14T21:42:46.3640953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3641413Z outputs = self.roberta( 2025-08-14T21:42:46.3641832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3642285Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3642732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3643187Z layer_outputs = layer_module( 2025-08-14T21:42:46.3643584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3644008Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3644472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3644927Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3645529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3645949Z return func(*args, **kwargs) 2025-08-14T21:42:46.3646380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.3646830Z self_outputs = self.self( 2025-08-14T21:42:46.3647229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3647706Z return func(*args, **kwargs) 2025-08-14T21:42:46.3648127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:42:46.3648728Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:46.3649020Z 2025-08-14T21:42:46.3649140Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3649540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3649921Z return mod(**inputs) 2025-08-14T21:42:46.3650338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3650779Z outputs = self.roberta( 2025-08-14T21:42:46.3651190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3651638Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3652080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3652545Z layer_outputs = layer_module( 2025-08-14T21:42:46.3652918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3653341Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3653792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3654243Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3654660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3655064Z return func(*args, **kwargs) 2025-08-14T21:42:46.3655492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.3655929Z self_outputs = self.self( 2025-08-14T21:42:46.3656307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3656695Z return func(*args, **kwargs) 2025-08-14T21:42:46.3657105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:42:46.3657527Z self.key(current_states) 2025-08-14T21:42:46.3657655Z 2025-08-14T21:42:46.3657764Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3658138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3658475Z return mod(**inputs) 2025-08-14T21:42:46.3658878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3659305Z outputs = self.roberta( 2025-08-14T21:42:46.3659713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3660133Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3660555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3660984Z layer_outputs = layer_module( 2025-08-14T21:42:46.3661338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3661721Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3662147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3662582Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3662997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3663391Z return func(*args, **kwargs) 2025-08-14T21:42:46.3663805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.3664226Z self_outputs = self.self( 2025-08-14T21:42:46.3664597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3664984Z return func(*args, **kwargs) 2025-08-14T21:42:46.3665409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:42:46.3665827Z self.value(current_states) 2025-08-14T21:42:46.3665955Z 2025-08-14T21:42:46.3666040Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.3666292Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3666669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3666999Z return mod(**inputs) 2025-08-14T21:42:46.3667417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3667844Z outputs = self.roberta( 2025-08-14T21:42:46.3668255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3668690Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3669134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3669592Z layer_outputs = layer_module( 2025-08-14T21:42:46.3669965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3670366Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3670800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3671237Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3671643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3672044Z return func(*args, **kwargs) 2025-08-14T21:42:46.3672463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.3672889Z self_outputs = self.self( 2025-08-14T21:42:46.3673272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3673674Z return func(*args, **kwargs) 2025-08-14T21:42:46.3674093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:42:46.3674586Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:46.3674791Z 2025-08-14T21:42:46.3674903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3675284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3675620Z return mod(**inputs) 2025-08-14T21:42:46.3676034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3676465Z outputs = self.roberta( 2025-08-14T21:42:46.3676851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3677253Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3677662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3678087Z layer_outputs = layer_module( 2025-08-14T21:42:46.3678424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3678784Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3679192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3679607Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3679982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3680374Z return func(*args, **kwargs) 2025-08-14T21:42:46.3680766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:42:46.3681222Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:46.3681668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:42:46.3682102Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.3682241Z 2025-08-14T21:42:46.3682351Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3682698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3683036Z return mod(**inputs) 2025-08-14T21:42:46.3683417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3683822Z outputs = self.roberta( 2025-08-14T21:42:46.3684201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3684607Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3685081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3685547Z layer_outputs = layer_module( 2025-08-14T21:42:46.3685923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3686320Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3686770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.3687215Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.3687640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.3688361Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.3688819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.3689334Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.3689813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:42:46.3690231Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.3690369Z 2025-08-14T21:42:46.3690480Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3690836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3691168Z return mod(**inputs) 2025-08-14T21:42:46.3691559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3691951Z outputs = self.roberta( 2025-08-14T21:42:46.3692336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3692773Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3693183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3693590Z layer_outputs = layer_module( 2025-08-14T21:42:46.3693949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3694316Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3694731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.3695171Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.3695568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.3695961Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.3696384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.3696865Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.3697357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:42:46.3697799Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:46.3698191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:46.3698552Z return self.act(input) 2025-08-14T21:42:46.3698670Z 2025-08-14T21:42:46.3698788Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3699165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3699498Z return mod(**inputs) 2025-08-14T21:42:46.3699903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3700329Z outputs = self.roberta( 2025-08-14T21:42:46.3700726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3701159Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3701589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3702026Z layer_outputs = layer_module( 2025-08-14T21:42:46.3702391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3702767Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3703200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.3703643Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.3704067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.3704487Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.3704952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:42:46.3705475Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:46.3705970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:42:46.3706418Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.3706589Z 2025-08-14T21:42:46.3706700Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3707085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3707453Z return mod(**inputs) 2025-08-14T21:42:46.3707857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3708300Z outputs = self.roberta( 2025-08-14T21:42:46.3708703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3709137Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3709567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3710033Z layer_outputs = layer_module( 2025-08-14T21:42:46.3710389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3710784Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3711211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3711653Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3712073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3712471Z return func(*args, **kwargs) 2025-08-14T21:42:46.3712925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.3713375Z self_outputs = self.self( 2025-08-14T21:42:46.3713755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3714147Z return func(*args, **kwargs) 2025-08-14T21:42:46.3714561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:42:46.3715106Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:46.3715376Z 2025-08-14T21:42:46.3715482Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3715844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3716167Z return mod(**inputs) 2025-08-14T21:42:46.3716546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3716949Z outputs = self.roberta( 2025-08-14T21:42:46.3717332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3717733Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3718128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3718560Z layer_outputs = layer_module( 2025-08-14T21:42:46.3718924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3719293Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3719722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3720156Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3720556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3720942Z return func(*args, **kwargs) 2025-08-14T21:42:46.3721356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.3721780Z self_outputs = self.self( 2025-08-14T21:42:46.3722145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3722566Z return func(*args, **kwargs) 2025-08-14T21:42:46.3722984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:42:46.3723409Z self.key(current_states) 2025-08-14T21:42:46.3723531Z 2025-08-14T21:42:46.3723641Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3724030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3724376Z return mod(**inputs) 2025-08-14T21:42:46.3724795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3725305Z outputs = self.roberta( 2025-08-14T21:42:46.3725730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3726180Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3726610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3727083Z layer_outputs = layer_module( 2025-08-14T21:42:46.3727467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3727837Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3728295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3728738Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3729140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3729520Z return func(*args, **kwargs) 2025-08-14T21:42:46.3729932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.3730360Z self_outputs = self.self( 2025-08-14T21:42:46.3730736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3731120Z return func(*args, **kwargs) 2025-08-14T21:42:46.3731550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:42:46.3731975Z self.value(current_states) 2025-08-14T21:42:46.3732103Z 2025-08-14T21:42:46.3732200Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.3732445Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3732821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3733160Z return mod(**inputs) 2025-08-14T21:42:46.3733560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3733988Z outputs = self.roberta( 2025-08-14T21:42:46.3734411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3734835Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3735262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3735687Z layer_outputs = layer_module( 2025-08-14T21:42:46.3736052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3736419Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3736862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3737305Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3737870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3738276Z return func(*args, **kwargs) 2025-08-14T21:42:46.3738699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.3739130Z self_outputs = self.self( 2025-08-14T21:42:46.3739504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3739947Z return func(*args, **kwargs) 2025-08-14T21:42:46.3740360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:42:46.3740839Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:46.3741025Z 2025-08-14T21:42:46.3741134Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3741516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3741866Z return mod(**inputs) 2025-08-14T21:42:46.3742282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3742694Z outputs = self.roberta( 2025-08-14T21:42:46.3743110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3743521Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3743915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3744314Z layer_outputs = layer_module( 2025-08-14T21:42:46.3744660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3745021Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3745420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3745830Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3746228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3746623Z return func(*args, **kwargs) 2025-08-14T21:42:46.3747010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:42:46.3747469Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:46.3747922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:42:46.3748337Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.3748489Z 2025-08-14T21:42:46.3748600Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3748978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3749321Z return mod(**inputs) 2025-08-14T21:42:46.3749718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3750145Z outputs = self.roberta( 2025-08-14T21:42:46.3750543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3750947Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3751338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3751742Z layer_outputs = layer_module( 2025-08-14T21:42:46.3752089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3752493Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3752920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.3753360Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.3753785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.3754197Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.3754674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.3755146Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.3755586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:42:46.3755997Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.3756141Z 2025-08-14T21:42:46.3756245Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3756623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3756947Z return mod(**inputs) 2025-08-14T21:42:46.3757351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3757763Z outputs = self.roberta( 2025-08-14T21:42:46.3758153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3758572Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3758994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3759427Z layer_outputs = layer_module( 2025-08-14T21:42:46.3759787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3760169Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3760603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.3761045Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.3761466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.3761884Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.3762346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.3762856Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.3763328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:42:46.3763801Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:46.3764208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:46.3764561Z return self.act(input) 2025-08-14T21:42:46.3764688Z 2025-08-14T21:42:46.3764801Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3765249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3765617Z return mod(**inputs) 2025-08-14T21:42:46.3766041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3766472Z outputs = self.roberta( 2025-08-14T21:42:46.3766880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3767336Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3767765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3768196Z layer_outputs = layer_module( 2025-08-14T21:42:46.3768566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3768946Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3769382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.3769843Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.3770266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.3770669Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.3771125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:42:46.3771665Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:46.3772150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:42:46.3772610Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.3772764Z 2025-08-14T21:42:46.3772873Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3773248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3773582Z return mod(**inputs) 2025-08-14T21:42:46.3773989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3774388Z outputs = self.roberta( 2025-08-14T21:42:46.3774773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3775170Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3775563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3775961Z layer_outputs = layer_module( 2025-08-14T21:42:46.3776297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3776655Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3777059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3777467Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3777850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3778247Z return func(*args, **kwargs) 2025-08-14T21:42:46.3778663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.3779102Z self_outputs = self.self( 2025-08-14T21:42:46.3779477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3779875Z return func(*args, **kwargs) 2025-08-14T21:42:46.3780290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:42:46.3780861Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:46.3781131Z 2025-08-14T21:42:46.3781235Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3781593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3781940Z return mod(**inputs) 2025-08-14T21:42:46.3782349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3782780Z outputs = self.roberta( 2025-08-14T21:42:46.3783185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3783612Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3784027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3784490Z layer_outputs = layer_module( 2025-08-14T21:42:46.3784854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3785234Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3785670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3786116Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3786551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3786952Z return func(*args, **kwargs) 2025-08-14T21:42:46.3787388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.3787836Z self_outputs = self.self( 2025-08-14T21:42:46.3788209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3788604Z return func(*args, **kwargs) 2025-08-14T21:42:46.3789019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:42:46.3789460Z self.key(current_states) 2025-08-14T21:42:46.3789581Z 2025-08-14T21:42:46.3789689Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3790072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3790416Z return mod(**inputs) 2025-08-14T21:42:46.3790809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3791236Z outputs = self.roberta( 2025-08-14T21:42:46.3791716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3792166Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3792587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3793042Z layer_outputs = layer_module( 2025-08-14T21:42:46.3793409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3793796Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3794232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3794672Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3795073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3795468Z return func(*args, **kwargs) 2025-08-14T21:42:46.3795882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.3796316Z self_outputs = self.self( 2025-08-14T21:42:46.3796690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3797098Z return func(*args, **kwargs) 2025-08-14T21:42:46.3797525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:42:46.3797963Z self.value(current_states) 2025-08-14T21:42:46.3798086Z 2025-08-14T21:42:46.3798179Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.3798422Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3798797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3799163Z return mod(**inputs) 2025-08-14T21:42:46.3799561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3799965Z outputs = self.roberta( 2025-08-14T21:42:46.3800349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3800753Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3801150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3801572Z layer_outputs = layer_module( 2025-08-14T21:42:46.3801918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3802300Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3802734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3803171Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3803573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3803967Z return func(*args, **kwargs) 2025-08-14T21:42:46.3804397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.3804825Z self_outputs = self.self( 2025-08-14T21:42:46.3805291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3805695Z return func(*args, **kwargs) 2025-08-14T21:42:46.3806120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:42:46.3806604Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:46.3806790Z 2025-08-14T21:42:46.3806896Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3807258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3807584Z return mod(**inputs) 2025-08-14T21:42:46.3807965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3808364Z outputs = self.roberta( 2025-08-14T21:42:46.3808747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3809154Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3809544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3809949Z layer_outputs = layer_module( 2025-08-14T21:42:46.3810292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3810650Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3811044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3811452Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3811857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3812213Z return func(*args, **kwargs) 2025-08-14T21:42:46.3812605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:42:46.3813062Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:46.3813513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:42:46.3813945Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.3814088Z 2025-08-14T21:42:46.3814190Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3814545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3814868Z return mod(**inputs) 2025-08-14T21:42:46.3815241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3815658Z outputs = self.roberta( 2025-08-14T21:42:46.3816091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3816500Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3816969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3817401Z layer_outputs = layer_module( 2025-08-14T21:42:46.3817764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3818133Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3818561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.3819004Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.3819425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.3819844Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.3820297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.3820810Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.3821273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:42:46.3821714Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.3821864Z 2025-08-14T21:42:46.3821972Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3822350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3822687Z return mod(**inputs) 2025-08-14T21:42:46.3823089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3823513Z outputs = self.roberta( 2025-08-14T21:42:46.3823917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3824349Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3824771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3825200Z layer_outputs = layer_module( 2025-08-14T21:42:46.3825559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3825934Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3826379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.3826843Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.3827260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.3827675Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.3828134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.3828664Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.3829130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:42:46.3829602Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:46.3830005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:46.3830362Z return self.act(input) 2025-08-14T21:42:46.3830489Z 2025-08-14T21:42:46.3830595Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3830998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3831343Z return mod(**inputs) 2025-08-14T21:42:46.3831755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3832187Z outputs = self.roberta( 2025-08-14T21:42:46.3832591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3833008Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3833404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3833808Z layer_outputs = layer_module( 2025-08-14T21:42:46.3834155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3834502Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3834910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.3835322Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.3835720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.3836108Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.3836542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:42:46.3837035Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:46.3837487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:42:46.3838037Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.3838188Z 2025-08-14T21:42:46.3838296Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3838681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3839025Z return mod(**inputs) 2025-08-14T21:42:46.3839434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3839870Z outputs = self.roberta( 2025-08-14T21:42:46.3840286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3840739Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3841189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3841685Z layer_outputs = layer_module( 2025-08-14T21:42:46.3842059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3842439Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3842868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3843318Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3843753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3844178Z return func(*args, **kwargs) 2025-08-14T21:42:46.3844604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.3845109Z self_outputs = self.self( 2025-08-14T21:42:46.3845529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3845932Z return func(*args, **kwargs) 2025-08-14T21:42:46.3846438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:42:46.3847032Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:46.3847327Z 2025-08-14T21:42:46.3847438Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3847821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3848164Z return mod(**inputs) 2025-08-14T21:42:46.3848565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3848990Z outputs = self.roberta( 2025-08-14T21:42:46.3849399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3849819Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3850249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3850679Z layer_outputs = layer_module( 2025-08-14T21:42:46.3851048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3851419Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3851846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3852285Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3852681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3853069Z return func(*args, **kwargs) 2025-08-14T21:42:46.3853483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.3853906Z self_outputs = self.self( 2025-08-14T21:42:46.3854276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3854666Z return func(*args, **kwargs) 2025-08-14T21:42:46.3855082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:42:46.3855512Z self.key(current_states) 2025-08-14T21:42:46.3855634Z 2025-08-14T21:42:46.3855743Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3856129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3856504Z return mod(**inputs) 2025-08-14T21:42:46.3856899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3857332Z outputs = self.roberta( 2025-08-14T21:42:46.3857739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3858180Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3858595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3859048Z layer_outputs = layer_module( 2025-08-14T21:42:46.3859418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3859799Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3860222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3860665Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3861097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3861460Z return func(*args, **kwargs) 2025-08-14T21:42:46.3861873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.3862280Z self_outputs = self.self( 2025-08-14T21:42:46.3862641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3863004Z return func(*args, **kwargs) 2025-08-14T21:42:46.3863394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:42:46.3863802Z self.value(current_states) 2025-08-14T21:42:46.3863921Z 2025-08-14T21:42:46.3864003Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.3864244Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3864604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3864932Z return mod(**inputs) 2025-08-14T21:42:46.3865308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3865712Z outputs = self.roberta( 2025-08-14T21:42:46.3866102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3866497Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3866892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3867294Z layer_outputs = layer_module( 2025-08-14T21:42:46.3867639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3867994Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3868430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3868868Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3869267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3869650Z return func(*args, **kwargs) 2025-08-14T21:42:46.3870064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.3870491Z self_outputs = self.self( 2025-08-14T21:42:46.3870862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3871272Z return func(*args, **kwargs) 2025-08-14T21:42:46.3871696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:42:46.3872154Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:46.3872341Z 2025-08-14T21:42:46.3872442Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3872802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3873151Z return mod(**inputs) 2025-08-14T21:42:46.3873534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3873934Z outputs = self.roberta( 2025-08-14T21:42:46.3874322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3874733Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3875127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3875551Z layer_outputs = layer_module( 2025-08-14T21:42:46.3875896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3876265Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3876665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3877079Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3877460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3877819Z return func(*args, **kwargs) 2025-08-14T21:42:46.3878209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:42:46.3878687Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:46.3879176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:42:46.3879612Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.3879766Z 2025-08-14T21:42:46.3879874Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3880252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3880601Z return mod(**inputs) 2025-08-14T21:42:46.3880996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3881423Z outputs = self.roberta( 2025-08-14T21:42:46.3881837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3882258Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3882699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3883137Z layer_outputs = layer_module( 2025-08-14T21:42:46.3883501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3883869Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3884299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.3884752Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.3885244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.3885672Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.3886165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.3886699Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.3887177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:42:46.3887635Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.3887789Z 2025-08-14T21:42:46.3887900Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3888316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3888658Z return mod(**inputs) 2025-08-14T21:42:46.3889066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3889496Z outputs = self.roberta( 2025-08-14T21:42:46.3889896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3890349Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3890769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3891191Z layer_outputs = layer_module( 2025-08-14T21:42:46.3891569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3891952Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3892383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.3892821Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.3893236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.3893655Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.3894115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.3894614Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.3895089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:42:46.3895556Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:46.3895939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:46.3896283Z return self.act(input) 2025-08-14T21:42:46.3896408Z 2025-08-14T21:42:46.3896518Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3896893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3897241Z return mod(**inputs) 2025-08-14T21:42:46.3897640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3898069Z outputs = self.roberta( 2025-08-14T21:42:46.3898477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3898898Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3899337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3899764Z layer_outputs = layer_module( 2025-08-14T21:42:46.3900134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3900506Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3902020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.3902485Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.3902911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.3903321Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.3903779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:42:46.3904332Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:46.3904827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:42:46.3905282Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.3905434Z 2025-08-14T21:42:46.3905545Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3905928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3906291Z return mod(**inputs) 2025-08-14T21:42:46.3906717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3907150Z outputs = self.roberta( 2025-08-14T21:42:46.3907583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3908008Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3908433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3908869Z layer_outputs = layer_module( 2025-08-14T21:42:46.3909224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3909605Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3910034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3910470Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3910864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3911255Z return func(*args, **kwargs) 2025-08-14T21:42:46.3911666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.3912085Z self_outputs = self.self( 2025-08-14T21:42:46.3912468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3912856Z return func(*args, **kwargs) 2025-08-14T21:42:46.3913267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:42:46.3913831Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:46.3914115Z 2025-08-14T21:42:46.3914226Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3914607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3914952Z return mod(**inputs) 2025-08-14T21:42:46.3915353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3915795Z outputs = self.roberta( 2025-08-14T21:42:46.3916215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3916636Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3917086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3917506Z layer_outputs = layer_module( 2025-08-14T21:42:46.3917873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3918245Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3918678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3919153Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3919553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3919934Z return func(*args, **kwargs) 2025-08-14T21:42:46.3920347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.3920773Z self_outputs = self.self( 2025-08-14T21:42:46.3921148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3921565Z return func(*args, **kwargs) 2025-08-14T21:42:46.3921987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:42:46.3922445Z self.key(current_states) 2025-08-14T21:42:46.3922568Z 2025-08-14T21:42:46.3922675Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3923053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3923401Z return mod(**inputs) 2025-08-14T21:42:46.3923803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3924240Z outputs = self.roberta( 2025-08-14T21:42:46.3924654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3925189Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3925635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3926076Z layer_outputs = layer_module( 2025-08-14T21:42:46.3926460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3926844Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3927268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3927708Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3928120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3928515Z return func(*args, **kwargs) 2025-08-14T21:42:46.3928955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.3929409Z self_outputs = self.self( 2025-08-14T21:42:46.3929800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3930206Z return func(*args, **kwargs) 2025-08-14T21:42:46.3930631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:42:46.3931074Z self.value(current_states) 2025-08-14T21:42:46.3931201Z 2025-08-14T21:42:46.3931291Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.3931553Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3931948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3932330Z return mod(**inputs) 2025-08-14T21:42:46.3932748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3933193Z outputs = self.roberta( 2025-08-14T21:42:46.3933617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3934057Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3934498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3934958Z layer_outputs = layer_module( 2025-08-14T21:42:46.3935347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3935729Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3936172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3936622Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3937056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3937451Z return func(*args, **kwargs) 2025-08-14T21:42:46.3938139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.3938592Z self_outputs = self.self( 2025-08-14T21:42:46.3938980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3939384Z return func(*args, **kwargs) 2025-08-14T21:42:46.3939813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:42:46.3940317Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:46.3940521Z 2025-08-14T21:42:46.3940635Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3941027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3941382Z return mod(**inputs) 2025-08-14T21:42:46.3941797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3942240Z outputs = self.roberta( 2025-08-14T21:42:46.3942660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3943113Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3943550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3943996Z layer_outputs = layer_module( 2025-08-14T21:42:46.3944374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3944763Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3945215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3945665Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3946084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3946469Z return func(*args, **kwargs) 2025-08-14T21:42:46.3946897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:42:46.3947391Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:46.3947880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:42:46.3948358Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.3948512Z 2025-08-14T21:42:46.3948622Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3949001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3949344Z return mod(**inputs) 2025-08-14T21:42:46.3949737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3950194Z outputs = self.roberta( 2025-08-14T21:42:46.3950607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3951027Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3951453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3951880Z layer_outputs = layer_module( 2025-08-14T21:42:46.3952250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3952655Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3953090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.3953559Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.3953977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.3954395Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.3954851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.3955354Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.3955819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:42:46.3956257Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.3956406Z 2025-08-14T21:42:46.3956514Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3956887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3957221Z return mod(**inputs) 2025-08-14T21:42:46.3957630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3958030Z outputs = self.roberta( 2025-08-14T21:42:46.3958406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3958807Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3959208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3959610Z layer_outputs = layer_module( 2025-08-14T21:42:46.3959951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3960331Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3960756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.3961192Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.3961607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.3962018Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.3962475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.3962993Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.3963461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:42:46.3963927Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:46.3964325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:46.3964675Z return self.act(input) 2025-08-14T21:42:46.3964800Z 2025-08-14T21:42:46.3964927Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3965401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3965756Z return mod(**inputs) 2025-08-14T21:42:46.3966168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3966612Z outputs = self.roberta( 2025-08-14T21:42:46.3967035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3967486Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3967914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3968364Z layer_outputs = layer_module( 2025-08-14T21:42:46.3968736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3969109Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3969545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.3969986Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.3970401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.3970822Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.3971257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:42:46.3971774Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:46.3972257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:42:46.3972698Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.3972852Z 2025-08-14T21:42:46.3972962Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3973337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3973673Z return mod(**inputs) 2025-08-14T21:42:46.3974075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3974492Z outputs = self.roberta( 2025-08-14T21:42:46.3974892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3975322Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3975743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3976170Z layer_outputs = layer_module( 2025-08-14T21:42:46.3976528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3976913Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3977342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3977818Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3978221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3978619Z return func(*args, **kwargs) 2025-08-14T21:42:46.3979044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.3979468Z self_outputs = self.self( 2025-08-14T21:42:46.3979855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3980269Z return func(*args, **kwargs) 2025-08-14T21:42:46.3980687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:42:46.3981258Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:46.3981552Z 2025-08-14T21:42:46.3981664Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3982058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3982426Z return mod(**inputs) 2025-08-14T21:42:46.3982827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3983256Z outputs = self.roberta( 2025-08-14T21:42:46.3983674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3984095Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3984514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3984937Z layer_outputs = layer_module( 2025-08-14T21:42:46.3985298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3985671Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3986105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3986544Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3986937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3987323Z return func(*args, **kwargs) 2025-08-14T21:42:46.3987735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.3988157Z self_outputs = self.self( 2025-08-14T21:42:46.3988527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3988918Z return func(*args, **kwargs) 2025-08-14T21:42:46.3989335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:42:46.3989764Z self.key(current_states) 2025-08-14T21:42:46.3989886Z 2025-08-14T21:42:46.3989998Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3990376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3990719Z return mod(**inputs) 2025-08-14T21:42:46.3991117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3991547Z outputs = self.roberta( 2025-08-14T21:42:46.3991952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.3992378Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.3992802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.3993232Z layer_outputs = layer_module( 2025-08-14T21:42:46.3993582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.3993935Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.3994347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.3994762Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.3995162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3995522Z return func(*args, **kwargs) 2025-08-14T21:42:46.3995911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.3996310Z self_outputs = self.self( 2025-08-14T21:42:46.3996670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.3997033Z return func(*args, **kwargs) 2025-08-14T21:42:46.3997444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:42:46.3997852Z self.value(current_states) 2025-08-14T21:42:46.3997967Z 2025-08-14T21:42:46.3998068Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.3998308Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.3998668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.3998993Z return mod(**inputs) 2025-08-14T21:42:46.3999393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.3999818Z outputs = self.roberta( 2025-08-14T21:42:46.4000238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4000635Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4001045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4001471Z layer_outputs = layer_module( 2025-08-14T21:42:46.4001835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4002207Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4002639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4003076Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4003466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4003857Z return func(*args, **kwargs) 2025-08-14T21:42:46.4004272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4004699Z self_outputs = self.self( 2025-08-14T21:42:46.4005153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4005572Z return func(*args, **kwargs) 2025-08-14T21:42:46.4006007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:42:46.4006515Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:46.4006712Z 2025-08-14T21:42:46.4006825Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4007215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4007607Z return mod(**inputs) 2025-08-14T21:42:46.4008020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4008467Z outputs = self.roberta( 2025-08-14T21:42:46.4008890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4009338Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4009768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4010233Z layer_outputs = layer_module( 2025-08-14T21:42:46.4010618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4011013Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4011453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4011915Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4012344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4012741Z return func(*args, **kwargs) 2025-08-14T21:42:46.4013183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:42:46.4013675Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:46.4014132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:42:46.4014539Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.4014683Z 2025-08-14T21:42:46.4014785Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4015143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4015467Z return mod(**inputs) 2025-08-14T21:42:46.4015841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4016247Z outputs = self.roberta( 2025-08-14T21:42:46.4016629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4017026Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4017428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4017829Z layer_outputs = layer_module( 2025-08-14T21:42:46.4018171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4018521Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4018926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.4019341Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.4019738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.4020135Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.4020572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.4021047Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.4021486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:42:46.4021898Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.4022064Z 2025-08-14T21:42:46.4022167Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4022527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4022847Z return mod(**inputs) 2025-08-14T21:42:46.4023231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4023637Z outputs = self.roberta( 2025-08-14T21:42:46.4024014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4024442Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4024844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4025246Z layer_outputs = layer_module( 2025-08-14T21:42:46.4025582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4025944Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4026368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.4026788Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.4027205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.4027603Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.4028036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.4028508Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.4028982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:42:46.4029456Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:46.4029858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:46.4030209Z return self.act(input) 2025-08-14T21:42:46.4030336Z 2025-08-14T21:42:46.4030446Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4030831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4031174Z return mod(**inputs) 2025-08-14T21:42:46.4031573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4031998Z outputs = self.roberta( 2025-08-14T21:42:46.4032402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4032834Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4033257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4033691Z layer_outputs = layer_module( 2025-08-14T21:42:46.4034058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4034442Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4034887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.4035334Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.4035750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.4036185Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.4036642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:42:46.4037202Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:46.4037826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:42:46.4038293Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.4038450Z 2025-08-14T21:42:46.4038563Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4038946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4039070Z return mod(**inputs) 2025-08-14T21:42:46.4039366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4039450Z outputs = self.roberta( 2025-08-14T21:42:46.4039758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4039842Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4040180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4040258Z layer_outputs = layer_module( 2025-08-14T21:42:46.4040502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4040619Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4040916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4041014Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4041268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4041352Z return func(*args, **kwargs) 2025-08-14T21:42:46.4041653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4041729Z self_outputs = self.self( 2025-08-14T21:42:46.4041989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4042064Z return func(*args, **kwargs) 2025-08-14T21:42:46.4042345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:42:46.4042572Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:46.4042577Z 2025-08-14T21:42:46.4042686Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4042902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4042970Z return mod(**inputs) 2025-08-14T21:42:46.4043257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4043340Z outputs = self.roberta( 2025-08-14T21:42:46.4043622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4043707Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4044003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4044081Z layer_outputs = layer_module( 2025-08-14T21:42:46.4044318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4044400Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4044752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4044876Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4045192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4045284Z return func(*args, **kwargs) 2025-08-14T21:42:46.4045580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4045660Z self_outputs = self.self( 2025-08-14T21:42:46.4045926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4046031Z return func(*args, **kwargs) 2025-08-14T21:42:46.4046316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:42:46.4046400Z self.key(current_states) 2025-08-14T21:42:46.4046405Z 2025-08-14T21:42:46.4046513Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4046731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4046800Z return mod(**inputs) 2025-08-14T21:42:46.4047106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4047189Z outputs = self.roberta( 2025-08-14T21:42:46.4047493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4047585Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4047875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4047951Z layer_outputs = layer_module( 2025-08-14T21:42:46.4048195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4048278Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4048566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4048660Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4048913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4048997Z return func(*args, **kwargs) 2025-08-14T21:42:46.4049291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4049367Z self_outputs = self.self( 2025-08-14T21:42:46.4049627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4049698Z return func(*args, **kwargs) 2025-08-14T21:42:46.4049988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:42:46.4050065Z self.value(current_states) 2025-08-14T21:42:46.4050069Z 2025-08-14T21:42:46.4050156Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.4050271Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4050482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4050553Z return mod(**inputs) 2025-08-14T21:42:46.4050848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4050930Z outputs = self.roberta( 2025-08-14T21:42:46.4051205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4051278Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4051546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4051650Z layer_outputs = layer_module( 2025-08-14T21:42:46.4051881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4051962Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4052254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4052337Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4052660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4052731Z return func(*args, **kwargs) 2025-08-14T21:42:46.4053030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4053112Z self_outputs = self.self( 2025-08-14T21:42:46.4053362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4053457Z return func(*args, **kwargs) 2025-08-14T21:42:46.4053743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:42:46.4053897Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:46.4053901Z 2025-08-14T21:42:46.4054017Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4054237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4054305Z return mod(**inputs) 2025-08-14T21:42:46.4054593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4054663Z outputs = self.roberta( 2025-08-14T21:42:46.4054943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4055019Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4055289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4055370Z layer_outputs = layer_module( 2025-08-14T21:42:46.4055591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4055677Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4055949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4056029Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4056272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4056341Z return func(*args, **kwargs) 2025-08-14T21:42:46.4056613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:42:46.4056750Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:46.4057021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:42:46.4057112Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.4057117Z 2025-08-14T21:42:46.4057220Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4057418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4057490Z return mod(**inputs) 2025-08-14T21:42:46.4057761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4057853Z outputs = self.roberta( 2025-08-14T21:42:46.4058122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4058194Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4058464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4058535Z layer_outputs = layer_module( 2025-08-14T21:42:46.4058754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4058888Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4059162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.4059253Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.4059516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.4059592Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.4059931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.4060051Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.4060342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:42:46.4060427Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.4060430Z 2025-08-14T21:42:46.4060530Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4060733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4060799Z return mod(**inputs) 2025-08-14T21:42:46.4061069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4061145Z outputs = self.roberta( 2025-08-14T21:42:46.4061409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4061489Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4061759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4061831Z layer_outputs = layer_module( 2025-08-14T21:42:46.4062056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4062132Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4062396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.4062489Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.4062747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.4062831Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.4063127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.4063245Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.4063520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:42:46.4063631Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:46.4063856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:46.4063925Z return self.act(input) 2025-08-14T21:42:46.4063947Z 2025-08-14T21:42:46.4064049Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4064253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4064319Z return mod(**inputs) 2025-08-14T21:42:46.4064584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4064659Z outputs = self.roberta( 2025-08-14T21:42:46.4064923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4065023Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4065296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4065367Z layer_outputs = layer_module( 2025-08-14T21:42:46.4065594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4065674Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4065966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.4066051Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.4066330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.4066415Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.4066709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:42:46.4066837Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:46.4067106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:42:46.4067186Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.4067189Z 2025-08-14T21:42:46.4067296Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4067489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4067555Z return mod(**inputs) 2025-08-14T21:42:46.4067834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4067901Z outputs = self.roberta( 2025-08-14T21:42:46.4068170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4068241Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4068508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4068587Z layer_outputs = layer_module( 2025-08-14T21:42:46.4068807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4068886Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4069166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4069248Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4069494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4069565Z return func(*args, **kwargs) 2025-08-14T21:42:46.4069835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4069914Z self_outputs = self.self( 2025-08-14T21:42:46.4070152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4070247Z return func(*args, **kwargs) 2025-08-14T21:42:46.4070528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:42:46.4070730Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:46.4070733Z 2025-08-14T21:42:46.4070842Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4071038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4071118Z return mod(**inputs) 2025-08-14T21:42:46.4071402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4071471Z outputs = self.roberta( 2025-08-14T21:42:46.4071752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4071827Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4072122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4072204Z layer_outputs = layer_module( 2025-08-14T21:42:46.4072453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4072542Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4072829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4072907Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4073145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4073211Z return func(*args, **kwargs) 2025-08-14T21:42:46.4073475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4073550Z self_outputs = self.self( 2025-08-14T21:42:46.4073786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4073862Z return func(*args, **kwargs) 2025-08-14T21:42:46.4074127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:42:46.4074200Z self.key(current_states) 2025-08-14T21:42:46.4074204Z 2025-08-14T21:42:46.4074310Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4074505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4074576Z return mod(**inputs) 2025-08-14T21:42:46.4074848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4074918Z outputs = self.roberta( 2025-08-14T21:42:46.4075201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4075272Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4075532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4075608Z layer_outputs = layer_module( 2025-08-14T21:42:46.4075821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4075903Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4076162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4076257Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4076495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4076561Z return func(*args, **kwargs) 2025-08-14T21:42:46.4076824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4076900Z self_outputs = self.self( 2025-08-14T21:42:46.4077135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4077232Z return func(*args, **kwargs) 2025-08-14T21:42:46.4077505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:42:46.4077578Z self.value(current_states) 2025-08-14T21:42:46.4077582Z 2025-08-14T21:42:46.4077672Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.4077776Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4077981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4078062Z return mod(**inputs) 2025-08-14T21:42:46.4078346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4078427Z outputs = self.roberta( 2025-08-14T21:42:46.4078734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4078814Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4079119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4079195Z layer_outputs = layer_module( 2025-08-14T21:42:46.4079433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4079527Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4079827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4079918Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4080182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4080254Z return func(*args, **kwargs) 2025-08-14T21:42:46.4080561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4080633Z self_outputs = self.self( 2025-08-14T21:42:46.4080895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4080966Z return func(*args, **kwargs) 2025-08-14T21:42:46.4081252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:42:46.4081400Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:46.4081406Z 2025-08-14T21:42:46.4081513Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4081731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4081802Z return mod(**inputs) 2025-08-14T21:42:46.4082090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4082172Z outputs = self.roberta( 2025-08-14T21:42:46.4082469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4082546Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4082848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4082950Z layer_outputs = layer_module( 2025-08-14T21:42:46.4083188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4083271Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4083568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4083660Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4083949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4084029Z return func(*args, **kwargs) 2025-08-14T21:42:46.4084325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:42:46.4084459Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:46.4084771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:42:46.4084861Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.4084865Z 2025-08-14T21:42:46.4084971Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4085293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4085369Z return mod(**inputs) 2025-08-14T21:42:46.4085666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4085740Z outputs = self.roberta( 2025-08-14T21:42:46.4086038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4086126Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4086412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4086498Z layer_outputs = layer_module( 2025-08-14T21:42:46.4086734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4086816Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4087125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.4087217Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.4087488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.4087579Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.4087896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.4088031Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.4088317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:42:46.4088404Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.4088408Z 2025-08-14T21:42:46.4088524Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4088736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4088817Z return mod(**inputs) 2025-08-14T21:42:46.4089103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4089174Z outputs = self.roberta( 2025-08-14T21:42:46.4089476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4089571Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4089860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4089941Z layer_outputs = layer_module( 2025-08-14T21:42:46.4090171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4090261Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4090564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.4090650Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.4090928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.4091008Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.4091331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.4091470Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.4091756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:42:46.4091895Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:46.4092119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:46.4092195Z return self.act(input) 2025-08-14T21:42:46.4092208Z 2025-08-14T21:42:46.4092315Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4092524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4092603Z return mod(**inputs) 2025-08-14T21:42:46.4092892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4092965Z outputs = self.roberta( 2025-08-14T21:42:46.4093249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4093321Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4093592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4093663Z layer_outputs = layer_module( 2025-08-14T21:42:46.4093873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4093956Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4094214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.4094297Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.4094558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.4094632Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.4094929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:42:46.4095058Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:46.4095322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:42:46.4095412Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.4095416Z 2025-08-14T21:42:46.4095514Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4095715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4095798Z return mod(**inputs) 2025-08-14T21:42:46.4096074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4096150Z outputs = self.roberta( 2025-08-14T21:42:46.4096424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4096495Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4096771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4096856Z layer_outputs = layer_module( 2025-08-14T21:42:46.4097074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4097148Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4097409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4097494Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4097750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4097830Z return func(*args, **kwargs) 2025-08-14T21:42:46.4098109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4098179Z self_outputs = self.self( 2025-08-14T21:42:46.4098417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4098486Z return func(*args, **kwargs) 2025-08-14T21:42:46.4098750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:42:46.4098963Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:46.4098967Z 2025-08-14T21:42:46.4099070Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4099272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4099338Z return mod(**inputs) 2025-08-14T21:42:46.4099606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4099686Z outputs = self.roberta( 2025-08-14T21:42:46.4099960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4100039Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4100298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4100368Z layer_outputs = layer_module( 2025-08-14T21:42:46.4100587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4100664Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4100922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4101010Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4101242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4101317Z return func(*args, **kwargs) 2025-08-14T21:42:46.4101579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4101647Z self_outputs = self.self( 2025-08-14T21:42:46.4101886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4101971Z return func(*args, **kwargs) 2025-08-14T21:42:46.4102232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:42:46.4102306Z self.key(current_states) 2025-08-14T21:42:46.4102309Z 2025-08-14T21:42:46.4102407Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4102604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4102687Z return mod(**inputs) 2025-08-14T21:42:46.4102953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4103027Z outputs = self.roberta( 2025-08-14T21:42:46.4103284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4103361Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4103636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4103706Z layer_outputs = layer_module( 2025-08-14T21:42:46.4103928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4104016Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4104275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4104362Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4104590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4104663Z return func(*args, **kwargs) 2025-08-14T21:42:46.4104923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4104990Z self_outputs = self.self( 2025-08-14T21:42:46.4105228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4105292Z return func(*args, **kwargs) 2025-08-14T21:42:46.4105555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:42:46.4105634Z self.value(current_states) 2025-08-14T21:42:46.4105637Z 2025-08-14T21:42:46.4105717Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.4105823Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4106014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4106076Z return mod(**inputs) 2025-08-14T21:42:46.4106352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4106417Z outputs = self.roberta( 2025-08-14T21:42:46.4106689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4106759Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4107019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4107097Z layer_outputs = layer_module( 2025-08-14T21:42:46.4107306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4107379Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4107648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4107750Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4107992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4108059Z return func(*args, **kwargs) 2025-08-14T21:42:46.4108322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4108399Z self_outputs = self.self( 2025-08-14T21:42:46.4108631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4108714Z return func(*args, **kwargs) 2025-08-14T21:42:46.4108990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:42:46.4109119Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:46.4109123Z 2025-08-14T21:42:46.4109231Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4109424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4109503Z return mod(**inputs) 2025-08-14T21:42:46.4109783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4109849Z outputs = self.roberta( 2025-08-14T21:42:46.4110138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4110223Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4110481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4110557Z layer_outputs = layer_module( 2025-08-14T21:42:46.4110762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4110837Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4111098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4111176Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4111410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4111474Z return func(*args, **kwargs) 2025-08-14T21:42:46.4111725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:42:46.4111855Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:46.4112104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:42:46.4112189Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.4112193Z 2025-08-14T21:42:46.4112287Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4112471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4112539Z return mod(**inputs) 2025-08-14T21:42:46.4112795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4112860Z outputs = self.roberta( 2025-08-14T21:42:46.4113122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4113191Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4113449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4113515Z layer_outputs = layer_module( 2025-08-14T21:42:46.4113736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4113817Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4114070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.4114157Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.4114402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.4114489Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.4114782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.4114892Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.4115144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:42:46.4115231Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.4115234Z 2025-08-14T21:42:46.4115350Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4115553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4115618Z return mod(**inputs) 2025-08-14T21:42:46.4115892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4115969Z outputs = self.roberta( 2025-08-14T21:42:46.4116230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4116308Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4116574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4116646Z layer_outputs = layer_module( 2025-08-14T21:42:46.4116870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4116947Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4117222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.4117309Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.4117561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.4117643Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.4117936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.4118048Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.4118324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:42:46.4118437Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:46.4118652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:46.4118723Z return self.act(input) 2025-08-14T21:42:46.4118728Z 2025-08-14T21:42:46.4118829Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4119032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4119098Z return mod(**inputs) 2025-08-14T21:42:46.4119370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4119444Z outputs = self.roberta( 2025-08-14T21:42:46.4119713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4119808Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4120077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4120147Z layer_outputs = layer_module( 2025-08-14T21:42:46.4120372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4120450Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4120745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.4120826Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.4121083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.4121171Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.4121472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:42:46.4121617Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:46.4121910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:42:46.4121992Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.4121996Z 2025-08-14T21:42:46.4122108Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4122306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4122371Z return mod(**inputs) 2025-08-14T21:42:46.4122650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4122720Z outputs = self.roberta( 2025-08-14T21:42:46.4122995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4123068Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4123337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4123419Z layer_outputs = layer_module( 2025-08-14T21:42:46.4123663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4123741Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4124016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4124097Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4124343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4124414Z return func(*args, **kwargs) 2025-08-14T21:42:46.4124685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4124769Z self_outputs = self.self( 2025-08-14T21:42:46.4125102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4125199Z return func(*args, **kwargs) 2025-08-14T21:42:46.4125498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:42:46.4125723Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:46.4125728Z 2025-08-14T21:42:46.4125849Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4126102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4126175Z return mod(**inputs) 2025-08-14T21:42:46.4126484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4126562Z outputs = self.roberta( 2025-08-14T21:42:46.4126863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4126942Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4127250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4127337Z layer_outputs = layer_module( 2025-08-14T21:42:46.4127568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4127660Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4127946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4128050Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4128310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4128382Z return func(*args, **kwargs) 2025-08-14T21:42:46.4128688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4128773Z self_outputs = self.self( 2025-08-14T21:42:46.4129025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4129104Z return func(*args, **kwargs) 2025-08-14T21:42:46.4129406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:42:46.4129482Z self.key(current_states) 2025-08-14T21:42:46.4129486Z 2025-08-14T21:42:46.4129600Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4129809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4129878Z return mod(**inputs) 2025-08-14T21:42:46.4130172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4130244Z outputs = self.roberta( 2025-08-14T21:42:46.4130537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4130614Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4130913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4130997Z layer_outputs = layer_module( 2025-08-14T21:42:46.4131226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4131317Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4131601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4131688Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4131943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4132017Z return func(*args, **kwargs) 2025-08-14T21:42:46.4132300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4132382Z self_outputs = self.self( 2025-08-14T21:42:46.4132632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4132728Z return func(*args, **kwargs) 2025-08-14T21:42:46.4133018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:42:46.4133088Z self.value(current_states) 2025-08-14T21:42:46.4133091Z 2025-08-14T21:42:46.4133178Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.4133280Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4133484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4133566Z return mod(**inputs) 2025-08-14T21:42:46.4133844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4133919Z outputs = self.roberta( 2025-08-14T21:42:46.4134201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4134277Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4134590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4134665Z layer_outputs = layer_module( 2025-08-14T21:42:46.4134914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4134995Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4135280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4135370Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4135619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4135690Z return func(*args, **kwargs) 2025-08-14T21:42:46.4135982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4136056Z self_outputs = self.self( 2025-08-14T21:42:46.4136310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4136380Z return func(*args, **kwargs) 2025-08-14T21:42:46.4136663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:42:46.4136810Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:46.4136814Z 2025-08-14T21:42:46.4136919Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4137133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4137202Z return mod(**inputs) 2025-08-14T21:42:46.4137491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4137569Z outputs = self.roberta( 2025-08-14T21:42:46.4138031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4138115Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4138421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4138498Z layer_outputs = layer_module( 2025-08-14T21:42:46.4138737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4138819Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4139120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4139272Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4139512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4139581Z return func(*args, **kwargs) 2025-08-14T21:42:46.4139860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:42:46.4139990Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:46.4140272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:42:46.4140385Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.4140389Z 2025-08-14T21:42:46.4140489Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4140695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4140764Z return mod(**inputs) 2025-08-14T21:42:46.4141048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4141141Z outputs = self.roberta( 2025-08-14T21:42:46.4141414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4141517Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4141788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4141869Z layer_outputs = layer_module( 2025-08-14T21:42:46.4142087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4142165Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4142439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.4142524Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.4142793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.4142881Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.4143180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.4143310Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.4143583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:42:46.4143664Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.4143668Z 2025-08-14T21:42:46.4143779Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4143976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4144050Z return mod(**inputs) 2025-08-14T21:42:46.4144320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4144388Z outputs = self.roberta( 2025-08-14T21:42:46.4144665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4144737Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4145008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4145084Z layer_outputs = layer_module( 2025-08-14T21:42:46.4145301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4145384Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4145671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.4145753Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.4146020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.4146097Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.4146406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.4146542Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.4146814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:42:46.4146933Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:46.4147145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:46.4147215Z return self.act(input) 2025-08-14T21:42:46.4147218Z 2025-08-14T21:42:46.4147345Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4147544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4147616Z return mod(**inputs) 2025-08-14T21:42:46.4147905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4147976Z outputs = self.roberta( 2025-08-14T21:42:46.4148257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4148333Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4148617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4148698Z layer_outputs = layer_module( 2025-08-14T21:42:46.4148927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4149016Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4149300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.4149386Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.4149668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.4149747Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.4150071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:42:46.4150217Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:46.4150487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:42:46.4150576Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.4150580Z 2025-08-14T21:42:46.4150680Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4150886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4150951Z return mod(**inputs) 2025-08-14T21:42:46.4151224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4151298Z outputs = self.roberta( 2025-08-14T21:42:46.4151596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4151672Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4151990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4152063Z layer_outputs = layer_module( 2025-08-14T21:42:46.4152300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4152381Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4152667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4152779Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4153032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4153103Z return func(*args, **kwargs) 2025-08-14T21:42:46.4153409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4153485Z self_outputs = self.self( 2025-08-14T21:42:46.4153774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4153847Z return func(*args, **kwargs) 2025-08-14T21:42:46.4154135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:42:46.4154375Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:46.4154380Z 2025-08-14T21:42:46.4154490Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4154706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4154774Z return mod(**inputs) 2025-08-14T21:42:46.4155059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4155141Z outputs = self.roberta( 2025-08-14T21:42:46.4155427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4155503Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4155811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4155885Z layer_outputs = layer_module( 2025-08-14T21:42:46.4156124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4156207Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4156491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4156587Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4156840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4156920Z return func(*args, **kwargs) 2025-08-14T21:42:46.4157221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4157295Z self_outputs = self.self( 2025-08-14T21:42:46.4157556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4157631Z return func(*args, **kwargs) 2025-08-14T21:42:46.4157916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:42:46.4157996Z self.key(current_states) 2025-08-14T21:42:46.4158000Z 2025-08-14T21:42:46.4158106Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4158323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4158440Z return mod(**inputs) 2025-08-14T21:42:46.4158735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4158816Z outputs = self.roberta( 2025-08-14T21:42:46.4159096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4159174Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4159451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4159520Z layer_outputs = layer_module( 2025-08-14T21:42:46.4159736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4159809Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4160066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4160153Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4160397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4160471Z return func(*args, **kwargs) 2025-08-14T21:42:46.4160751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4160820Z self_outputs = self.self( 2025-08-14T21:42:46.4161060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4161125Z return func(*args, **kwargs) 2025-08-14T21:42:46.4161385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:42:46.4161465Z self.value(current_states) 2025-08-14T21:42:46.4161468Z 2025-08-14T21:42:46.4161546Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.4161653Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4161850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4161912Z return mod(**inputs) 2025-08-14T21:42:46.4162188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4162254Z outputs = self.roberta( 2025-08-14T21:42:46.4162525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4162596Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4162857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4162934Z layer_outputs = layer_module( 2025-08-14T21:42:46.4163153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4163232Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4163508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4163606Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4163850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4163920Z return func(*args, **kwargs) 2025-08-14T21:42:46.4164189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4164269Z self_outputs = self.self( 2025-08-14T21:42:46.4164506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4164590Z return func(*args, **kwargs) 2025-08-14T21:42:46.4164870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:42:46.4165070Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:46.4165077Z 2025-08-14T21:42:46.4165203Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4165410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4165501Z return mod(**inputs) 2025-08-14T21:42:46.4165796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4165868Z outputs = self.roberta( 2025-08-14T21:42:46.4166169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4166244Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4166532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4166613Z layer_outputs = layer_module( 2025-08-14T21:42:46.4166864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4166943Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4167217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4167298Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4167539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4167607Z return func(*args, **kwargs) 2025-08-14T21:42:46.4167870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:42:46.4168002Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:46.4168267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:42:46.4168360Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.4168364Z 2025-08-14T21:42:46.4168464Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4168660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4168736Z return mod(**inputs) 2025-08-14T21:42:46.4169005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4169072Z outputs = self.roberta( 2025-08-14T21:42:46.4169346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4169417Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4169694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4169765Z layer_outputs = layer_module( 2025-08-14T21:42:46.4169995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4170079Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4170339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.4170426Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.4170681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.4170781Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.4171092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.4171220Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.4171481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:42:46.4171570Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.4171592Z 2025-08-14T21:42:46.4171692Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4171890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4171953Z return mod(**inputs) 2025-08-14T21:42:46.4172217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4172293Z outputs = self.roberta( 2025-08-14T21:42:46.4172565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4172647Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4172925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4172996Z layer_outputs = layer_module( 2025-08-14T21:42:46.4173218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4173295Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4173558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.4173648Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.4173901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.4173983Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.4174278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.4174392Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.4174663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:42:46.4174774Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:46.4174985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:46.4175053Z return self.act(input) 2025-08-14T21:42:46.4175057Z 2025-08-14T21:42:46.4175154Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4175354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4175422Z return mod(**inputs) 2025-08-14T21:42:46.4175696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4175772Z outputs = self.roberta( 2025-08-14T21:42:46.4176042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4176122Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4176391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4176461Z layer_outputs = layer_module( 2025-08-14T21:42:46.4176688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4177075Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4177503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.4177933Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.4178343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.4178791Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.4179251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:42:46.4179807Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:46.4180297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:42:46.4180710Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.4180859Z 2025-08-14T21:42:46.4180961Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4181337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4181661Z return mod(**inputs) 2025-08-14T21:42:46.4182047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4182468Z outputs = self.roberta( 2025-08-14T21:42:46.4182857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4183259Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4183662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4184064Z layer_outputs = layer_module( 2025-08-14T21:42:46.4184409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4184764Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4185176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4185591Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4185964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4186343Z return func(*args, **kwargs) 2025-08-14T21:42:46.4186740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4187144Z self_outputs = self.self( 2025-08-14T21:42:46.4187496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4187867Z return func(*args, **kwargs) 2025-08-14T21:42:46.4188265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 325, in forward 2025-08-14T21:42:46.4188808Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:42:46.4189071Z 2025-08-14T21:42:46.4189173Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4189536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4189862Z return mod(**inputs) 2025-08-14T21:42:46.4190237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4190644Z outputs = self.roberta( 2025-08-14T21:42:46.4191029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4191452Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4191847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4192256Z layer_outputs = layer_module( 2025-08-14T21:42:46.4192607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4192958Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4193363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4193798Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4194177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4194555Z return func(*args, **kwargs) 2025-08-14T21:42:46.4194939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4195331Z self_outputs = self.self( 2025-08-14T21:42:46.4195700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4196053Z return func(*args, **kwargs) 2025-08-14T21:42:46.4196454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 353, in forward 2025-08-14T21:42:46.4196850Z self.key(current_states) 2025-08-14T21:42:46.4196964Z 2025-08-14T21:42:46.4197066Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4197415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4197736Z return mod(**inputs) 2025-08-14T21:42:46.4198113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4198499Z outputs = self.roberta( 2025-08-14T21:42:46.4198872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4199269Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4199645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4200038Z layer_outputs = layer_module( 2025-08-14T21:42:46.4200370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4200720Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4201107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4201503Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4201873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4202233Z return func(*args, **kwargs) 2025-08-14T21:42:46.4202609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4203000Z self_outputs = self.self( 2025-08-14T21:42:46.4203348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4203698Z return func(*args, **kwargs) 2025-08-14T21:42:46.4204082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 358, in forward 2025-08-14T21:42:46.4204479Z self.value(current_states) 2025-08-14T21:42:46.4204598Z 2025-08-14T21:42:46.4204686Z cudagraph partition due to non gpu ops 2025-08-14T21:42:46.4204916Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4205382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4205713Z return mod(**inputs) 2025-08-14T21:42:46.4206095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4206509Z outputs = self.roberta( 2025-08-14T21:42:46.4206898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4207308Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4207739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4208151Z layer_outputs = layer_module( 2025-08-14T21:42:46.4208502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4208879Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4209274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4209716Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4210096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4210457Z return func(*args, **kwargs) 2025-08-14T21:42:46.4210868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 467, in forward 2025-08-14T21:42:46.4211285Z self_outputs = self.self( 2025-08-14T21:42:46.4211653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4212019Z return func(*args, **kwargs) 2025-08-14T21:42:46.4212417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 389, in forward 2025-08-14T21:42:46.4212885Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:42:46.4213074Z 2025-08-14T21:42:46.4213178Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4213538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4213866Z return mod(**inputs) 2025-08-14T21:42:46.4214253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4214652Z outputs = self.roberta( 2025-08-14T21:42:46.4215034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4215442Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4215846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4216251Z layer_outputs = layer_module( 2025-08-14T21:42:46.4216597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4216960Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4217367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 540, in forward 2025-08-14T21:42:46.4217790Z self_attention_outputs = self.attention( 2025-08-14T21:42:46.4218174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:42:46.4218543Z return func(*args, **kwargs) 2025-08-14T21:42:46.4218926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 477, in forward 2025-08-14T21:42:46.4219386Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:42:46.4219877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 413, in forward 2025-08-14T21:42:46.4220298Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.4220437Z 2025-08-14T21:42:46.4220542Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4220906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4221233Z return mod(**inputs) 2025-08-14T21:42:46.4221612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4222040Z outputs = self.roberta( 2025-08-14T21:42:46.4222431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4222843Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4223236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4223644Z layer_outputs = layer_module( 2025-08-14T21:42:46.4224023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4224375Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4224805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.4225232Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.4225645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.4226038Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.4226475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.4226958Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.4227435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 493, in forward 2025-08-14T21:42:46.4227882Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.4228033Z 2025-08-14T21:42:46.4228140Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4228525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4228878Z return mod(**inputs) 2025-08-14T21:42:46.4229283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4229719Z outputs = self.roberta( 2025-08-14T21:42:46.4230140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4230577Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4231028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4231482Z layer_outputs = layer_module( 2025-08-14T21:42:46.4231855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4232249Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4232684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.4233130Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.4233546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.4233973Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.4234433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 578, in feed_forward_chunk 2025-08-14T21:42:46.4234984Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:42:46.4235453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 494, in forward 2025-08-14T21:42:46.4235927Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:42:46.4236334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:42:46.4236711Z return self.act(input) 2025-08-14T21:42:46.4236830Z 2025-08-14T21:42:46.4236939Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4237320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4237783Z return mod(**inputs) 2025-08-14T21:42:46.4238199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1038, in forward 2025-08-14T21:42:46.4238639Z outputs = self.roberta( 2025-08-14T21:42:46.4239098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 950, in forward 2025-08-14T21:42:46.4239535Z encoder_outputs = self.encoder( 2025-08-14T21:42:46.4239991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 632, in forward 2025-08-14T21:42:46.4240423Z layer_outputs = layer_module( 2025-08-14T21:42:46.4240794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:42:46.4241168Z return super().__call__(*args, **kwargs) 2025-08-14T21:42:46.4241605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 570, in forward 2025-08-14T21:42:46.4242050Z layer_output = apply_chunking_to_forward( 2025-08-14T21:42:46.4242484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:42:46.4242905Z return forward_fn(*input_tensors) 2025-08-14T21:42:46.4243372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 579, in feed_forward_chunk 2025-08-14T21:42:46.4243893Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:42:46.4244388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 507, in forward 2025-08-14T21:42:46.4244833Z hidden_states = self.dense(hidden_states) 2025-08-14T21:42:46.4244992Z 2025-08-14T21:42:46.4245296Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4245723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4246087Z return mod(**inputs) 2025-08-14T21:42:46.4246502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1052, in forward 2025-08-14T21:42:46.4246968Z prediction_scores = self.lm_head(sequence_output) 2025-08-14T21:42:46.4247429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 756, in forward 2025-08-14T21:42:46.4247858Z x = self.dense(features) 2025-08-14T21:42:46.4247989Z 2025-08-14T21:42:46.4248098Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4248483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4248827Z return mod(**inputs) 2025-08-14T21:42:46.4249230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1052, in forward 2025-08-14T21:42:46.4249667Z prediction_scores = self.lm_head(sequence_output) 2025-08-14T21:42:46.4250145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 761, in forward 2025-08-14T21:42:46.4250551Z x = self.decoder(x) 2025-08-14T21:42:46.4250662Z 2025-08-14T21:42:46.4250760Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:42:46.4251108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:42:46.4251427Z return mod(**inputs) 2025-08-14T21:42:46.4251798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/camembert/modeling_camembert.py", line 1059, in forward 2025-08-14T21:42:46.4252342Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:42:46.4252573Z 2025-08-14T21:42:55.3224218Z Compilation time (from dynamo_timed): 15.325529934 2025-08-14T21:42:55.3300814Z pass 2025-08-14T21:42:55.3301175Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:42:55.3302274Z TIMING: _recursive_pre_grad_passes:0.0076 _recursive_joint_graph_passes:0.37942 _recursive_post_grad_passes:0.08341 async_compile.wait:0.7585 code_gen:7.74728 inductor_compile:8.97138 backend_compile:12.18861 gc:0.00054 entire_frame_compile:15.32553 total_wall_time:15.32553 2025-08-14T21:42:55.3303257Z STATS: call_* op count: 297 | FakeTensorMode.__torch_dispatch__:12436 | FakeTensor.__torch_dispatch__:4756 | ProxyTorchDispatchMode.__torch_dispatch__:4530 2025-08-14T21:42:55.3303741Z Dynamo produced 1 graphs covering 297 ops with 0 graph breaks (0 unique) 2025-08-14T21:43:00.5465393Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:00.5466436Z from pkg_resources import resource_filename 2025-08-14T21:43:01.1871980Z 2025-08-14T21:43:10.2468543Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:43:10.2468838Z loading model: 0it [00:09, ?it/s] 2025-08-14T21:43:10.2492532Z cpu eval DebertaV2ForMaskedLM 2025-08-14T21:43:10.3850088Z Compilation time (from dynamo_timed): 0 2025-08-14T21:43:10.3850380Z pass_due_to_skip 2025-08-14T21:43:10.3855312Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:43:10.3855725Z TIMING: total_wall_time:0 2025-08-14T21:43:10.3855945Z STATS: call_* op count: 0 2025-08-14T21:43:10.3856219Z Dynamo produced 0 graphs covering 0 ops with 0 graph breaks (0 unique) 2025-08-14T21:43:14.9949702Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:43:14.9951413Z from pkg_resources import resource_filename 2025-08-14T21:43:15.6229754Z 2025-08-14T21:43:22.9066367Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:43:22.9066869Z loading model: 0it [00:07, ?it/s] 2025-08-14T21:43:22.9091398Z cpu eval DebertaV2ForQuestionAnswering 2025-08-14T21:43:26.1704901Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:43:27.7192254Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:43:29.0794106Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:43:43.9247531Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9252233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9256756Z return mod(**inputs) 2025-08-14T21:43:43.9262619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9267733Z outputs = self.deberta( 2025-08-14T21:43:43.9268512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9269055Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9269551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9270239Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9270970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9271395Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9271833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9272317Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9272846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9273303Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9273807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9274398Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9274689Z 2025-08-14T21:43:43.9274813Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9275214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9275583Z return mod(**inputs) 2025-08-14T21:43:43.9276010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9276471Z outputs = self.deberta( 2025-08-14T21:43:43.9276886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9277336Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9277770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9278193Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9278579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9278978Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9279427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9279890Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9280335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9280781Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9281217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:43.9281761Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9282017Z 2025-08-14T21:43:43.9282131Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9282516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9282864Z return mod(**inputs) 2025-08-14T21:43:43.9283260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9283720Z outputs = self.deberta( 2025-08-14T21:43:43.9284128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9284554Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9284966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9285426Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9285993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9286406Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9286900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9287353Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9287801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9288228Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9288704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9289306Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9289896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9290423Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9290631Z 2025-08-14T21:43:43.9290744Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9291129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9291477Z return mod(**inputs) 2025-08-14T21:43:43.9291882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9292313Z outputs = self.deberta( 2025-08-14T21:43:43.9292725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9293147Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9293572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9294021Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9294410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9294786Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9295212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9295656Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9296102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9296551Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9296957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9297506Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9297780Z 2025-08-14T21:43:43.9297896Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9298256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9298585Z return mod(**inputs) 2025-08-14T21:43:43.9298991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9299385Z outputs = self.deberta( 2025-08-14T21:43:43.9299772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9300175Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9300568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9300995Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9301357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9301714Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9302110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9302530Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9302966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9303379Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9303792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9304329Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9304598Z 2025-08-14T21:43:43.9304706Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9305068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9305381Z return mod(**inputs) 2025-08-14T21:43:43.9305761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9306162Z outputs = self.deberta( 2025-08-14T21:43:43.9306537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9306928Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9307322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9307732Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9308082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9308453Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9308856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9309273Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9309680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9310077Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9310474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9310985Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9311229Z 2025-08-14T21:43:43.9311330Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9311687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9312000Z return mod(**inputs) 2025-08-14T21:43:43.9312366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9312776Z outputs = self.deberta( 2025-08-14T21:43:43.9313149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9313548Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9313933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9314352Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9314724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9315070Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9315465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9315862Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9316263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9316649Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9317030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9317549Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9318089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9318574Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9318765Z 2025-08-14T21:43:43.9318871Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9319232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9319549Z return mod(**inputs) 2025-08-14T21:43:43.9319915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9320308Z outputs = self.deberta( 2025-08-14T21:43:43.9320690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9321090Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9321479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9321897Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9322277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9322628Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9323035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9323468Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9323908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9324324Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9324749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:43.9325173Z context_layer = torch.bmm( 2025-08-14T21:43:43.9325294Z 2025-08-14T21:43:43.9325409Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9325901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9326255Z return mod(**inputs) 2025-08-14T21:43:43.9326688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9327105Z outputs = self.deberta( 2025-08-14T21:43:43.9327479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9327878Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9328266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9328684Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9329038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9329386Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9329775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9330188Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9330625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9331043Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9331444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:43.9331959Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:43.9332206Z 2025-08-14T21:43:43.9332307Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9332656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9332966Z return mod(**inputs) 2025-08-14T21:43:43.9333338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9333732Z outputs = self.deberta( 2025-08-14T21:43:43.9334104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9334526Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9334919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9335337Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9335683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9336028Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9336422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9336829Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9337239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:43.9337839Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:43.9338303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:43.9338714Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9338874Z 2025-08-14T21:43:43.9338977Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9339333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9339653Z return mod(**inputs) 2025-08-14T21:43:43.9340019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9340454Z outputs = self.deberta( 2025-08-14T21:43:43.9340820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9341208Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9341583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9341986Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9342343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9342729Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9343129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9343571Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9344009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:43.9344405Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9344545Z 2025-08-14T21:43:43.9344673Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9345023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9345334Z return mod(**inputs) 2025-08-14T21:43:43.9345721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9346116Z outputs = self.deberta( 2025-08-14T21:43:43.9346493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9346879Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9347286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9347728Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9348121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9348498Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9348899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9349348Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9349817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:43.9350285Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:43.9350693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:43.9351064Z return self.act(input) 2025-08-14T21:43:43.9351183Z 2025-08-14T21:43:43.9351294Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9351686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9352039Z return mod(**inputs) 2025-08-14T21:43:43.9352451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9352882Z outputs = self.deberta( 2025-08-14T21:43:43.9353307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9353745Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9354165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9354617Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9355037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9355434Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9355858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:43.9356336Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:43.9356773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:43.9357201Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9357340Z 2025-08-14T21:43:43.9357443Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9357801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9358126Z return mod(**inputs) 2025-08-14T21:43:43.9358502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9358901Z outputs = self.deberta( 2025-08-14T21:43:43.9359297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9359716Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9360112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9360521Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9360889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9361247Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9361639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9362062Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9362489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9362908Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9363334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9363879Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9364132Z 2025-08-14T21:43:43.9364248Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9364619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9364968Z return mod(**inputs) 2025-08-14T21:43:43.9365373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9365897Z outputs = self.deberta( 2025-08-14T21:43:43.9366347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9366775Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9367187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9367595Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9367947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9368307Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9368709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9369156Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9369571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9369983Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9370444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:43.9370949Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9371219Z 2025-08-14T21:43:43.9371319Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9371664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9371985Z return mod(**inputs) 2025-08-14T21:43:43.9372359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9372761Z outputs = self.deberta( 2025-08-14T21:43:43.9373163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9373564Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9373975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9374406Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9374761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9375100Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9375490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9375894Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9376297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9376685Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9377073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9377585Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9378132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9378616Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9378805Z 2025-08-14T21:43:43.9378909Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9379263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9379576Z return mod(**inputs) 2025-08-14T21:43:43.9379952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9380372Z outputs = self.deberta( 2025-08-14T21:43:43.9380748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9381139Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9381531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9381941Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9382295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9382647Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9383054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9384333Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9384734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9385132Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9385531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9386078Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9386338Z 2025-08-14T21:43:43.9386439Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9386789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9387113Z return mod(**inputs) 2025-08-14T21:43:43.9387514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9387942Z outputs = self.deberta( 2025-08-14T21:43:43.9388361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9388789Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9389227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9389635Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9389988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9390334Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9390721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9391115Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9391521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9391912Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9392291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9392794Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9393045Z 2025-08-14T21:43:43.9393152Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9393485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9393792Z return mod(**inputs) 2025-08-14T21:43:43.9394151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9394529Z outputs = self.deberta( 2025-08-14T21:43:43.9394885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9395267Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9395644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9396039Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9396376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9396713Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9397092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9397529Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9397939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9398332Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9398723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9399220Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9399484Z 2025-08-14T21:43:43.9399589Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9399931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9400246Z return mod(**inputs) 2025-08-14T21:43:43.9400615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9401005Z outputs = self.deberta( 2025-08-14T21:43:43.9401388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9401801Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9402212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9402631Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9402992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9403350Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9403743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9404147Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9404551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9404946Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9405347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9406091Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9406676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9407206Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9407399Z 2025-08-14T21:43:43.9407532Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9407884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9408199Z return mod(**inputs) 2025-08-14T21:43:43.9408612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9409043Z outputs = self.deberta( 2025-08-14T21:43:43.9409475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9409896Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9410311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9410749Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9411124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9411512Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9411999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9412455Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9412911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9413343Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9413779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:43.9414225Z context_layer = torch.bmm( 2025-08-14T21:43:43.9414349Z 2025-08-14T21:43:43.9414458Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9414845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9415188Z return mod(**inputs) 2025-08-14T21:43:43.9415548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9415927Z outputs = self.deberta( 2025-08-14T21:43:43.9416300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9416692Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9417081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9417491Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9417869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9418247Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9418649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9419070Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9419495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9419882Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9420274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:43.9420782Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:43.9421014Z 2025-08-14T21:43:43.9421120Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9421450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9421755Z return mod(**inputs) 2025-08-14T21:43:43.9422117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9422495Z outputs = self.deberta( 2025-08-14T21:43:43.9422848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9423224Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9423597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9423981Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9424326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9424666Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9425050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9425439Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9425856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:43.9426281Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:43.9426700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:43.9427090Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9427232Z 2025-08-14T21:43:43.9427330Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9427698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9428010Z return mod(**inputs) 2025-08-14T21:43:43.9428383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9428789Z outputs = self.deberta( 2025-08-14T21:43:43.9429169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9429558Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9429930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9430348Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9430696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9431043Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9431433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9431886Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9432312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:43.9432708Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9432847Z 2025-08-14T21:43:43.9432949Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9433295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9433603Z return mod(**inputs) 2025-08-14T21:43:43.9433985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9434392Z outputs = self.deberta( 2025-08-14T21:43:43.9434764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9435170Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9435564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9435975Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9436332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9436689Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9437091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9437531Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9438207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:43.9438654Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:43.9439040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:43.9439419Z return self.act(input) 2025-08-14T21:43:43.9439537Z 2025-08-14T21:43:43.9439641Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9440000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9440324Z return mod(**inputs) 2025-08-14T21:43:43.9440698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9441102Z outputs = self.deberta( 2025-08-14T21:43:43.9441483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9441911Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9442295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9442704Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9443066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9443413Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9443848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:43.9444368Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:43.9444831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:43.9445240Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9445392Z 2025-08-14T21:43:43.9445569Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9445964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9446307Z return mod(**inputs) 2025-08-14T21:43:43.9446702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9447100Z outputs = self.deberta( 2025-08-14T21:43:43.9447479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9447872Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9448270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9448686Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9449046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9449397Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9449800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9450220Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9450631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9451031Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9451414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9451901Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9452131Z 2025-08-14T21:43:43.9452229Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9452568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9452875Z return mod(**inputs) 2025-08-14T21:43:43.9453237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9453628Z outputs = self.deberta( 2025-08-14T21:43:43.9453989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9454371Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9454739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9455133Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9455498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9455839Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9456213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9456609Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9457004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9457441Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9457824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:43.9458324Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9458545Z 2025-08-14T21:43:43.9458652Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9458994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9459301Z return mod(**inputs) 2025-08-14T21:43:43.9459684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9460072Z outputs = self.deberta( 2025-08-14T21:43:43.9460429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9460811Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9461188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9461582Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9461918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9462258Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9462639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9463027Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9463422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9463805Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9464184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9464666Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9465181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9465651Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9465828Z 2025-08-14T21:43:43.9465932Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9466260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9466587Z return mod(**inputs) 2025-08-14T21:43:43.9466949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9467330Z outputs = self.deberta( 2025-08-14T21:43:43.9467683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9468063Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9468435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9468844Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9469194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9469539Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9469925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9470319Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9470736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9471126Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9471524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9472032Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9472288Z 2025-08-14T21:43:43.9472385Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9472728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9473028Z return mod(**inputs) 2025-08-14T21:43:43.9473392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9473770Z outputs = self.deberta( 2025-08-14T21:43:43.9474132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9474523Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9474899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9475293Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9475636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9475967Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9476350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9476749Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9477140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9477524Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9477907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9478410Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9478658Z 2025-08-14T21:43:43.9478756Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9479091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9479399Z return mod(**inputs) 2025-08-14T21:43:43.9479758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9480153Z outputs = self.deberta( 2025-08-14T21:43:43.9480517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9480899Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9481268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9481686Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9482027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9482363Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9482738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9483136Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9483550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9483935Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9484325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9484819Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9485050Z 2025-08-14T21:43:43.9485153Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9485548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9485873Z return mod(**inputs) 2025-08-14T21:43:43.9486249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9486652Z outputs = self.deberta( 2025-08-14T21:43:43.9487029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9487436Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9487834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9488240Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9488590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9488944Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9489334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9489728Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9490124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9490509Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9490889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9491367Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9491889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9492360Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9492533Z 2025-08-14T21:43:43.9492638Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9492969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9493296Z return mod(**inputs) 2025-08-14T21:43:43.9493661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9494048Z outputs = self.deberta( 2025-08-14T21:43:43.9494402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9494785Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9495161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9495564Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9495909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9496250Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9496641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9497063Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9497462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9497844Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9498242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:43.9498617Z context_layer = torch.bmm( 2025-08-14T21:43:43.9498736Z 2025-08-14T21:43:43.9498833Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9499172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9499481Z return mod(**inputs) 2025-08-14T21:43:43.9499841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9500220Z outputs = self.deberta( 2025-08-14T21:43:43.9500580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9500950Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9501329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9523007Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9523450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9523840Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9524261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9524715Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9525147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9525652Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9526067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:43.9526609Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:43.9526859Z 2025-08-14T21:43:43.9526966Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9527327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9527649Z return mod(**inputs) 2025-08-14T21:43:43.9528034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9528523Z outputs = self.deberta( 2025-08-14T21:43:43.9528926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9529319Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9529712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9530122Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9530525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9530889Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9531341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9531755Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9532162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:43.9532631Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:43.9533069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:43.9533498Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9533642Z 2025-08-14T21:43:43.9533746Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9534101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9534422Z return mod(**inputs) 2025-08-14T21:43:43.9534790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9535190Z outputs = self.deberta( 2025-08-14T21:43:43.9535558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9535956Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9536341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9536749Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9537130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9537527Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9538158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9538701Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9539190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:43.9539598Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9539733Z 2025-08-14T21:43:43.9539846Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9540187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9540507Z return mod(**inputs) 2025-08-14T21:43:43.9540882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9541279Z outputs = self.deberta( 2025-08-14T21:43:43.9541651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9542049Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9542437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9542889Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9543250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9543603Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9543996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9544424Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9544893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:43.9545323Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:43.9545695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:43.9546025Z return self.act(input) 2025-08-14T21:43:43.9546141Z 2025-08-14T21:43:43.9546242Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9546622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9546932Z return mod(**inputs) 2025-08-14T21:43:43.9547325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9547716Z outputs = self.deberta( 2025-08-14T21:43:43.9548084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9548470Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9548854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9549260Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9549619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9549959Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9550351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:43.9550801Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:43.9551240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:43.9551642Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9551781Z 2025-08-14T21:43:43.9551885Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9552237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9552548Z return mod(**inputs) 2025-08-14T21:43:43.9552920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9553310Z outputs = self.deberta( 2025-08-14T21:43:43.9553681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9554070Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9554455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9554860Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9555209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9555557Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9555961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9556380Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9556770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9557160Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9557551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9558050Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9558304Z 2025-08-14T21:43:43.9558404Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9558750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9559071Z return mod(**inputs) 2025-08-14T21:43:43.9559424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9559827Z outputs = self.deberta( 2025-08-14T21:43:43.9560223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9560629Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9561032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9561447Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9561803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9562155Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9562547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9562967Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9563386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9563782Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9564182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:43.9564686Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9564917Z 2025-08-14T21:43:43.9565027Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9565377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9565779Z return mod(**inputs) 2025-08-14T21:43:43.9566166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9566570Z outputs = self.deberta( 2025-08-14T21:43:43.9566951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9567343Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9567729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9568130Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9568491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9568856Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9569262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9569677Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9570118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9570574Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9570967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9571475Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9572028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9572566Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9572753Z 2025-08-14T21:43:43.9572865Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9573217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9573541Z return mod(**inputs) 2025-08-14T21:43:43.9573936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9574332Z outputs = self.deberta( 2025-08-14T21:43:43.9574716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9575135Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9575539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9575947Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9576312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9576671Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9577068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9577497Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9577925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9578329Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9578730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9579276Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9579544Z 2025-08-14T21:43:43.9579653Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9580006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9580330Z return mod(**inputs) 2025-08-14T21:43:43.9580714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9581110Z outputs = self.deberta( 2025-08-14T21:43:43.9581493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9581898Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9582294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9582704Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9583072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9583430Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9583838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9584268Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9584676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9585068Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9585449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9585971Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9586253Z 2025-08-14T21:43:43.9586354Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9586705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9587018Z return mod(**inputs) 2025-08-14T21:43:43.9587393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9587786Z outputs = self.deberta( 2025-08-14T21:43:43.9588179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9588586Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9589032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9589476Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9589863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9590225Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9590636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9591047Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9591447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9591844Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9592238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9592745Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9592982Z 2025-08-14T21:43:43.9593082Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9593428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9593746Z return mod(**inputs) 2025-08-14T21:43:43.9594113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9594510Z outputs = self.deberta( 2025-08-14T21:43:43.9594884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9595275Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9595653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9596057Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9596414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9596757Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9597162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9597582Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9598023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9598419Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9598821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9599338Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9599890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9600400Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9600592Z 2025-08-14T21:43:43.9600701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9601065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9601396Z return mod(**inputs) 2025-08-14T21:43:43.9601800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9602204Z outputs = self.deberta( 2025-08-14T21:43:43.9602601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9603001Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9603396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9603817Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9604181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9604532Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9604956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9605413Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9605953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9606363Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9606810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:43.9607259Z context_layer = torch.bmm( 2025-08-14T21:43:43.9607384Z 2025-08-14T21:43:43.9607495Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9607931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9608283Z return mod(**inputs) 2025-08-14T21:43:43.9608670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9609072Z outputs = self.deberta( 2025-08-14T21:43:43.9609460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9609867Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9610266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9610686Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9611049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9611410Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9611806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9612257Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9612680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9613090Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9613488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:43.9614000Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:43.9614260Z 2025-08-14T21:43:43.9614381Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9614721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9615029Z return mod(**inputs) 2025-08-14T21:43:43.9615406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9615801Z outputs = self.deberta( 2025-08-14T21:43:43.9616187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9616581Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9616981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9617385Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9617734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9618079Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9618467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9618876Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9619275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:43.9619696Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:43.9620115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:43.9620499Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9620637Z 2025-08-14T21:43:43.9620734Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9621074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9621383Z return mod(**inputs) 2025-08-14T21:43:43.9621736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9622119Z outputs = self.deberta( 2025-08-14T21:43:43.9622480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9622851Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9623227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9623617Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9623958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9624290Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9624674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9625095Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9625515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:43.9625931Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9626069Z 2025-08-14T21:43:43.9626166Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9626508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9626806Z return mod(**inputs) 2025-08-14T21:43:43.9627165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9627562Z outputs = self.deberta( 2025-08-14T21:43:43.9627931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9628314Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9628699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9629103Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9629471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9629816Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9630222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9630656Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9631078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:43.9631496Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:43.9631862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:43.9632196Z return self.act(input) 2025-08-14T21:43:43.9632303Z 2025-08-14T21:43:43.9632401Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9632752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9633067Z return mod(**inputs) 2025-08-14T21:43:43.9633435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9633816Z outputs = self.deberta( 2025-08-14T21:43:43.9634186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9634575Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9634948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9635348Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9635702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9636047Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9636436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:43.9636883Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:43.9637322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:43.9637868Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9638004Z 2025-08-14T21:43:43.9638108Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9638457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9638771Z return mod(**inputs) 2025-08-14T21:43:43.9639200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9639602Z outputs = self.deberta( 2025-08-14T21:43:43.9639972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9640369Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9640762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9641193Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9641558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9641913Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9642321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9642741Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9643192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9643604Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9644024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9644543Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9644791Z 2025-08-14T21:43:43.9644892Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9645249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9645645Z return mod(**inputs) 2025-08-14T21:43:43.9646061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9646498Z outputs = self.deberta( 2025-08-14T21:43:43.9646892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9647283Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9647674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9648090Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9648475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9648871Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9649312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9649777Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9650225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9650671Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9651110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:43.9651661Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9651908Z 2025-08-14T21:43:43.9652018Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9652398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9652747Z return mod(**inputs) 2025-08-14T21:43:43.9653149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9653598Z outputs = self.deberta( 2025-08-14T21:43:43.9653950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9654321Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9654681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9655082Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9655428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9655787Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9656165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9656576Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9656964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9657342Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9657737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9658237Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9658763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9659236Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9659416Z 2025-08-14T21:43:43.9659511Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9659842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9660143Z return mod(**inputs) 2025-08-14T21:43:43.9660487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9660862Z outputs = self.deberta( 2025-08-14T21:43:43.9661218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9661595Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9661956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9662340Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9662679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9663007Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9663393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9663794Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9664193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9664573Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9664951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9665453Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9665699Z 2025-08-14T21:43:43.9665797Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9666121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9666447Z return mod(**inputs) 2025-08-14T21:43:43.9666801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9667169Z outputs = self.deberta( 2025-08-14T21:43:43.9667527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9667910Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9668289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9668693Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9669035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9669372Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9669757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9670149Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9670568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9670962Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9671367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9671875Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9672130Z 2025-08-14T21:43:43.9672228Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9672565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9672865Z return mod(**inputs) 2025-08-14T21:43:43.9673229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9673615Z outputs = self.deberta( 2025-08-14T21:43:43.9673978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9674352Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9674726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9675115Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9675451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9675789Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9676171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9676565Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9676950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9677345Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9677747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9678260Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9678513Z 2025-08-14T21:43:43.9678616Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9678965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9679278Z return mod(**inputs) 2025-08-14T21:43:43.9679635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9680054Z outputs = self.deberta( 2025-08-14T21:43:43.9680436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9680839Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9681228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9681648Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9682048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9682399Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9682795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9683215Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9683636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9684085Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9684489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9685018Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9685660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9686161Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9686361Z 2025-08-14T21:43:43.9686465Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9686823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9687151Z return mod(**inputs) 2025-08-14T21:43:43.9687518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9687920Z outputs = self.deberta( 2025-08-14T21:43:43.9688306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9688709Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9689100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9689517Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9689881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9690228Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9690633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9691055Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9691473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9691876Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9692277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:43.9692680Z context_layer = torch.bmm( 2025-08-14T21:43:43.9692798Z 2025-08-14T21:43:43.9692912Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9693110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9693177Z return mod(**inputs) 2025-08-14T21:43:43.9693483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9693554Z outputs = self.deberta( 2025-08-14T21:43:43.9693824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9693905Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9694170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9694285Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9694506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9694584Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9694857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9694949Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9695239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9695325Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9695611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:43.9695812Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:43.9695818Z 2025-08-14T21:43:43.9695921Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9696120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9696195Z return mod(**inputs) 2025-08-14T21:43:43.9696465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9696543Z outputs = self.deberta( 2025-08-14T21:43:43.9696812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9696888Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9697164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9697249Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9697477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9697556Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9697820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9697919Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9698185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:43.9698300Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:43.9698575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:43.9698660Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9698664Z 2025-08-14T21:43:43.9698772Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9698971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9699036Z return mod(**inputs) 2025-08-14T21:43:43.9699313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9699402Z outputs = self.deberta( 2025-08-14T21:43:43.9699676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9699749Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9700014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9700104Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9700323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9700418Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9700695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9700816Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9701090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:43.9701175Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9701178Z 2025-08-14T21:43:43.9701295Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9701502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9701583Z return mod(**inputs) 2025-08-14T21:43:43.9701861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9701941Z outputs = self.deberta( 2025-08-14T21:43:43.9702199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9702275Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9702532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9702615Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9702836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9702910Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9703174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9703289Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9703546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:43.9703659Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:43.9703868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:43.9703945Z return self.act(input) 2025-08-14T21:43:43.9703948Z 2025-08-14T21:43:43.9704044Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9704227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9704296Z return mod(**inputs) 2025-08-14T21:43:43.9704555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9704620Z outputs = self.deberta( 2025-08-14T21:43:43.9704879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9704949Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9705205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9705285Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9705515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9705596Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9705849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:43.9705982Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:43.9706236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:43.9706335Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9706339Z 2025-08-14T21:43:43.9706441Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9706627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9706689Z return mod(**inputs) 2025-08-14T21:43:43.9706954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9707020Z outputs = self.deberta( 2025-08-14T21:43:43.9707302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9707374Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9707645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9707736Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9707942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9708021Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9708274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9708363Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9708625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9708699Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9708954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9709141Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9709146Z 2025-08-14T21:43:43.9709250Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9709446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9709507Z return mod(**inputs) 2025-08-14T21:43:43.9709764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9709838Z outputs = self.deberta( 2025-08-14T21:43:43.9710095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9710172Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9710428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9710508Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9710726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9710798Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9711050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9711162Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9711418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9711499Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9711753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:43.9711922Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9711943Z 2025-08-14T21:43:43.9712049Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9712236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9712307Z return mod(**inputs) 2025-08-14T21:43:43.9712565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9712632Z outputs = self.deberta( 2025-08-14T21:43:43.9712908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9712976Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9713240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9713328Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9713535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9713616Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9713869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9713954Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9714219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9714291Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9714551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9714728Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9715021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9715157Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9715160Z 2025-08-14T21:43:43.9715256Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9715451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9715514Z return mod(**inputs) 2025-08-14T21:43:43.9715773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9715844Z outputs = self.deberta( 2025-08-14T21:43:43.9716098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9716167Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9716429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9716509Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9716723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9716798Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9717067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9717159Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9717411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9717490Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9717751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9717964Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9717968Z 2025-08-14T21:43:43.9718069Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9718250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9718312Z return mod(**inputs) 2025-08-14T21:43:43.9718571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9718649Z outputs = self.deberta( 2025-08-14T21:43:43.9718913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9718984Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9719255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9719347Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9719557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9719638Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9719892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9719979Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9720239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9720310Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9720580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9720781Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9720785Z 2025-08-14T21:43:43.9720882Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9721074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9721136Z return mod(**inputs) 2025-08-14T21:43:43.9721395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9721468Z outputs = self.deberta( 2025-08-14T21:43:43.9721720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9721797Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9722048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9722130Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9722347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9722423Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9722680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9722791Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9723052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9723135Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9723396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9723583Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9723604Z 2025-08-14T21:43:43.9723710Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9723898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9723967Z return mod(**inputs) 2025-08-14T21:43:43.9724228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9724296Z outputs = self.deberta( 2025-08-14T21:43:43.9724577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9724649Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9724936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9725020Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9725237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9725325Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9725688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9725792Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9726083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9726165Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9726453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9726650Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9726983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9727130Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9727133Z 2025-08-14T21:43:43.9727229Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9727421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9727485Z return mod(**inputs) 2025-08-14T21:43:43.9727751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9727828Z outputs = self.deberta( 2025-08-14T21:43:43.9728073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9728139Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9728393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9728472Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9728679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9728751Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9729026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9729118Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9729362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9729442Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9729688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:43.9729780Z context_layer = torch.bmm( 2025-08-14T21:43:43.9729783Z 2025-08-14T21:43:43.9729885Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9730074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9730136Z return mod(**inputs) 2025-08-14T21:43:43.9730407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9730471Z outputs = self.deberta( 2025-08-14T21:43:43.9730750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9730820Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9731089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9731180Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9731387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9731468Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9731723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9731811Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9732072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9732145Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9732409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:43.9732591Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:43.9732596Z 2025-08-14T21:43:43.9732687Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9732877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9732936Z return mod(**inputs) 2025-08-14T21:43:43.9733863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9733934Z outputs = self.deberta( 2025-08-14T21:43:43.9734184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9734259Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9734507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9734587Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9734800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9734871Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9735125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9735229Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9735474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:43.9735590Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:43.9735838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:43.9735917Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9735928Z 2025-08-14T21:43:43.9736041Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9736219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9736287Z return mod(**inputs) 2025-08-14T21:43:43.9736538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9736603Z outputs = self.deberta( 2025-08-14T21:43:43.9736855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9736937Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9737188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9737280Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9737485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9737564Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9737929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9738050Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9738315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:43.9738395Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9738398Z 2025-08-14T21:43:43.9738508Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9738694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9738769Z return mod(**inputs) 2025-08-14T21:43:43.9739027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9739092Z outputs = self.deberta( 2025-08-14T21:43:43.9739349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9739418Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9739664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9739754Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9739958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9740032Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9740288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9740398Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9740651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:43.9740753Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:43.9740948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:43.9741058Z return self.act(input) 2025-08-14T21:43:43.9741062Z 2025-08-14T21:43:43.9741156Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9741345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9741406Z return mod(**inputs) 2025-08-14T21:43:43.9741655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9741723Z outputs = self.deberta( 2025-08-14T21:43:43.9742001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9742069Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9742321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9742399Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9742609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9742681Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9742949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:43.9743104Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:43.9743353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:43.9743438Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9743441Z 2025-08-14T21:43:43.9743534Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9743718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9743787Z return mod(**inputs) 2025-08-14T21:43:43.9744041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9744104Z outputs = self.deberta( 2025-08-14T21:43:43.9744360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9744430Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9744687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9744765Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9744966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9745045Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9745292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9745384Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9745634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9745704Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9745961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9746132Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9746137Z 2025-08-14T21:43:43.9746229Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9746417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9746478Z return mod(**inputs) 2025-08-14T21:43:43.9746735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9746815Z outputs = self.deberta( 2025-08-14T21:43:43.9747062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9747136Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9747382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9747465Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9747703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9747774Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9748015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9748101Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9748344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9748437Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9748683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:43.9748867Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9748872Z 2025-08-14T21:43:43.9748966Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9749148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9749216Z return mod(**inputs) 2025-08-14T21:43:43.9749466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9749539Z outputs = self.deberta( 2025-08-14T21:43:43.9749785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9749852Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9750107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9750185Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9750387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9750466Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9750712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9750803Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9751049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9751119Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9751371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9751540Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9751828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9751951Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9751955Z 2025-08-14T21:43:43.9752049Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9752241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9752328Z return mod(**inputs) 2025-08-14T21:43:43.9752586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9752650Z outputs = self.deberta( 2025-08-14T21:43:43.9752897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9752973Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9753219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9753316Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9753526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9753599Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9753852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9753936Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9754198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9754276Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9754543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9754749Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9754753Z 2025-08-14T21:43:43.9754846Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9755029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9755098Z return mod(**inputs) 2025-08-14T21:43:43.9755348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9755414Z outputs = self.deberta( 2025-08-14T21:43:43.9755668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9755734Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9755988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9756067Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9756266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9756347Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9756593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9756684Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9756932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9757001Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9757257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9757454Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9757458Z 2025-08-14T21:43:43.9757560Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9757749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9757810Z return mod(**inputs) 2025-08-14T21:43:43.9758098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9758162Z outputs = self.deberta( 2025-08-14T21:43:43.9758420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9758497Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9758757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9758864Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9759072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9759145Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9759405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9759492Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9759773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9759847Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9760118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9760305Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9760310Z 2025-08-14T21:43:43.9760406Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9760591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9760659Z return mod(**inputs) 2025-08-14T21:43:43.9760918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9760992Z outputs = self.deberta( 2025-08-14T21:43:43.9761247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9761314Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9761574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9761653Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9761867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9761939Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9762191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9762283Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9762537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9762612Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9762899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9763081Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9763386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9763522Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9763526Z 2025-08-14T21:43:43.9763622Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9763815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9763896Z return mod(**inputs) 2025-08-14T21:43:43.9764160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9764226Z outputs = self.deberta( 2025-08-14T21:43:43.9764480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9764558Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9764831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9764911Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9765124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9765198Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9765461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9765635Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9765895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9765994Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9766252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:43.9766332Z context_layer = torch.bmm( 2025-08-14T21:43:43.9766336Z 2025-08-14T21:43:43.9766435Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9766629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9766702Z return mod(**inputs) 2025-08-14T21:43:43.9766969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9767039Z outputs = self.deberta( 2025-08-14T21:43:43.9767311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9767381Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9767652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9767747Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9767960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9768044Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9768297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9768401Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9768657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9768727Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9768987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:43.9769163Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:43.9769168Z 2025-08-14T21:43:43.9769264Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9769460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9769521Z return mod(**inputs) 2025-08-14T21:43:43.9769786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9769871Z outputs = self.deberta( 2025-08-14T21:43:43.9770126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9770205Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9770460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9770549Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9770771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9770845Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9771105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9771191Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9771445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:43.9771574Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:43.9771830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:43.9771939Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9771943Z 2025-08-14T21:43:43.9772042Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9772229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9772299Z return mod(**inputs) 2025-08-14T21:43:43.9772557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9772631Z outputs = self.deberta( 2025-08-14T21:43:43.9772885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9772952Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9773211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9773292Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9773499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9773582Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9773834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9773952Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9774269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:43.9774350Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9774353Z 2025-08-14T21:43:43.9774456Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9774642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9774712Z return mod(**inputs) 2025-08-14T21:43:43.9774969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9775033Z outputs = self.deberta( 2025-08-14T21:43:43.9775295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9775362Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9775613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9775730Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9775939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9776018Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9776272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9776382Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9776670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:43.9776770Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:43.9776968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:43.9777034Z return self.act(input) 2025-08-14T21:43:43.9777037Z 2025-08-14T21:43:43.9777128Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9777332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9777393Z return mod(**inputs) 2025-08-14T21:43:43.9777655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9777727Z outputs = self.deberta( 2025-08-14T21:43:43.9777977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9778051Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9778301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9778379Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9778593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9778666Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9778925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:43.9779049Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:43.9779300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:43.9779386Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9779390Z 2025-08-14T21:43:43.9779486Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9779671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9779743Z return mod(**inputs) 2025-08-14T21:43:43.9779999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9780071Z outputs = self.deberta( 2025-08-14T21:43:43.9780330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9780398Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9780650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9780728Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9780937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9781009Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9781252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9781367Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9781615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9781685Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9781939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9782108Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9782153Z 2025-08-14T21:43:43.9782255Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9782436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9782495Z return mod(**inputs) 2025-08-14T21:43:43.9782751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9782817Z outputs = self.deberta( 2025-08-14T21:43:43.9783084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9783153Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9783415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9783502Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9783704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9783775Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9784027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9784113Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9784363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9784433Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9784675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:43.9784846Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9784851Z 2025-08-14T21:43:43.9784943Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9785129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9785189Z return mod(**inputs) 2025-08-14T21:43:43.9785438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9785507Z outputs = self.deberta( 2025-08-14T21:43:43.9785754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9785820Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9786076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9786155Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9786364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9786436Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9786684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9786778Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9787045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9787125Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9787375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9787548Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9787844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9787996Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9788000Z 2025-08-14T21:43:43.9788098Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9788281Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9788342Z return mod(**inputs) 2025-08-14T21:43:43.9788621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9788687Z outputs = self.deberta( 2025-08-14T21:43:43.9788933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9789020Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9789271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9789358Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9789559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9789629Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9789884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9789966Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9790219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9790290Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9790540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9790741Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9790744Z 2025-08-14T21:43:43.9790838Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9791027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9791089Z return mod(**inputs) 2025-08-14T21:43:43.9791339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9791409Z outputs = self.deberta( 2025-08-14T21:43:43.9791657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9791724Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9791980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9792057Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9792266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9792337Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9792585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9792693Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9792939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9793008Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9793259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9793449Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9793466Z 2025-08-14T21:43:43.9793568Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9793748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9793811Z return mod(**inputs) 2025-08-14T21:43:43.9794070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9794134Z outputs = self.deberta( 2025-08-14T21:43:43.9794411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9794479Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9795105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9795196Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9795398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9795477Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9795727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9795813Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9796073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9796144Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9796397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9796583Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9796587Z 2025-08-14T21:43:43.9796682Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9796870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9796933Z return mod(**inputs) 2025-08-14T21:43:43.9797192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9797267Z outputs = self.deberta( 2025-08-14T21:43:43.9797525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9797603Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9797860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9797941Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9798163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9798237Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9798495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9798590Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9798858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9798938Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9799187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9799364Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9799686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9799805Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9799808Z 2025-08-14T21:43:43.9799907Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9800086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9800148Z return mod(**inputs) 2025-08-14T21:43:43.9800424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9800490Z outputs = self.deberta( 2025-08-14T21:43:43.9800766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9800836Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9801089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9801176Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9801381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9801454Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9801716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9801803Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9802065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9802138Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9802391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:43.9802470Z context_layer = torch.bmm( 2025-08-14T21:43:43.9802473Z 2025-08-14T21:43:43.9802568Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9802758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9802843Z return mod(**inputs) 2025-08-14T21:43:43.9803103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9803171Z outputs = self.deberta( 2025-08-14T21:43:43.9803422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9803489Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9803748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9803828Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9804039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9804110Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9804361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9804477Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9804740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9804814Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9805085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:43.9805274Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:43.9805297Z 2025-08-14T21:43:43.9805412Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9805805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9805881Z return mod(**inputs) 2025-08-14T21:43:43.9806183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9806259Z outputs = self.deberta( 2025-08-14T21:43:43.9806575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9806659Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9806934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9807024Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9807238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9807312Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9807574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9807663Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9807923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:43.9808035Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:43.9808292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:43.9808383Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9808387Z 2025-08-14T21:43:43.9808483Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9808677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9808738Z return mod(**inputs) 2025-08-14T21:43:43.9808994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9809066Z outputs = self.deberta( 2025-08-14T21:43:43.9809321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9809396Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9809653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9809734Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9809948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9810022Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9810277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9810397Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9810692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:43.9810797Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9810801Z 2025-08-14T21:43:43.9810899Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9811086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9811158Z return mod(**inputs) 2025-08-14T21:43:43.9811414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9811502Z outputs = self.deberta( 2025-08-14T21:43:43.9811760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9811830Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9812090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9812174Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9812402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9812484Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9812749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9812868Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9813123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:43.9813226Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:43.9813430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:43.9813498Z return self.act(input) 2025-08-14T21:43:43.9813502Z 2025-08-14T21:43:43.9813607Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9813794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9813857Z return mod(**inputs) 2025-08-14T21:43:43.9814175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9814239Z outputs = self.deberta( 2025-08-14T21:43:43.9814491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9814567Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9814817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9814901Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9815107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9815181Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9815440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:43.9815564Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:43.9815830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:43.9815908Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9815912Z 2025-08-14T21:43:43.9816007Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9816198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9816281Z return mod(**inputs) 2025-08-14T21:43:43.9816539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9816614Z outputs = self.deberta( 2025-08-14T21:43:43.9816872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9816944Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9817199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9817301Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9817516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9817588Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9817840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9817938Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9818259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9818340Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9818598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9818771Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9818784Z 2025-08-14T21:43:43.9818877Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9819055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9819123Z return mod(**inputs) 2025-08-14T21:43:43.9819372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9819435Z outputs = self.deberta( 2025-08-14T21:43:43.9819691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9819760Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9820014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9820091Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9820296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9820374Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9820619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9820705Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9820959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9821027Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9821278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:43.9821442Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9821447Z 2025-08-14T21:43:43.9821539Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9821724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9821784Z return mod(**inputs) 2025-08-14T21:43:43.9822037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9822117Z outputs = self.deberta( 2025-08-14T21:43:43.9822365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9822439Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9822688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9822763Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9822987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9823059Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9823309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9823393Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9823637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9823731Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9823980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9824173Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9824461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9824582Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9824586Z 2025-08-14T21:43:43.9824687Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9824868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9824937Z return mod(**inputs) 2025-08-14T21:43:43.9825189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9825252Z outputs = self.deberta( 2025-08-14T21:43:43.9825505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9825574Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9825818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9825905Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9826104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9826182Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9826428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9826511Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9826769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9826842Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9827103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9827304Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9827307Z 2025-08-14T21:43:43.9827402Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9827594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9827674Z return mod(**inputs) 2025-08-14T21:43:43.9827931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9828006Z outputs = self.deberta( 2025-08-14T21:43:43.9828273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9828349Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9828595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9828692Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9828905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9828979Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9829232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9829317Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9829576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9829656Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9829916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9830109Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9830120Z 2025-08-14T21:43:43.9830213Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9830396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9830463Z return mod(**inputs) 2025-08-14T21:43:43.9830716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9830780Z outputs = self.deberta( 2025-08-14T21:43:43.9831037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9831106Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9831358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9831438Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9831639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9831719Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9831964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9832049Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9832302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9832374Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9832627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9832803Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9832807Z 2025-08-14T21:43:43.9832902Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9833093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9833153Z return mod(**inputs) 2025-08-14T21:43:43.9833410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9833489Z outputs = self.deberta( 2025-08-14T21:43:43.9833737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9833813Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9834060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9834137Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9834364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9834436Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9834687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9834773Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9835019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9835116Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9835362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9835559Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9835848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9835968Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9835972Z 2025-08-14T21:43:43.9836073Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9836255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9836323Z return mod(**inputs) 2025-08-14T21:43:43.9836576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9836640Z outputs = self.deberta( 2025-08-14T21:43:43.9836895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9836961Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9837208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9837292Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9837497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9837575Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9837963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9838056Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9838313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9838385Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9838641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:43.9838712Z context_layer = torch.bmm( 2025-08-14T21:43:43.9838715Z 2025-08-14T21:43:43.9838811Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9839000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9839062Z return mod(**inputs) 2025-08-14T21:43:43.9839361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9839433Z outputs = self.deberta( 2025-08-14T21:43:43.9839679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9839754Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9840000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9840106Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9840315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9840389Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9840645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9840735Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9841012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9841095Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9841375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:43.9841560Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:43.9841572Z 2025-08-14T21:43:43.9841668Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9841855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9841926Z return mod(**inputs) 2025-08-14T21:43:43.9842185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9842252Z outputs = self.deberta( 2025-08-14T21:43:43.9842515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9842586Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9842849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9842927Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9843136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9843216Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9843470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9843556Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9843815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:43.9843925Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:43.9844187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:43.9844266Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9844270Z 2025-08-14T21:43:43.9844366Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9844560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9844623Z return mod(**inputs) 2025-08-14T21:43:43.9844885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9844969Z outputs = self.deberta( 2025-08-14T21:43:43.9845222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9845298Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9845610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9845704Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9845929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9846032Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9846303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9846419Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9846689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:43.9846778Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9846782Z 2025-08-14T21:43:43.9846904Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9847100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9847178Z return mod(**inputs) 2025-08-14T21:43:43.9847439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9847515Z outputs = self.deberta( 2025-08-14T21:43:43.9847767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9847836Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9848095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9848177Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9848394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9848467Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9848720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9848838Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9849092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:43.9849203Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:43.9849403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:43.9849470Z return self.act(input) 2025-08-14T21:43:43.9849474Z 2025-08-14T21:43:43.9849574Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9849764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9849826Z return mod(**inputs) 2025-08-14T21:43:43.9850091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9850156Z outputs = self.deberta( 2025-08-14T21:43:43.9850420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9850489Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9850742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9850828Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9851058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9851138Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9851391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:43.9851516Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:43.9851777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:43.9851876Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9851880Z 2025-08-14T21:43:43.9851975Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9852203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9852269Z return mod(**inputs) 2025-08-14T21:43:43.9852535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9852613Z outputs = self.deberta( 2025-08-14T21:43:43.9852867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9852943Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9853213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9853304Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9853510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9853583Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9853841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9853928Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9854184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9854265Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9854522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9854706Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9854711Z 2025-08-14T21:43:43.9854807Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9854994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9855064Z return mod(**inputs) 2025-08-14T21:43:43.9855321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9855393Z outputs = self.deberta( 2025-08-14T21:43:43.9855652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9855720Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9855984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9856063Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9856274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9856356Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9856611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9856725Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9856980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9857055Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9857318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:43.9857487Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9857505Z 2025-08-14T21:43:43.9857611Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9857801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9857863Z return mod(**inputs) 2025-08-14T21:43:43.9858127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9858193Z outputs = self.deberta( 2025-08-14T21:43:43.9858471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9858541Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9858814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9858903Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9859112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9859185Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9859447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9859532Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9859795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9859867Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9860119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9860303Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9860592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9860724Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9860727Z 2025-08-14T21:43:43.9860822Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9861011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9861084Z return mod(**inputs) 2025-08-14T21:43:43.9861344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9861411Z outputs = self.deberta( 2025-08-14T21:43:43.9861672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9861740Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9862000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9862080Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9862287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9862369Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9862641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9862733Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9862984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9863056Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9863323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9863530Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9863534Z 2025-08-14T21:43:43.9863634Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9863818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9863881Z return mod(**inputs) 2025-08-14T21:43:43.9864138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9864213Z outputs = self.deberta( 2025-08-14T21:43:43.9864461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9864548Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9864798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9864885Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9865086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9865156Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9865409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9865493Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9865738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9865813Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9866060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9866259Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9866262Z 2025-08-14T21:43:43.9866356Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9866537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9866606Z return mod(**inputs) 2025-08-14T21:43:43.9866861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9866931Z outputs = self.deberta( 2025-08-14T21:43:43.9867187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9867254Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9867515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9867595Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9867809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9867883Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9868140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9868252Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9868514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9868587Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9868857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9869038Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9869058Z 2025-08-14T21:43:43.9869174Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9869359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9869422Z return mod(**inputs) 2025-08-14T21:43:43.9869689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9869758Z outputs = self.deberta( 2025-08-14T21:43:43.9870032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9870103Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9870372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9870462Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9870670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9870743Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9871003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9871091Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9871348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9871420Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9871677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9871861Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9872144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9872272Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9872276Z 2025-08-14T21:43:43.9872371Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9872551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9872622Z return mod(**inputs) 2025-08-14T21:43:43.9872872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9872942Z outputs = self.deberta( 2025-08-14T21:43:43.9873189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9873257Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9873510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9873587Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9873788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9873867Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9874150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9874242Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9874487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9874557Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9874808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:43.9874891Z context_layer = torch.bmm( 2025-08-14T21:43:43.9874895Z 2025-08-14T21:43:43.9874994Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9875175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9875235Z return mod(**inputs) 2025-08-14T21:43:43.9875497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9875561Z outputs = self.deberta( 2025-08-14T21:43:43.9875825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9875903Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9876166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9876254Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9876453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9876526Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9876780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9876866Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9877116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9877196Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9877454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:43.9877638Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:43.9877642Z 2025-08-14T21:43:43.9877741Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9877935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9878006Z return mod(**inputs) 2025-08-14T21:43:43.9878271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9878345Z outputs = self.deberta( 2025-08-14T21:43:43.9878607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9878677Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9878956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9879035Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9879249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9879323Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9879575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9879688Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9879940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:43.9880048Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:43.9880309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:43.9880389Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9880393Z 2025-08-14T21:43:43.9880510Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9880699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9880761Z return mod(**inputs) 2025-08-14T21:43:43.9881029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9881093Z outputs = self.deberta( 2025-08-14T21:43:43.9881358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9881456Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9881709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9881809Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9882017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9882092Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9882350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9882462Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9882724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:43.9882803Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9882806Z 2025-08-14T21:43:43.9882901Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9883091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9883155Z return mod(**inputs) 2025-08-14T21:43:43.9883414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9883479Z outputs = self.deberta( 2025-08-14T21:43:43.9883730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9883806Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9884061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9884140Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9884354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9884427Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9884685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9884796Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9885048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:43.9885159Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:43.9885358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:43.9885448Z return self.act(input) 2025-08-14T21:43:43.9885451Z 2025-08-14T21:43:43.9885619Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9885819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9885892Z return mod(**inputs) 2025-08-14T21:43:43.9886156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9886224Z outputs = self.deberta( 2025-08-14T21:43:43.9886514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9886584Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9886857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9886935Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9887140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9887238Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9887489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:43.9887637Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:43.9887891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:43.9887970Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9887974Z 2025-08-14T21:43:43.9888079Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9888266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9888329Z return mod(**inputs) 2025-08-14T21:43:43.9888591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9888657Z outputs = self.deberta( 2025-08-14T21:43:43.9888919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9888987Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9889240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9889329Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9889533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9889607Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9889864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9889953Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9890212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9890285Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9890537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9890718Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9890722Z 2025-08-14T21:43:43.9890816Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9891010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9891072Z return mod(**inputs) 2025-08-14T21:43:43.9891327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9891421Z outputs = self.deberta( 2025-08-14T21:43:43.9891681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9891755Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9892010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9892109Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9892321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9892394Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9892653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9892749Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9893024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9893107Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9893375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:43.9893545Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9893550Z 2025-08-14T21:43:43.9893654Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9893839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9893906Z return mod(**inputs) 2025-08-14T21:43:43.9894163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9894229Z outputs = self.deberta( 2025-08-14T21:43:43.9894493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9894561Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9894818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9894904Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9895113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9895193Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9895446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9895530Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9895796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9895868Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9896131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9896305Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9896601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9896731Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9896734Z 2025-08-14T21:43:43.9896827Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9897015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9897093Z return mod(**inputs) 2025-08-14T21:43:43.9897352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9897423Z outputs = self.deberta( 2025-08-14T21:43:43.9897677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9897744Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9898008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9898102Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9898314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9898389Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9898641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9898733Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9899007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9899087Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9899350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9899545Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9899548Z 2025-08-14T21:43:43.9899650Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9899833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9899898Z return mod(**inputs) 2025-08-14T21:43:43.9900155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9900221Z outputs = self.deberta( 2025-08-14T21:43:43.9900480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9900550Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9900805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9900893Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9901098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9901178Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9901429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9901514Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9901777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9901847Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9902101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9902301Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9902304Z 2025-08-14T21:43:43.9902401Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9902600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9902662Z return mod(**inputs) 2025-08-14T21:43:43.9902941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9903015Z outputs = self.deberta( 2025-08-14T21:43:43.9903269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9903344Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9903595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9903700Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9903914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9903987Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9904236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9904341Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9904604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9904684Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9904954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9905133Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9905137Z 2025-08-14T21:43:43.9905241Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9905429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9905499Z return mod(**inputs) 2025-08-14T21:43:43.9905755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9905821Z outputs = self.deberta( 2025-08-14T21:43:43.9906079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9906148Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9906398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9906485Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9906693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9906773Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9907025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9907112Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9907371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9907444Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9907702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9907879Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9908169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9908301Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9908304Z 2025-08-14T21:43:43.9908411Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9908600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9908679Z return mod(**inputs) 2025-08-14T21:43:43.9908931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9909002Z outputs = self.deberta( 2025-08-14T21:43:43.9909250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9909317Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9909583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9909661Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9909866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9909938Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9910184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9910286Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9910535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9910627Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9910873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:43.9910941Z context_layer = torch.bmm( 2025-08-14T21:43:43.9910944Z 2025-08-14T21:43:43.9911041Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9911221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9911280Z return mod(**inputs) 2025-08-14T21:43:43.9911538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9911601Z outputs = self.deberta( 2025-08-14T21:43:43.9911853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9911920Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9912169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9912253Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9912453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9912531Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9912775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9912860Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9913116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9913186Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9913435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:43.9913615Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:43.9913619Z 2025-08-14T21:43:43.9913713Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9913901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9913962Z return mod(**inputs) 2025-08-14T21:43:43.9914211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9914301Z outputs = self.deberta( 2025-08-14T21:43:43.9914547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9914621Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9914867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9914944Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9915170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9915241Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9915486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9915579Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9915830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:43.9915961Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:43.9916217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:43.9916311Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9916315Z 2025-08-14T21:43:43.9916422Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9916610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9916680Z return mod(**inputs) 2025-08-14T21:43:43.9916938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9917004Z outputs = self.deberta( 2025-08-14T21:43:43.9917263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9917334Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9917590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9917679Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9917887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9917970Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9918223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9918335Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9918598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:43.9918675Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9918678Z 2025-08-14T21:43:43.9918783Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9918967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9919030Z return mod(**inputs) 2025-08-14T21:43:43.9919297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9919363Z outputs = self.deberta( 2025-08-14T21:43:43.9919616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9919692Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9919946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9920050Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9920259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9920332Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9920594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9920702Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9920977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:43.9921080Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:43.9921279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:43.9921356Z return self.act(input) 2025-08-14T21:43:43.9921359Z 2025-08-14T21:43:43.9921453Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9921655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9921726Z return mod(**inputs) 2025-08-14T21:43:43.9922000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9922074Z outputs = self.deberta( 2025-08-14T21:43:43.9922328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9922396Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9922655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9922735Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9922950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9923024Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9923276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:43.9923408Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:43.9923658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:43.9923737Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9923748Z 2025-08-14T21:43:43.9923841Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9924028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9924098Z return mod(**inputs) 2025-08-14T21:43:43.9924353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9924418Z outputs = self.deberta( 2025-08-14T21:43:43.9924674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9924740Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9924997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9925078Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9925283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9925365Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9925701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9925836Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9926116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9926194Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9926469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9926652Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9926681Z 2025-08-14T21:43:43.9926782Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9926979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9927040Z return mod(**inputs) 2025-08-14T21:43:43.9927303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9927369Z outputs = self.deberta( 2025-08-14T21:43:43.9927640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9927718Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9927987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9928069Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9928282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9928356Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9928614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9928702Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9928956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9929035Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9929287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:43.9929460Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9929464Z 2025-08-14T21:43:43.9929561Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9929747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9929815Z return mod(**inputs) 2025-08-14T21:43:43.9930071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9930144Z outputs = self.deberta( 2025-08-14T21:43:43.9930401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9930469Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9930729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9930809Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9931017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9931099Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9931351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9931445Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9931717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9931789Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9932047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9932222Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9932513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9932659Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9932662Z 2025-08-14T21:43:43.9932761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9932957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9933020Z return mod(**inputs) 2025-08-14T21:43:43.9933292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9933366Z outputs = self.deberta( 2025-08-14T21:43:43.9933632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9933709Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9933965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9934045Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9934258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9934330Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9934601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9934683Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9934927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9935007Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9935253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9935445Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9935456Z 2025-08-14T21:43:43.9935548Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9935729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9935798Z return mod(**inputs) 2025-08-14T21:43:43.9936047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9936111Z outputs = self.deberta( 2025-08-14T21:43:43.9936365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9936434Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9936688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9936765Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9936965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9937045Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9937292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9937395Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9937809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9937889Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9938145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9938377Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9938381Z 2025-08-14T21:43:43.9938475Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9938667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9938729Z return mod(**inputs) 2025-08-14T21:43:43.9938989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9939052Z outputs = self.deberta( 2025-08-14T21:43:43.9939326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9939403Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9939675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9939762Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9939967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9940038Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9940294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9940381Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9940630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9940710Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9940958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9941138Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9941143Z 2025-08-14T21:43:43.9941237Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9941417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9941484Z return mod(**inputs) 2025-08-14T21:43:43.9941734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9941806Z outputs = self.deberta( 2025-08-14T21:43:43.9942054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9942121Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9942373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9942451Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9942654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9942733Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9942977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9943092Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9943340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9943411Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9943665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9943837Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9944144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9944263Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9944266Z 2025-08-14T21:43:43.9944361Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9944549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9944611Z return mod(**inputs) 2025-08-14T21:43:43.9944881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9944945Z outputs = self.deberta( 2025-08-14T21:43:43.9945232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9945307Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9945562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9945641Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9945854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9945927Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9946184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9946270Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9946520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9946600Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9946859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:43.9946936Z context_layer = torch.bmm( 2025-08-14T21:43:43.9946939Z 2025-08-14T21:43:43.9947035Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9947223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9947303Z return mod(**inputs) 2025-08-14T21:43:43.9947559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9947623Z outputs = self.deberta( 2025-08-14T21:43:43.9947881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9947948Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9948203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9948284Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9948488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9948569Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9948816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9948923Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9949179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9949249Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9949504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:43.9949676Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:43.9949695Z 2025-08-14T21:43:43.9949789Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9949979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9950039Z return mod(**inputs) 2025-08-14T21:43:43.9950296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9950362Z outputs = self.deberta( 2025-08-14T21:43:43.9950626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9950701Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9950961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9951048Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9951254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9951326Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9951582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9951668Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9951915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:43.9952029Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:43.9952278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:43.9952363Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9952366Z 2025-08-14T21:43:43.9952461Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9952643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9952711Z return mod(**inputs) 2025-08-14T21:43:43.9952963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9953033Z outputs = self.deberta( 2025-08-14T21:43:43.9953281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9953350Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9953608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9953685Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9953887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9953970Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9954217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9954330Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9954592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:43.9954667Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9954672Z 2025-08-14T21:43:43.9954774Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9954954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9955020Z return mod(**inputs) 2025-08-14T21:43:43.9955268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9955349Z outputs = self.deberta( 2025-08-14T21:43:43.9955600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9955666Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9955909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9955994Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9956211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9956290Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9956553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9956661Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9956922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:43.9957026Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:43.9957233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:43.9957300Z return self.act(input) 2025-08-14T21:43:43.9957303Z 2025-08-14T21:43:43.9957399Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9957596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9957659Z return mod(**inputs) 2025-08-14T21:43:43.9957912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9957985Z outputs = self.deberta( 2025-08-14T21:43:43.9958236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9958312Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9958571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9958647Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9958856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9958929Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9959177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:43.9959307Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:43.9959558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:43.9959644Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9959648Z 2025-08-14T21:43:43.9959743Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9959929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9960019Z return mod(**inputs) 2025-08-14T21:43:43.9960281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9960357Z outputs = self.deberta( 2025-08-14T21:43:43.9960610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9960678Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9960942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9961038Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9961255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9961328Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9961586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9961680Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9961950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9962025Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9962317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9962494Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9962499Z 2025-08-14T21:43:43.9962603Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9962791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9962854Z return mod(**inputs) 2025-08-14T21:43:43.9963118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9963184Z outputs = self.deberta( 2025-08-14T21:43:43.9963448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9963516Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9963770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9963857Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9964061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9964133Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9964394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9964479Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9964753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9964825Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9965079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:43.9965251Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9965255Z 2025-08-14T21:43:43.9965350Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9965599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9965670Z return mod(**inputs) 2025-08-14T21:43:43.9965930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9966024Z outputs = self.deberta( 2025-08-14T21:43:43.9966285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9966354Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9966622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9966704Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9966936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9967008Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9967259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9967355Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9967612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9967716Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9967974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9968163Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9968462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9968587Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9968591Z 2025-08-14T21:43:43.9968692Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9968878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9968942Z return mod(**inputs) 2025-08-14T21:43:43.9969205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9969270Z outputs = self.deberta( 2025-08-14T21:43:43.9969559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9969633Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9969887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9969974Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9970179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9970254Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9970514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9970600Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9970858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9970929Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9971183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9971387Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9971390Z 2025-08-14T21:43:43.9971486Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9971675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9971767Z return mod(**inputs) 2025-08-14T21:43:43.9972036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9972108Z outputs = self.deberta( 2025-08-14T21:43:43.9972372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9972441Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9972712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9972808Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9973030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9973102Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9973346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9973436Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9973718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9973790Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9974058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:43.9974252Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:43.9974255Z 2025-08-14T21:43:43.9974356Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9974540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9974600Z return mod(**inputs) 2025-08-14T21:43:43.9974869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9974933Z outputs = self.deberta( 2025-08-14T21:43:43.9975194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9975261Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9975516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9975602Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9975807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9975879Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9976140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9976225Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9976489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9976561Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9976814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9977001Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9977005Z 2025-08-14T21:43:43.9977099Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9977293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9977353Z return mod(**inputs) 2025-08-14T21:43:43.9977615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9977707Z outputs = self.deberta( 2025-08-14T21:43:43.9977964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9978039Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9978294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9978372Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9978602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9978676Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9978935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9979028Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9979274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9979365Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9979615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:43.9979802Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:43.9980093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:43.9980213Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:43.9980216Z 2025-08-14T21:43:43.9980315Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9980497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9980560Z return mod(**inputs) 2025-08-14T21:43:43.9980820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9980883Z outputs = self.deberta( 2025-08-14T21:43:43.9981131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9981204Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9981450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9981537Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9981737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9981809Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9982062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9982147Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9982400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9982472Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9982715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:43.9982792Z context_layer = torch.bmm( 2025-08-14T21:43:43.9982795Z 2025-08-14T21:43:43.9982890Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9983071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9983139Z return mod(**inputs) 2025-08-14T21:43:43.9983407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9983477Z outputs = self.deberta( 2025-08-14T21:43:43.9983725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9983793Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9984048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9984147Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9984360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9984432Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9984686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9984780Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9985056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9985128Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9985396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:43.9985573Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:43.9985577Z 2025-08-14T21:43:43.9985677Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9985859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9985920Z return mod(**inputs) 2025-08-14T21:43:43.9986178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9986241Z outputs = self.deberta( 2025-08-14T21:43:43.9986494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9986560Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9986809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9986894Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9987098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9987170Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9987423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9987507Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9987760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:43.9987870Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:43.9988124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:43.9988211Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9988214Z 2025-08-14T21:43:43.9988312Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9988505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9988568Z return mod(**inputs) 2025-08-14T21:43:43.9988823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9988914Z outputs = self.deberta( 2025-08-14T21:43:43.9989175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9989245Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9989513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9989592Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9989821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9989917Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9990163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9990282Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9990534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:43.9990619Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9990638Z 2025-08-14T21:43:43.9990735Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9990923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9991010Z return mod(**inputs) 2025-08-14T21:43:43.9991266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9991332Z outputs = self.deberta( 2025-08-14T21:43:43.9991590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9991658Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9991917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9991997Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9992204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9992284Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9992534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:43.9992651Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:43.9992901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:43.9993005Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:43.9993212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:43.9993280Z return self.act(input) 2025-08-14T21:43:43.9993283Z 2025-08-14T21:43:43.9993377Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9993570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9993632Z return mod(**inputs) 2025-08-14T21:43:43.9993895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9993959Z outputs = self.deberta( 2025-08-14T21:43:43.9994209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9994286Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9994537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9994649Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9994856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9994930Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9995186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:43.9995310Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:43.9995568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:43.9995672Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:43.9995675Z 2025-08-14T21:43:43.9995770Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9995962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9996025Z return mod(**inputs) 2025-08-14T21:43:43.9996281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9996368Z outputs = self.deberta( 2025-08-14T21:43:43.9996626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9996703Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9996990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9997074Z output_states, attn_weights = layer_module( 2025-08-14T21:43:43.9997295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:43.9997372Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:43.9997633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:43.9997731Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:43.9997993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:43.9998077Z self_output, att_matrix = self.self( 2025-08-14T21:43:43.9998338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:43.9998518Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:43.9998524Z 2025-08-14T21:43:43.9998631Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:43.9998833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:43.9998902Z return mod(**inputs) 2025-08-14T21:43:43.9999162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:43.9999227Z outputs = self.deberta( 2025-08-14T21:43:43.9999490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:43.9999559Z encoder_outputs = self.encoder( 2025-08-14T21:43:43.9999817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:43.9999904Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0000112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0000190Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0000488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0000593Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0000866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0000939Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0001205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:44.0001376Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0001399Z 2025-08-14T21:43:44.0001498Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0001695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0001762Z return mod(**inputs) 2025-08-14T21:43:44.0002054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0002126Z outputs = self.deberta( 2025-08-14T21:43:44.0002436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0002519Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0002828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0002918Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0003159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0003240Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0003537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0003632Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0003925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0004014Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0004314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:44.0004516Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:44.0004836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:44.0004975Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:44.0004979Z 2025-08-14T21:43:44.0005094Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0005299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0005371Z return mod(**inputs) 2025-08-14T21:43:44.0005753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0005835Z outputs = self.deberta( 2025-08-14T21:43:44.0006143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0006224Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0006523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0006626Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0006879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0006970Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0007274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0007372Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0007662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0007744Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0008026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:44.0008279Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:44.0008283Z 2025-08-14T21:43:44.0008392Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0008608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0008679Z return mod(**inputs) 2025-08-14T21:43:44.0008970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0009068Z outputs = self.deberta( 2025-08-14T21:43:44.0009352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0009451Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0009735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0009830Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0010067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0010149Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0010425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0010529Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0010812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0010898Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0011178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:44.0011392Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:44.0011403Z 2025-08-14T21:43:44.0011509Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0011716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0011791Z return mod(**inputs) 2025-08-14T21:43:44.0012078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0012150Z outputs = self.deberta( 2025-08-14T21:43:44.0012439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0012519Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0012807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0012896Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0013124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0013212Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0013490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0013613Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0013910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0013990Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0014285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:44.0014482Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0014504Z 2025-08-14T21:43:44.0014612Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0014830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0014899Z return mod(**inputs) 2025-08-14T21:43:44.0015195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0015269Z outputs = self.deberta( 2025-08-14T21:43:44.0015565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0015646Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0015918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0016001Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0016224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0016305Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0016594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0016691Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0016969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0017058Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0017343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:44.0017546Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0017869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:44.0018005Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:44.0018009Z 2025-08-14T21:43:44.0018122Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0018333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0018408Z return mod(**inputs) 2025-08-14T21:43:44.0018697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0018769Z outputs = self.deberta( 2025-08-14T21:43:44.0019064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0019142Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0019425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0019520Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0019747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0019836Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0020138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0020233Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0020523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0020602Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0020891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:44.0020984Z context_layer = torch.bmm( 2025-08-14T21:43:44.0020988Z 2025-08-14T21:43:44.0021094Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0021308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0021376Z return mod(**inputs) 2025-08-14T21:43:44.0021665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0021744Z outputs = self.deberta( 2025-08-14T21:43:44.0022906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0022999Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0023305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0023398Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0023643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0023726Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0024012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0024119Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0024404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0024495Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0024782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:44.0024983Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:44.0024997Z 2025-08-14T21:43:44.0025105Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0025317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0025396Z return mod(**inputs) 2025-08-14T21:43:44.0025695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0025769Z outputs = self.deberta( 2025-08-14T21:43:44.0026072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0026151Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0026450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0026541Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0026782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0026872Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0027161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0027289Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0027576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:44.0027697Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:44.0027987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:44.0028074Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0028078Z 2025-08-14T21:43:44.0028206Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0028423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0028492Z return mod(**inputs) 2025-08-14T21:43:44.0028784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0028858Z outputs = self.deberta( 2025-08-14T21:43:44.0029140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0029239Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0029524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0029629Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0029867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0029949Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0030236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:44.0030360Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:44.0030642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:44.0030736Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0030741Z 2025-08-14T21:43:44.0030847Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0031064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0031135Z return mod(**inputs) 2025-08-14T21:43:44.0031422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0031503Z outputs = self.deberta( 2025-08-14T21:43:44.0031787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0031863Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0032152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0032240Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0032477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0032558Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0032839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:44.0032968Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:44.0033247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:44.0033369Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:44.0033591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:44.0033687Z return self.act(input) 2025-08-14T21:43:44.0033691Z 2025-08-14T21:43:44.0033803Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0034014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0034084Z return mod(**inputs) 2025-08-14T21:43:44.0034375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0034446Z outputs = self.deberta( 2025-08-14T21:43:44.0034755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0034831Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0035113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0035209Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0035438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0035540Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0035824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:44.0035986Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:44.0036278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:44.0036367Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0036371Z 2025-08-14T21:43:44.0036478Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0036692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0036762Z return mod(**inputs) 2025-08-14T21:43:44.0037054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0037126Z outputs = self.deberta( 2025-08-14T21:43:44.0037415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0037500Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0037938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0038045Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0038286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0038370Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0038670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0038771Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0039066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0039159Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0039458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:44.0039662Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:44.0039668Z 2025-08-14T21:43:44.0039777Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0039988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0040068Z return mod(**inputs) 2025-08-14T21:43:44.0040401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0040482Z outputs = self.deberta( 2025-08-14T21:43:44.0040765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0040843Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0041133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0041246Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0041479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0041571Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0041861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0041969Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0042285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0042368Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0042683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:44.0042871Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0042877Z 2025-08-14T21:43:44.0042991Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0043200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0043267Z return mod(**inputs) 2025-08-14T21:43:44.0043564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0043639Z outputs = self.deberta( 2025-08-14T21:43:44.0043927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0044010Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0044294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0044390Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0044623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0044705Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0044996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0045090Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0045384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0045464Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0045810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:44.0046019Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:44.0046352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:44.0046499Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:44.0046503Z 2025-08-14T21:43:44.0046612Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0046818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0046918Z return mod(**inputs) 2025-08-14T21:43:44.0047205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0047277Z outputs = self.deberta( 2025-08-14T21:43:44.0047571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0047647Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0047937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0048046Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0048280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0048370Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0048656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0048779Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0049062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0049142Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0049446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:44.0049644Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:44.0049648Z 2025-08-14T21:43:44.0049742Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0049933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0049998Z return mod(**inputs) 2025-08-14T21:43:44.0050260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0050326Z outputs = self.deberta( 2025-08-14T21:43:44.0050578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0050656Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0050907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0050994Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0051207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0051279Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0051530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0051615Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0051860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0051936Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0052183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:44.0052382Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:44.0052385Z 2025-08-14T21:43:44.0052476Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0052657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0052725Z return mod(**inputs) 2025-08-14T21:43:44.0052993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0053062Z outputs = self.deberta( 2025-08-14T21:43:44.0053315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0053383Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0053643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0053739Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0053949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0054028Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0054281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0054374Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0054642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0054714Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0054998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:44.0055175Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0055179Z 2025-08-14T21:43:44.0055280Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0055462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0055523Z return mod(**inputs) 2025-08-14T21:43:44.0055781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0055845Z outputs = self.deberta( 2025-08-14T21:43:44.0056107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0056177Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0056432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0056519Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0056726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0056801Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0057061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0057152Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0057418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0057492Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0057752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:44.0057943Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0058246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:44.0058383Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:44.0058387Z 2025-08-14T21:43:44.0058485Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0058696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0058768Z return mod(**inputs) 2025-08-14T21:43:44.0059041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0059107Z outputs = self.deberta( 2025-08-14T21:43:44.0059365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0059434Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0059711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0059790Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0059998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0060080Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0060332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0060459Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0060711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0060798Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0061057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:44.0061126Z context_layer = torch.bmm( 2025-08-14T21:43:44.0061130Z 2025-08-14T21:43:44.0061226Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0061415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0061477Z return mod(**inputs) 2025-08-14T21:43:44.0061739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0061803Z outputs = self.deberta( 2025-08-14T21:43:44.0062054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0062130Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0062381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0062468Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0062672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0062744Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0062998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0063087Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0063338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0063418Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0063668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:44.0063852Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:44.0063856Z 2025-08-14T21:43:44.0063952Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0064137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0064207Z return mod(**inputs) 2025-08-14T21:43:44.0064460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0064550Z outputs = self.deberta( 2025-08-14T21:43:44.0064802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0064870Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0065128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0065207Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0065429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0065511Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0065761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0065855Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0066106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:44.0066229Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:44.0066505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:44.0066587Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0066591Z 2025-08-14T21:43:44.0066694Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0066880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0066941Z return mod(**inputs) 2025-08-14T21:43:44.0067209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0067276Z outputs = self.deberta( 2025-08-14T21:43:44.0067535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0067613Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0067873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0067963Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0068176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0068253Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0068517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:44.0068631Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:44.0068907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:44.0068986Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0068990Z 2025-08-14T21:43:44.0069086Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0069279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0069342Z return mod(**inputs) 2025-08-14T21:43:44.0069598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0069670Z outputs = self.deberta( 2025-08-14T21:43:44.0069923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0069999Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0070251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0070347Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0070563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0070638Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0070895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:44.0071005Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:44.0071279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:44.0071393Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:44.0071593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:44.0071664Z return self.act(input) 2025-08-14T21:43:44.0071675Z 2025-08-14T21:43:44.0071771Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0071974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0072046Z return mod(**inputs) 2025-08-14T21:43:44.0072322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0072389Z outputs = self.deberta( 2025-08-14T21:43:44.0072653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0072722Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0072981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0073062Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0073271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0073351Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0073602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:44.0073728Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:44.0073992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:44.0074071Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0074075Z 2025-08-14T21:43:44.0074178Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0074360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0074425Z return mod(**inputs) 2025-08-14T21:43:44.0074687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0074763Z outputs = self.deberta( 2025-08-14T21:43:44.0075023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0075094Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0075352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0075438Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0075645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0075717Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0075976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0076079Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0076337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0076409Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0076659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:44.0076864Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:44.0076867Z 2025-08-14T21:43:44.0076964Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0077155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0077218Z return mod(**inputs) 2025-08-14T21:43:44.0077479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0077555Z outputs = self.deberta( 2025-08-14T21:43:44.0077833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0077905Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0078192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0078276Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0078498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0078573Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0078831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0078927Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0079187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0079269Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0079538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:44.0079704Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0079709Z 2025-08-14T21:43:44.0079813Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0079999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0080066Z return mod(**inputs) 2025-08-14T21:43:44.0080324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0080390Z outputs = self.deberta( 2025-08-14T21:43:44.0080655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0080724Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0080978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0081065Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0081273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0081353Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0081604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0081690Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0081970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0082041Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0082300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:44.0082474Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:44.0082762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:44.0082917Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:44.0082921Z 2025-08-14T21:43:44.0083015Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0083208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0083271Z return mod(**inputs) 2025-08-14T21:43:44.0083546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0083619Z outputs = self.deberta( 2025-08-14T21:43:44.0083886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0083956Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0084217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0084298Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0084511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0084587Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0084840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0084939Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0085198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0085272Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0085611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:44.0085827Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:44.0085831Z 2025-08-14T21:43:44.0085941Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0086137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0086209Z return mod(**inputs) 2025-08-14T21:43:44.0086511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0086587Z outputs = self.deberta( 2025-08-14T21:43:44.0086885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0086961Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0087232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0087327Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0087554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0087632Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0087903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0088017Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0088285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0088359Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0088618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:44.0088843Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:44.0088847Z 2025-08-14T21:43:44.0088946Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0089144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0089209Z return mod(**inputs) 2025-08-14T21:43:44.0089472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0089547Z outputs = self.deberta( 2025-08-14T21:43:44.0089824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0089904Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0090183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0090267Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0090490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0090566Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0090828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0090923Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0091183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0091263Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0091525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:44.0091709Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0091714Z 2025-08-14T21:43:44.0091819Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0092008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0092078Z return mod(**inputs) 2025-08-14T21:43:44.0092345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0092413Z outputs = self.deberta( 2025-08-14T21:43:44.0092684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0092755Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0093018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0093109Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0093324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0093406Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0093666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0093772Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0094036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0094110Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0094375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:44.0094559Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0094875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:44.0095010Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:44.0095013Z 2025-08-14T21:43:44.0095112Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0095312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0095376Z return mod(**inputs) 2025-08-14T21:43:44.0095654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0095729Z outputs = self.deberta( 2025-08-14T21:43:44.0096010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0096081Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0096354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0096435Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0096657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0096733Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0096995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0097092Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0097357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0097439Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0097701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:44.0097770Z context_layer = torch.bmm( 2025-08-14T21:43:44.0097774Z 2025-08-14T21:43:44.0097879Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0098071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0098135Z return mod(**inputs) 2025-08-14T21:43:44.0098409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0098474Z outputs = self.deberta( 2025-08-14T21:43:44.0098745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0098816Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0099082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0099169Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0099384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0099457Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0099731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0099841Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0100105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0100177Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0100435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:44.0100625Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:44.0100647Z 2025-08-14T21:43:44.0100746Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0100945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0101017Z return mod(**inputs) 2025-08-14T21:43:44.0101276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0101350Z outputs = self.deberta( 2025-08-14T21:43:44.0101619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0101696Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0101967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0102049Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0102266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0102339Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0102592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0102688Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0102942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:44.0103059Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:44.0103318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:44.0103396Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0103401Z 2025-08-14T21:43:44.0103505Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0103696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0103766Z return mod(**inputs) 2025-08-14T21:43:44.0104027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0104095Z outputs = self.deberta( 2025-08-14T21:43:44.0104365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0104437Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0104708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0104796Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0105005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0105087Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0105341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:44.0105452Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:44.0105742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:44.0105821Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0105825Z 2025-08-14T21:43:44.0105928Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0106117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0106182Z return mod(**inputs) 2025-08-14T21:43:44.0106444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0106526Z outputs = self.deberta( 2025-08-14T21:43:44.0106781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0106857Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0107109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0107200Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0107431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0107507Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0107792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:44.0107909Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:44.0108175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:44.0108283Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:44.0108486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:44.0108564Z return self.act(input) 2025-08-14T21:43:44.0108568Z 2025-08-14T21:43:44.0108665Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0108856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0108931Z return mod(**inputs) 2025-08-14T21:43:44.0109199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0109269Z outputs = self.deberta( 2025-08-14T21:43:44.0109527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0109594Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0109851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0109930Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0110139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0110212Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0110462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:44.0110593Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:44.0110844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:44.0110923Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0110935Z 2025-08-14T21:43:44.0111028Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0111213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0111299Z return mod(**inputs) 2025-08-14T21:43:44.0111553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0111621Z outputs = self.deberta( 2025-08-14T21:43:44.0111882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0111951Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0112209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0112305Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0112512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0112594Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0112850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0112941Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0113222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0113299Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0113584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:44.0113766Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:44.0113769Z 2025-08-14T21:43:44.0113867Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0114064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0114129Z return mod(**inputs) 2025-08-14T21:43:44.0114403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0114472Z outputs = self.deberta( 2025-08-14T21:43:44.0114740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0114821Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0115077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0115162Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0115376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0115453Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0115715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0115806Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0116063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0116147Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0116403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:44.0116579Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0116584Z 2025-08-14T21:43:44.0116683Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0116874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0116949Z return mod(**inputs) 2025-08-14T21:43:44.0117214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0117298Z outputs = self.deberta( 2025-08-14T21:43:44.0117576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0117650Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0117930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0118013Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0118247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0118333Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0118598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0118696Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0118965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0119058Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0119332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:44.0119531Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:44.0119847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:44.0119979Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:44.0119982Z 2025-08-14T21:43:44.0120085Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0120291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0120359Z return mod(**inputs) 2025-08-14T21:43:44.0120632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0120712Z outputs = self.deberta( 2025-08-14T21:43:44.0120984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0121067Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0121336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0121424Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0121650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0121731Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0122012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0122110Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0122379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0122466Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0122734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:44.0122947Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:44.0122958Z 2025-08-14T21:43:44.0123063Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0123262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0123356Z return mod(**inputs) 2025-08-14T21:43:44.0123636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0123705Z outputs = self.deberta( 2025-08-14T21:43:44.0123989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0124067Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0124365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0124486Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0124716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0124806Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0125089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0125187Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0125562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0125659Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0125983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:44.0126213Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:44.0126217Z 2025-08-14T21:43:44.0126328Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0126600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0126681Z return mod(**inputs) 2025-08-14T21:43:44.0126980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0127052Z outputs = self.deberta( 2025-08-14T21:43:44.0127339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0127425Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0127713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0127802Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0128042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0128124Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0128415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0128514Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0128799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0128890Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0129173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:44.0129382Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0129387Z 2025-08-14T21:43:44.0129495Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0129703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0129779Z return mod(**inputs) 2025-08-14T21:43:44.0130065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0130164Z outputs = self.deberta( 2025-08-14T21:43:44.0130447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0130523Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0130811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0130898Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0131146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0131233Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0131515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0131618Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0131904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0132002Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0132294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:44.0132509Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0132843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:44.0132981Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:44.0132985Z 2025-08-14T21:43:44.0133092Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0133312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0133379Z return mod(**inputs) 2025-08-14T21:43:44.0133671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0133751Z outputs = self.deberta( 2025-08-14T21:43:44.0134035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0134119Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0134408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0134500Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0134736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0134818Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0135106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0135203Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0135485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0135575Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0135857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:44.0135933Z context_layer = torch.bmm( 2025-08-14T21:43:44.0135944Z 2025-08-14T21:43:44.0136051Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0136260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0136336Z return mod(**inputs) 2025-08-14T21:43:44.0136641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0136713Z outputs = self.deberta( 2025-08-14T21:43:44.0137003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0137077Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0137367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0137472Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0137904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0138007Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0138308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0138413Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0138758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0138846Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0139180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:44.0139382Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:44.0139388Z 2025-08-14T21:43:44.0139495Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0139712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0139782Z return mod(**inputs) 2025-08-14T21:43:44.0140076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0140151Z outputs = self.deberta( 2025-08-14T21:43:44.0140439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0140525Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0140814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0140901Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0141147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0141230Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0141522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0141620Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0141906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:44.0142035Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:44.0142323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:44.0142417Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0142422Z 2025-08-14T21:43:44.0142528Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0142736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0142814Z return mod(**inputs) 2025-08-14T21:43:44.0143103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0143209Z outputs = self.deberta( 2025-08-14T21:43:44.0143495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0143572Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0143870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0143959Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0144193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0144307Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0144579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:44.0144699Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:44.0144955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:44.0145044Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0145063Z 2025-08-14T21:43:44.0145169Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0145354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0145439Z return mod(**inputs) 2025-08-14T21:43:44.0145697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0145763Z outputs = self.deberta( 2025-08-14T21:43:44.0146023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0146091Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0146340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0146428Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0146635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0146715Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0146972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:44.0147086Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:44.0147347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:44.0147453Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:44.0147663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:44.0147732Z return self.act(input) 2025-08-14T21:43:44.0147735Z 2025-08-14T21:43:44.0147833Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0148031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0148094Z return mod(**inputs) 2025-08-14T21:43:44.0148355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0148429Z outputs = self.deberta( 2025-08-14T21:43:44.0148694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0148769Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0149020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0149118Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0149335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0149409Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0149662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:44.0149795Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:44.0150046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:44.0150149Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0150152Z 2025-08-14T21:43:44.0150247Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0150432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0150505Z return mod(**inputs) 2025-08-14T21:43:44.0150761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0150856Z outputs = self.deberta( 2025-08-14T21:43:44.0151110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0151194Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0151455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0151535Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0151740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0151823Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0152077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0152177Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0152438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0152513Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0152782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:44.0152964Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:44.0152967Z 2025-08-14T21:43:44.0153071Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0153259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0153322Z return mod(**inputs) 2025-08-14T21:43:44.0153595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0153661Z outputs = self.deberta( 2025-08-14T21:43:44.0153934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0154003Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0154253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0154342Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0154547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0154620Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0154876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0154987Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0155249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0155321Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0155571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:44.0155743Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0155779Z 2025-08-14T21:43:44.0155876Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0156070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0156131Z return mod(**inputs) 2025-08-14T21:43:44.0156389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0156463Z outputs = self.deberta( 2025-08-14T21:43:44.0156732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0156802Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0157079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0157166Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0157387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0157461Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0157721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0157818Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0158078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0158162Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0158426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:44.0158605Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:44.0158907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:44.0159035Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:44.0159039Z 2025-08-14T21:43:44.0159145Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0159335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0159399Z return mod(**inputs) 2025-08-14T21:43:44.0159671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0159736Z outputs = self.deberta( 2025-08-14T21:43:44.0159996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0160074Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0160334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0160420Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0160632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0160707Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0161000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0161093Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0161363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0161446Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0161712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:44.0161946Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:44.0161950Z 2025-08-14T21:43:44.0162050Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0162260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0162332Z return mod(**inputs) 2025-08-14T21:43:44.0162606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0162697Z outputs = self.deberta( 2025-08-14T21:43:44.0162964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0163050Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0163328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0163414Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0163642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0163719Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0163985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0164084Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0164352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0164428Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0164703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:44.0164910Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:44.0164914Z 2025-08-14T21:43:44.0165023Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0165224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0165289Z return mod(**inputs) 2025-08-14T21:43:44.0165646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0165726Z outputs = self.deberta( 2025-08-14T21:43:44.0166021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0166099Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0166386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0166486Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0166729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0166812Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0167093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0167205Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0167475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0167550Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0167810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:44.0168004Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0168028Z 2025-08-14T21:43:44.0168129Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0168331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0168396Z return mod(**inputs) 2025-08-14T21:43:44.0168663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0168740Z outputs = self.deberta( 2025-08-14T21:43:44.0169013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0169084Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0169373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0169457Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0169676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0169751Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0170008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0170105Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0170360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0170440Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0170696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:44.0170880Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0171187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:44.0171315Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:44.0171318Z 2025-08-14T21:43:44.0171424Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0171616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0171679Z return mod(**inputs) 2025-08-14T21:43:44.0171949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0172016Z outputs = self.deberta( 2025-08-14T21:43:44.0172275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0172353Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0172614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0172701Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0172912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0173007Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0173273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0173362Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0173628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0173704Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0173968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:44.0174064Z context_layer = torch.bmm( 2025-08-14T21:43:44.0174067Z 2025-08-14T21:43:44.0174161Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0174340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0174410Z return mod(**inputs) 2025-08-14T21:43:44.0174658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0174740Z outputs = self.deberta( 2025-08-14T21:43:44.0174988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0175056Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0175333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0175415Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0175636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0175709Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0175963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0176055Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0176312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0176382Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0176647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:44.0176828Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:44.0176833Z 2025-08-14T21:43:44.0176940Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0177132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0177196Z return mod(**inputs) 2025-08-14T21:43:44.0177471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0177540Z outputs = self.deberta( 2025-08-14T21:43:44.0177810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0177880Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0178147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0178235Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0178453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0178528Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0178806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0178915Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0179177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:44.0179286Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:44.0179541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:44.0179628Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0179646Z 2025-08-14T21:43:44.0179742Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0179935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0179998Z return mod(**inputs) 2025-08-14T21:43:44.0180254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0180329Z outputs = self.deberta( 2025-08-14T21:43:44.0180583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0180672Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0180939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0181034Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0181251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0181326Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0181584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:44.0181705Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:44.0181967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:44.0182055Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0182060Z 2025-08-14T21:43:44.0182157Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0182347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0182420Z return mod(**inputs) 2025-08-14T21:43:44.0182694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0182760Z outputs = self.deberta( 2025-08-14T21:43:44.0183020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0183088Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0183346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0183426Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0183633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0183715Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0183967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:44.0184087Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:44.0184340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:44.0184445Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:44.0184651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:44.0184738Z return self.act(input) 2025-08-14T21:43:44.0184741Z 2025-08-14T21:43:44.0184837Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0185029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0185090Z return mod(**inputs) 2025-08-14T21:43:44.0185351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0185415Z outputs = self.deberta( 2025-08-14T21:43:44.0185680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0185754Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0186007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0186094Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0186300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0186389Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0186646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:44.0186788Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:44.0187049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:44.0187137Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0187140Z 2025-08-14T21:43:44.0187238Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0187434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0187499Z return mod(**inputs) 2025-08-14T21:43:44.0187762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0187838Z outputs = self.deberta( 2025-08-14T21:43:44.0188102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0188182Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0188448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0188534Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0188758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0188832Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0189083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0189180Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0189432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0189512Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0189766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:44.0189941Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:44.0189945Z 2025-08-14T21:43:44.0190048Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0190232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0190301Z return mod(**inputs) 2025-08-14T21:43:44.0190575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0190640Z outputs = self.deberta( 2025-08-14T21:43:44.0190901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0190970Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0191225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0191329Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0191535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0191616Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0191870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0191956Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0192235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0192310Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0192581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:44.0192759Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0192764Z 2025-08-14T21:43:44.0192864Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0193066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0193130Z return mod(**inputs) 2025-08-14T21:43:44.0193401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0193476Z outputs = self.deberta( 2025-08-14T21:43:44.0193744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0193823Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0194100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0194181Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0194406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0194483Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0194759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0194845Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0195103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0195185Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0195441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:44.0195615Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:44.0195914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:44.0196038Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:44.0196041Z 2025-08-14T21:43:44.0196146Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0196330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0196422Z return mod(**inputs) 2025-08-14T21:43:44.0196688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0196755Z outputs = self.deberta( 2025-08-14T21:43:44.0197015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0197085Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0197360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0197449Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0197658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0197740Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0197999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0198103Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0198368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0198458Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0198718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:44.0198930Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:44.0198933Z 2025-08-14T21:43:44.0199030Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0199225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0199291Z return mod(**inputs) 2025-08-14T21:43:44.0199551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0199627Z outputs = self.deberta( 2025-08-14T21:43:44.0199885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0199963Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0200227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0200309Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0200528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0200604Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0200858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0200957Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0201215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0201296Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0201553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:44.0201751Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:44.0201754Z 2025-08-14T21:43:44.0201859Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0202051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0202122Z return mod(**inputs) 2025-08-14T21:43:44.0202428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0202495Z outputs = self.deberta( 2025-08-14T21:43:44.0202761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0202831Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0203097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0203200Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0203412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0203495Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0203755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0203844Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0204131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0204208Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0204501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:44.0204692Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0204696Z 2025-08-14T21:43:44.0204799Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0205005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0205070Z return mod(**inputs) 2025-08-14T21:43:44.0205349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0205423Z outputs = self.deberta( 2025-08-14T21:43:44.0205785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0205876Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0206163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0206252Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0206493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0206576Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0206871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0206964Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0207228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0207313Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0207584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:44.0207773Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0208087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:44.0208224Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:44.0208228Z 2025-08-14T21:43:44.0208344Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0208588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0208663Z return mod(**inputs) 2025-08-14T21:43:44.0208955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0209027Z outputs = self.deberta( 2025-08-14T21:43:44.0209321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0209397Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0209705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0209799Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0210032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0210125Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0210406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0210518Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0210807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0210902Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0211187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:44.0211273Z context_layer = torch.bmm( 2025-08-14T21:43:44.0211276Z 2025-08-14T21:43:44.0211385Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0211602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0211672Z return mod(**inputs) 2025-08-14T21:43:44.0211961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0212043Z outputs = self.deberta( 2025-08-14T21:43:44.0212324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0212407Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0212692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0212782Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0213022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0213102Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0213386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0213491Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0213778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0213865Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0214150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:44.0214350Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:44.0214355Z 2025-08-14T21:43:44.0214470Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0214678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0214754Z return mod(**inputs) 2025-08-14T21:43:44.0215042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0215134Z outputs = self.deberta( 2025-08-14T21:43:44.0215425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0215502Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0215785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0215899Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0216131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0216225Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0216477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0216563Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0216837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:44.0216949Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:44.0217240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:44.0217321Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0217326Z 2025-08-14T21:43:44.0217424Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0217619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0217683Z return mod(**inputs) 2025-08-14T21:43:44.0217953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0218022Z outputs = self.deberta( 2025-08-14T21:43:44.0218281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0218363Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0218623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0218706Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0218932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0219008Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0219264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:44.0219375Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:44.0219626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:44.0219710Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0219715Z 2025-08-14T21:43:44.0219811Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0220002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0220065Z return mod(**inputs) 2025-08-14T21:43:44.0220325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0220398Z outputs = self.deberta( 2025-08-14T21:43:44.0220652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0220719Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0220978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0221077Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0221290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0221363Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0221616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:44.0221749Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:44.0222004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:44.0222115Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:44.0222315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:44.0222382Z return self.act(input) 2025-08-14T21:43:44.0222385Z 2025-08-14T21:43:44.0222483Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0222687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0222749Z return mod(**inputs) 2025-08-14T21:43:44.0223031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0223096Z outputs = self.deberta( 2025-08-14T21:43:44.0223354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0223422Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0223670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0223758Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0223965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0224038Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0224300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:44.0224427Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:44.0224689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:44.0224767Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0224770Z 2025-08-14T21:43:44.0224864Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0225060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0225124Z return mod(**inputs) 2025-08-14T21:43:44.0225391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0225456Z outputs = self.deberta( 2025-08-14T21:43:44.0225712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0225790Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0226048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0226127Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0226345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0226418Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0226681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0226791Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0227053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0227134Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0227394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:44.0227603Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:44.0227607Z 2025-08-14T21:43:44.0227706Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0227908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0227979Z return mod(**inputs) 2025-08-14T21:43:44.0228239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0228311Z outputs = self.deberta( 2025-08-14T21:43:44.0228583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0228653Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0228928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0229011Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0229220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0229302Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0229554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0229651Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0229905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0229977Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0230240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:44.0230414Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0230418Z 2025-08-14T21:43:44.0230524Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0230717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0230783Z return mod(**inputs) 2025-08-14T21:43:44.0231053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0231121Z outputs = self.deberta( 2025-08-14T21:43:44.0231386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0231463Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0231723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0231812Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0232026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0232101Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0232369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0232478Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0232742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0232818Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0233075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:44.0233261Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:44.0233587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:44.0233715Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:44.0233726Z 2025-08-14T21:43:44.0233827Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0234022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0234097Z return mod(**inputs) 2025-08-14T21:43:44.0234385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0234454Z outputs = self.deberta( 2025-08-14T21:43:44.0234746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0234821Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0235113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0235193Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0235408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0235492Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0235755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0235845Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0236115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0236194Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0236465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:44.0236673Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:44.0236677Z 2025-08-14T21:43:44.0236775Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0236976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0237043Z return mod(**inputs) 2025-08-14T21:43:44.0237316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0237385Z outputs = self.deberta( 2025-08-14T21:43:44.0237767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0237859Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0238146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0238245Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0238479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0238560Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0238847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0238989Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0239278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0239368Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0239651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:44.0239903Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:44.0239907Z 2025-08-14T21:43:44.0240015Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0240222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0240300Z return mod(**inputs) 2025-08-14T21:43:44.0240588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0240668Z outputs = self.deberta( 2025-08-14T21:43:44.0240985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0241064Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0241380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0241472Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0241704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0241794Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0242074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0242177Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0242458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0242537Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0242828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:44.0243025Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0243031Z 2025-08-14T21:43:44.0243144Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0243351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0243418Z return mod(**inputs) 2025-08-14T21:43:44.0243712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0243784Z outputs = self.deberta( 2025-08-14T21:43:44.0244066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0244152Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0244436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0244533Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0244767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0244847Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0245135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0245253Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0245607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0245698Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0245993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:44.0246203Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0246567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:44.0246700Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:44.0246704Z 2025-08-14T21:43:44.0246804Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0246997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0247069Z return mod(**inputs) 2025-08-14T21:43:44.0247350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0247421Z outputs = self.deberta( 2025-08-14T21:43:44.0247706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0247781Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0248053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0248137Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0248351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0248438Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0248694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0248795Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0249051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0249128Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0249396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:44.0249467Z context_layer = torch.bmm( 2025-08-14T21:43:44.0249470Z 2025-08-14T21:43:44.0249568Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0249769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0249834Z return mod(**inputs) 2025-08-14T21:43:44.0250101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0250170Z outputs = self.deberta( 2025-08-14T21:43:44.0250427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0250505Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0250765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0250853Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0251063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0251136Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0251400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0251533Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0251796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0251877Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0252139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:44.0252328Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:44.0252348Z 2025-08-14T21:43:44.0252449Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0252648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0252718Z return mod(**inputs) 2025-08-14T21:43:44.0252975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0253049Z outputs = self.deberta( 2025-08-14T21:43:44.0253319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0253389Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0253667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0253749Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0253955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0254037Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0254287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0254381Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0254635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:44.0254743Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:44.0255005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:44.0255084Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0255088Z 2025-08-14T21:43:44.0255190Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0255377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0255439Z return mod(**inputs) 2025-08-14T21:43:44.0255701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0255768Z outputs = self.deberta( 2025-08-14T21:43:44.0256022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0256101Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0256353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0256444Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0256652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0256729Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0257015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:44.0257139Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:44.0257446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:44.0257533Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0257538Z 2025-08-14T21:43:44.0257643Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0257862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0257931Z return mod(**inputs) 2025-08-14T21:43:44.0258213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0258527Z outputs = self.deberta( 2025-08-14T21:43:44.0258823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0258902Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0259155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0259239Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0259474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0259551Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0259833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:44.0259950Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:44.0260214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:44.0260330Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:44.0260542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:44.0260614Z return self.act(input) 2025-08-14T21:43:44.0260626Z 2025-08-14T21:43:44.0260725Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0260920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0260994Z return mod(**inputs) 2025-08-14T21:43:44.0261263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0261331Z outputs = self.deberta( 2025-08-14T21:43:44.0261604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0261677Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0261948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0262031Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0262246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0262330Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0262593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:44.0262722Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:44.0262993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:44.0263075Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0263078Z 2025-08-14T21:43:44.0263184Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0263376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0263460Z return mod(**inputs) 2025-08-14T21:43:44.0263751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0263817Z outputs = self.deberta( 2025-08-14T21:43:44.0264082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0264151Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0264409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0264523Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0264731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0264804Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0265070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0265158Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0265441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0265515Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0265790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:44.0265975Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:44.0265978Z 2025-08-14T21:43:44.0266075Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0266268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0266329Z return mod(**inputs) 2025-08-14T21:43:44.0266588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0266659Z outputs = self.deberta( 2025-08-14T21:43:44.0266914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0266984Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0267252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0267335Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0267554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0267628Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0267887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0267985Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0268248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0268327Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0268589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:44.0268763Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0268768Z 2025-08-14T21:43:44.0268873Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0269072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0269141Z return mod(**inputs) 2025-08-14T21:43:44.0269395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0269478Z outputs = self.deberta( 2025-08-14T21:43:44.0269748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0269818Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0270085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0270174Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0270403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0270485Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0270752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0270842Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0271119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0271209Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0271482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:44.0271681Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:44.0271984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:44.0272121Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:44.0272125Z 2025-08-14T21:43:44.0272226Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0272424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0272497Z return mod(**inputs) 2025-08-14T21:43:44.0272772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0272847Z outputs = self.deberta( 2025-08-14T21:43:44.0273128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0273200Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0273467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0273550Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0273769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0273846Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0274107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0274204Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0274467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0274540Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0274807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:44.0275012Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:44.0275016Z 2025-08-14T21:43:44.0275120Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0275313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0275397Z return mod(**inputs) 2025-08-14T21:43:44.0275672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0275738Z outputs = self.deberta( 2025-08-14T21:43:44.0276012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0276082Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0276348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0276456Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0276669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0276746Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0277015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0277107Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0277398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0277475Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0277754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:44.0277972Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:44.0277976Z 2025-08-14T21:43:44.0278087Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0278291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0278353Z return mod(**inputs) 2025-08-14T21:43:44.0278622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0278696Z outputs = self.deberta( 2025-08-14T21:43:44.0278961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0279039Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0279303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0279386Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0279607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0279682Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0279944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0280041Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0280303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0280381Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0280645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:44.0280830Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0280835Z 2025-08-14T21:43:44.0280943Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0281136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0281206Z return mod(**inputs) 2025-08-14T21:43:44.0281472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0281558Z outputs = self.deberta( 2025-08-14T21:43:44.0281825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0281894Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0282152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0282239Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0282472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0282557Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0282821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0282913Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0283203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0283280Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0283551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:44.0283753Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0284067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:44.0284204Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:44.0284207Z 2025-08-14T21:43:44.0284310Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0284516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0284581Z return mod(**inputs) 2025-08-14T21:43:44.0284850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0284926Z outputs = self.deberta( 2025-08-14T21:43:44.0285195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0285266Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0285609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0285703Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0285930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0286012Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0286277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0286376Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0286653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0286734Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0287026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:44.0287102Z context_layer = torch.bmm( 2025-08-14T21:43:44.0287106Z 2025-08-14T21:43:44.0287222Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0287426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0287526Z return mod(**inputs) 2025-08-14T21:43:44.0287830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0287900Z outputs = self.deberta( 2025-08-14T21:43:44.0288171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0288244Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0288512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0288621Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0288841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0288918Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0289193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0289285Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0289576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0289654Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0289941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:44.0290138Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:44.0290144Z 2025-08-14T21:43:44.0290246Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0290451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0290518Z return mod(**inputs) 2025-08-14T21:43:44.0290791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0290871Z outputs = self.deberta( 2025-08-14T21:43:44.0291145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0291224Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0291498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0291583Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0291810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0291889Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0292156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0292257Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0292532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:44.0292657Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:44.0292931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:44.0293015Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0293020Z 2025-08-14T21:43:44.0293130Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0293328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0293400Z return mod(**inputs) 2025-08-14T21:43:44.0293674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0293767Z outputs = self.deberta( 2025-08-14T21:43:44.0294046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0294120Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0294384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0294476Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0294696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0294798Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0295068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:44.0295186Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:44.0295466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:44.0295549Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0295569Z 2025-08-14T21:43:44.0295680Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0295877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0295958Z return mod(**inputs) 2025-08-14T21:43:44.0296235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0296304Z outputs = self.deberta( 2025-08-14T21:43:44.0296568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0296648Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0296915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0297005Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0297225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0297302Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0297580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:44.0297694Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:44.0297966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:44.0298082Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:44.0298295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:44.0298375Z return self.act(input) 2025-08-14T21:43:44.0298378Z 2025-08-14T21:43:44.0298477Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0298673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0298747Z return mod(**inputs) 2025-08-14T21:43:44.0299014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0299089Z outputs = self.deberta( 2025-08-14T21:43:44.0299358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0299430Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0299701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0299819Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0300029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0300112Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0300371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:44.0300506Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:44.0300766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:44.0300863Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0300867Z 2025-08-14T21:43:44.0300972Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0301163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0301235Z return mod(**inputs) 2025-08-14T21:43:44.0301496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0301576Z outputs = self.deberta( 2025-08-14T21:43:44.0301844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0301927Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0302186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0302274Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0302485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0302567Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0302824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0302913Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0303180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0303252Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0303517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:44.0303699Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:44.0303702Z 2025-08-14T21:43:44.0303801Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0303999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0304061Z return mod(**inputs) 2025-08-14T21:43:44.0304330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0304397Z outputs = self.deberta( 2025-08-14T21:43:44.0304654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0304732Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0304990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0305072Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0305289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0305363Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0305627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0305735Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0305993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0306076Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0306334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:44.0306512Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0306532Z 2025-08-14T21:43:44.0306633Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0306824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0306896Z return mod(**inputs) 2025-08-14T21:43:44.0307161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0307230Z outputs = self.deberta( 2025-08-14T21:43:44.0307511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0307583Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0307876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0307959Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0308173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0308256Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0308514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0308613Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0308872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0308944Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0309219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:44.0309393Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:44.0309683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:44.0309813Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:44.0309816Z 2025-08-14T21:43:44.0309909Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0310103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0310168Z return mod(**inputs) 2025-08-14T21:43:44.0310423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0310496Z outputs = self.deberta( 2025-08-14T21:43:44.0310747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0310822Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0311077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0311156Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0311368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0311442Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0311708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0311802Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0312054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0312132Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0312382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:44.0312597Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:44.0312608Z 2025-08-14T21:43:44.0312702Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0312885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0312956Z return mod(**inputs) 2025-08-14T21:43:44.0313230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0313297Z outputs = self.deberta( 2025-08-14T21:43:44.0313561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0313645Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0313906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0313985Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0314190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0314272Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0314524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0314610Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0314873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0314944Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0315204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:44.0315399Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:44.0315402Z 2025-08-14T21:43:44.0315497Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0315687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0315751Z return mod(**inputs) 2025-08-14T21:43:44.0316014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0316077Z outputs = self.deberta( 2025-08-14T21:43:44.0316329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0316402Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0316657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0316738Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0316957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0317030Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0317294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0317397Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0317658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0317740Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0317998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:44.0318188Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0318207Z 2025-08-14T21:43:44.0318307Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0318505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0318575Z return mod(**inputs) 2025-08-14T21:43:44.0318833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0318902Z outputs = self.deberta( 2025-08-14T21:43:44.0319173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0319243Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0319524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0319609Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0319821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0319903Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0320159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0320257Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0320515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0320590Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0320854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:44.0321033Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0321339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:44.0321466Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:44.0321470Z 2025-08-14T21:43:44.0321571Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0321775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0321840Z return mod(**inputs) 2025-08-14T21:43:44.0322109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0322186Z outputs = self.deberta( 2025-08-14T21:43:44.0322455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0322535Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0322803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0322886Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0323111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0323210Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0323486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0323577Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0323845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0323928Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0324196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:44.0324284Z context_layer = torch.bmm( 2025-08-14T21:43:44.0324296Z 2025-08-14T21:43:44.0324398Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0324596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0324672Z return mod(**inputs) 2025-08-14T21:43:44.0324949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0325033Z outputs = self.deberta( 2025-08-14T21:43:44.0325308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0325382Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0325772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0325873Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0326117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0326217Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0326514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0326616Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0326934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0327010Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0327291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:44.0327478Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:44.0327484Z 2025-08-14T21:43:44.0327595Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0327819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0327889Z return mod(**inputs) 2025-08-14T21:43:44.0328197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0328271Z outputs = self.deberta( 2025-08-14T21:43:44.0328566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0328651Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0328947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0329039Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0329287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0329371Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0329672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0329796Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0330094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:44.0330231Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:44.0330530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:44.0330632Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0330654Z 2025-08-14T21:43:44.0330766Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0330984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0331066Z return mod(**inputs) 2025-08-14T21:43:44.0331370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0331450Z outputs = self.deberta( 2025-08-14T21:43:44.0331784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0331863Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0332162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0332289Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0332530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0332625Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0332914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:44.0333049Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:44.0333337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:44.0333426Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0333430Z 2025-08-14T21:43:44.0333544Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0333761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0333827Z return mod(**inputs) 2025-08-14T21:43:44.0334101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0334168Z outputs = self.deberta( 2025-08-14T21:43:44.0334438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0334510Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0334774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0334865Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0335080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0335163Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0335425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:44.0335541Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:44.0335813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:44.0335924Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:44.0336133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:44.0336230Z return self.act(input) 2025-08-14T21:43:44.0336234Z 2025-08-14T21:43:44.0336334Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0336538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0336603Z return mod(**inputs) 2025-08-14T21:43:44.0336875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0336949Z outputs = self.deberta( 2025-08-14T21:43:44.0337232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0337311Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0337589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0337815Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0338064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0338183Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0338487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:44.0338668Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:44.0338956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:44.0339054Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0339058Z 2025-08-14T21:43:44.0339167Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0339376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0339467Z return mod(**inputs) 2025-08-14T21:43:44.0339741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0339817Z outputs = self.deberta( 2025-08-14T21:43:44.0340085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0340160Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0340439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0340525Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0340750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0340837Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0341108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0341210Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0341479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0341557Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0341835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:44.0342020Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:44.0342024Z 2025-08-14T21:43:44.0342131Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0342327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0342392Z return mod(**inputs) 2025-08-14T21:43:44.0342699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0342768Z outputs = self.deberta( 2025-08-14T21:43:44.0343037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0343117Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0343384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0343500Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0343717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0343795Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0344074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0344161Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0344437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0344510Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0344791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:44.0344971Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0344975Z 2025-08-14T21:43:44.0345072Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0345263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0345325Z return mod(**inputs) 2025-08-14T21:43:44.0345583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0345657Z outputs = self.deberta( 2025-08-14T21:43:44.0345911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0345980Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0346243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0346324Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0346542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0346615Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0346870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0346965Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0347218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0347291Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0347554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:44.0347732Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:44.0348032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:44.0348157Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:44.0348160Z 2025-08-14T21:43:44.0348256Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0348447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0348525Z return mod(**inputs) 2025-08-14T21:43:44.0348795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0348861Z outputs = self.deberta( 2025-08-14T21:43:44.0349121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0349198Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0349479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0349569Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0349787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0349862Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0350138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0350240Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0350498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0350595Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0350853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:44.0351064Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:44.0351067Z 2025-08-14T21:43:44.0351177Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0351361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0351432Z return mod(**inputs) 2025-08-14T21:43:44.0351687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0351758Z outputs = self.deberta( 2025-08-14T21:43:44.0352008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0352077Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0352337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0352419Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0352625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0352815Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0353094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0353233Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0353513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0353607Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0354102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:44.0354369Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:44.0354373Z 2025-08-14T21:43:44.0354490Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0354705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0354837Z return mod(**inputs) 2025-08-14T21:43:44.0355108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0355273Z outputs = self.deberta( 2025-08-14T21:43:44.0355554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0355644Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0355950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0356076Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0356292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0356444Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0356720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0356875Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0357175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0357272Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0357597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:44.0357812Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0357817Z 2025-08-14T21:43:44.0357960Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0358174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0358263Z return mod(**inputs) 2025-08-14T21:43:44.0358558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0358668Z outputs = self.deberta( 2025-08-14T21:43:44.0358985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0359073Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0359356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0359487Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0359721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0359837Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0360155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0360274Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0360584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0360682Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0360959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:44.0361219Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0361538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:44.0361734Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:44.0361738Z 2025-08-14T21:43:44.0361857Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0362091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0362213Z return mod(**inputs) 2025-08-14T21:43:44.0379530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0379699Z outputs = self.deberta( 2025-08-14T21:43:44.0380027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0380110Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0380494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0380589Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0380815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0380913Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0381183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0381331Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0381600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0381720Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0382002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:44.0382076Z context_layer = torch.bmm( 2025-08-14T21:43:44.0382083Z 2025-08-14T21:43:44.0382200Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0382402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0382471Z return mod(**inputs) 2025-08-14T21:43:44.0382748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0382821Z outputs = self.deberta( 2025-08-14T21:43:44.0383075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0383157Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0383413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0383510Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0383724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0383802Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0384067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0384158Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0384422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0384499Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0384752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:44.0384942Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:44.0384948Z 2025-08-14T21:43:44.0385050Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0385252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0385317Z return mod(**inputs) 2025-08-14T21:43:44.0385604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0385679Z outputs = self.deberta( 2025-08-14T21:43:44.0385935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0386009Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0386271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0386371Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0386589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0386665Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0386916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0387013Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0387288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:44.0387407Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:44.0387697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:44.0387783Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0387788Z 2025-08-14T21:43:44.0387896Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0388094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0388159Z return mod(**inputs) 2025-08-14T21:43:44.0388437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0388506Z outputs = self.deberta( 2025-08-14T21:43:44.0388781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0388850Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0389102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0389192Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0389397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0389476Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0389730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:44.0389841Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:44.0390088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:44.0390168Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0390172Z 2025-08-14T21:43:44.0390268Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0390456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0390525Z return mod(**inputs) 2025-08-14T21:43:44.0390780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0390851Z outputs = self.deberta( 2025-08-14T21:43:44.0391102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0391169Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0391446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0391526Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0391741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0391814Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0392068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:44.0392203Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:44.0392455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:44.0392562Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:44.0392774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:44.0392844Z return self.act(input) 2025-08-14T21:43:44.0392847Z 2025-08-14T21:43:44.0392951Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0393156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0393220Z return mod(**inputs) 2025-08-14T21:43:44.0393499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0393565Z outputs = self.deberta( 2025-08-14T21:43:44.0393825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0393901Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0394155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0394241Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0394449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0394522Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0394783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:44.0394910Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:44.0395169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:44.0395245Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0395249Z 2025-08-14T21:43:44.0395344Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0395537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0395602Z return mod(**inputs) 2025-08-14T21:43:44.0395862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0395926Z outputs = self.deberta( 2025-08-14T21:43:44.0396184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0396260Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0396512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0396591Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0396805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0396877Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0397136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0397241Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0397503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0397585Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0397844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:44.0398047Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:44.0398051Z 2025-08-14T21:43:44.0398146Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0398336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0398406Z return mod(**inputs) 2025-08-14T21:43:44.0398664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0398730Z outputs = self.deberta( 2025-08-14T21:43:44.0399014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0399086Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0399375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0399461Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0399682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0399770Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0400032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0400136Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0400402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0400479Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0400750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 237, in forward 2025-08-14T21:43:44.0400929Z key_layer = self.transpose_for_scores(self.key_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0400934Z 2025-08-14T21:43:44.0401037Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0401237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0401304Z return mod(**inputs) 2025-08-14T21:43:44.0401580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0401652Z outputs = self.deberta( 2025-08-14T21:43:44.0401919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0402002Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0402266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0402358Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0402579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0402659Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0402925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0403034Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0403292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0403375Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0403633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 236, in forward 2025-08-14T21:43:44.0403823Z query_layer = self.transpose_for_scores(self.query_proj(query_states), self.num_attention_heads) 2025-08-14T21:43:44.0404152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:44.0404290Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:44.0404294Z 2025-08-14T21:43:44.0404403Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0404606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0404680Z return mod(**inputs) 2025-08-14T21:43:44.0404985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0405055Z outputs = self.deberta( 2025-08-14T21:43:44.0405353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0405431Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0405933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0406030Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0406265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0406355Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0406640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0406736Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0407025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0407108Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0407399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:44.0407629Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:44.0407633Z 2025-08-14T21:43:44.0407742Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0407964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0408032Z return mod(**inputs) 2025-08-14T21:43:44.0408312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0408382Z outputs = self.deberta( 2025-08-14T21:43:44.0408647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0408729Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0408995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0409081Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0409312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0409389Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0409681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0409770Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0410039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0410123Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0410393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 248, in forward 2025-08-14T21:43:44.0410624Z attention_scores = torch.bmm(query_layer, key_layer.transpose(-1, -2) / scale.to(dtype=query_layer.dtype)) 2025-08-14T21:43:44.0410627Z 2025-08-14T21:43:44.0410730Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0410930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0411006Z return mod(**inputs) 2025-08-14T21:43:44.0411276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0411361Z outputs = self.deberta( 2025-08-14T21:43:44.0411640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0411711Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0412002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0412089Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0412307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0412392Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0412658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0412755Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0413031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0413103Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0413369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:44.0413553Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0413559Z 2025-08-14T21:43:44.0413662Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0413853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0413915Z return mod(**inputs) 2025-08-14T21:43:44.0414187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0414255Z outputs = self.deberta( 2025-08-14T21:43:44.0414517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0414595Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0414857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0414946Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0415161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0415237Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0415502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0415608Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0415875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0415958Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0416225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 238, in forward 2025-08-14T21:43:44.0416418Z value_layer = self.transpose_for_scores(self.value_proj(hidden_states), self.num_attention_heads) 2025-08-14T21:43:44.0416738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 194, in transpose_for_scores 2025-08-14T21:43:44.0416866Z return x.permute(0, 2, 1, 3).contiguous().view(-1, x.size(1), x.size(-1)) 2025-08-14T21:43:44.0416877Z 2025-08-14T21:43:44.0416976Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0417167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0417239Z return mod(**inputs) 2025-08-14T21:43:44.0417518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0417597Z outputs = self.deberta( 2025-08-14T21:43:44.0417878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0417947Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0418213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0418294Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0418505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0418588Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0418846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0418935Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0419204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0419279Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0419547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 268, in forward 2025-08-14T21:43:44.0419621Z context_layer = torch.bmm( 2025-08-14T21:43:44.0419624Z 2025-08-14T21:43:44.0419723Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0419921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0419987Z return mod(**inputs) 2025-08-14T21:43:44.0420257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0420325Z outputs = self.deberta( 2025-08-14T21:43:44.0420582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0420661Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0420920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0421003Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0421223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0421297Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0421561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0421666Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0421937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 371, in forward 2025-08-14T21:43:44.0422014Z self_output, att_matrix = self.self( 2025-08-14T21:43:44.0422266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 272, in forward 2025-08-14T21:43:44.0422450Z context_layer.view(-1, self.num_attention_heads, context_layer.size(-2), context_layer.size(-1)) 2025-08-14T21:43:44.0422470Z 2025-08-14T21:43:44.0422566Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0422751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0422822Z return mod(**inputs) 2025-08-14T21:43:44.0423080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0423146Z outputs = self.deberta( 2025-08-14T21:43:44.0423417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0423488Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0423771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0423856Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0424078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0424158Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0424411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 438, in forward 2025-08-14T21:43:44.0424504Z attention_output, att_matrix = self.attention( 2025-08-14T21:43:44.0424758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 381, in forward 2025-08-14T21:43:44.0424868Z attention_output = self.output(self_output, query_states) 2025-08-14T21:43:44.0425132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 52, in forward 2025-08-14T21:43:44.0425213Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0425219Z 2025-08-14T21:43:44.0425323Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0425513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0425576Z return mod(**inputs) 2025-08-14T21:43:44.0425846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0425914Z outputs = self.deberta( 2025-08-14T21:43:44.0426174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0426252Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0426511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0426599Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0426813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0426887Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0427153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:44.0427276Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:44.0427551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 400, in forward 2025-08-14T21:43:44.0427639Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0427643Z 2025-08-14T21:43:44.0427744Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0427937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0428010Z return mod(**inputs) 2025-08-14T21:43:44.0428273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0428372Z outputs = self.deberta( 2025-08-14T21:43:44.0428631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0428699Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0428966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0429054Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0429277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0429359Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0429624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 446, in forward 2025-08-14T21:43:44.0429744Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:43:44.0430000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 401, in forward 2025-08-14T21:43:44.0430107Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:43:44.0430315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:43:44.0430383Z return self.act(input) 2025-08-14T21:43:44.0430387Z 2025-08-14T21:43:44.0430489Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0430678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0430739Z return mod(**inputs) 2025-08-14T21:43:44.0431010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1231, in forward 2025-08-14T21:43:44.0431073Z outputs = self.deberta( 2025-08-14T21:43:44.0431332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 786, in forward 2025-08-14T21:43:44.0431407Z encoder_outputs = self.encoder( 2025-08-14T21:43:44.0431662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 659, in forward 2025-08-14T21:43:44.0431752Z output_states, attn_weights = layer_module( 2025-08-14T21:43:44.0431967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:43:44.0432044Z return super().__call__(*args, **kwargs) 2025-08-14T21:43:44.0432316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 447, in forward 2025-08-14T21:43:44.0432446Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:43:44.0432718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 415, in forward 2025-08-14T21:43:44.0432800Z hidden_states = self.dense(hidden_states) 2025-08-14T21:43:44.0432803Z 2025-08-14T21:43:44.0432901Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0433109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0433187Z return mod(**inputs) 2025-08-14T21:43:44.0433445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1244, in forward 2025-08-14T21:43:44.0433535Z logits = self.qa_outputs(sequence_output) 2025-08-14T21:43:44.0433538Z 2025-08-14T21:43:44.0433635Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0433833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0433897Z return mod(**inputs) 2025-08-14T21:43:44.0434182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1262, in forward 2025-08-14T21:43:44.0434292Z start_loss = loss_fct(start_logits, start_positions) 2025-08-14T21:43:44.0434296Z 2025-08-14T21:43:44.0434391Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:43:44.0434588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:43:44.0434652Z return mod(**inputs) 2025-08-14T21:43:44.0434935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1263, in forward 2025-08-14T21:43:44.0435033Z end_loss = loss_fct(end_logits, end_positions) 2025-08-14T21:43:44.0435037Z 2025-08-14T21:43:55.5561416Z Compilation time (from dynamo_timed): 24.188288642 2025-08-14T21:43:55.5563105Z pass 2025-08-14T21:43:55.5563473Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:43:55.5564245Z TIMING: _recursive_pre_grad_passes:0.01422 _recursive_joint_graph_passes:1.11833 _recursive_post_grad_passes:0.28953 async_compile.wait:0.47512 code_gen:10.1957 inductor_compile:12.88232 backend_compile:19.04901 gc:0.00052 entire_frame_compile:24.18829 total_wall_time:24.18829 2025-08-14T21:43:55.5565121Z STATS: call_* op count: 1087 | FakeTensorMode.__torch_dispatch__:30540 | FakeTensor.__torch_dispatch__:11359 | ProxyTorchDispatchMode.__torch_dispatch__:11524 2025-08-14T21:43:55.5565749Z Dynamo produced 1 graphs covering 1087 ops with 0 graph breaks (0 unique) 2025-08-14T21:44:00.8128919Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:44:00.8131547Z from pkg_resources import resource_filename 2025-08-14T21:44:01.4999673Z 2025-08-14T21:44:02.2248745Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:44:02.2249010Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:44:02.2258749Z cpu eval DistilBertForMaskedLM 2025-08-14T21:44:02.5613916Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:44:02.6168508Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:44:02.6708561Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:44:07.2919025Z cudagraph partition due to non gpu ops 2025-08-14T21:44:07.2919568Z cudagraph partition due to non gpu ops 2025-08-14T21:44:07.2920665Z cudagraph partition due to non gpu ops 2025-08-14T21:44:07.2921194Z cudagraph partition due to non gpu ops 2025-08-14T21:44:07.2921523Z cudagraph partition due to non gpu ops 2025-08-14T21:44:07.2921829Z cudagraph partition due to non gpu ops 2025-08-14T21:44:07.2922564Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.2923094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.2923514Z return mod(**inputs) 2025-08-14T21:44:07.2923991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.2924849Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.2925318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.2925950Z return self.transformer( 2025-08-14T21:44:07.2926419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.2926913Z layer_outputs = layer_module( 2025-08-14T21:44:07.2927328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.2927790Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.2928221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.2928641Z sa_output = self.attention( 2025-08-14T21:44:07.2929060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:44:07.2929594Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:44:07.2929784Z 2025-08-14T21:44:07.2929900Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.2930316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.2930653Z return mod(**inputs) 2025-08-14T21:44:07.2931048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.2931459Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.2931873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.2932285Z return self.transformer( 2025-08-14T21:44:07.2932680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.2933083Z layer_outputs = layer_module( 2025-08-14T21:44:07.2933440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.2933818Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.2934224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.2934637Z sa_output = self.attention( 2025-08-14T21:44:07.2935037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:44:07.2935471Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:07.2935639Z 2025-08-14T21:44:07.2935739Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.2936083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.2936390Z return mod(**inputs) 2025-08-14T21:44:07.2936759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.2937142Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.2937534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.2938246Z return self.transformer( 2025-08-14T21:44:07.2938624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.2939025Z layer_outputs = layer_module( 2025-08-14T21:44:07.2939363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.2939769Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.2940164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.2940564Z sa_output = self.attention( 2025-08-14T21:44:07.2940950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:44:07.2941401Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:07.2941576Z 2025-08-14T21:44:07.2941701Z cudagraph partition due to non gpu ops 2025-08-14T21:44:07.2941936Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.2942277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.2942585Z return mod(**inputs) 2025-08-14T21:44:07.2942954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.2943352Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.2943770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.2944157Z return self.transformer( 2025-08-14T21:44:07.2944554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.2944946Z layer_outputs = layer_module( 2025-08-14T21:44:07.2945266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.2945610Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.2946001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.2946386Z sa_output = self.attention( 2025-08-14T21:44:07.2946757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:44:07.2947199Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:44:07.2947374Z 2025-08-14T21:44:07.2947478Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.2947816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.2948120Z return mod(**inputs) 2025-08-14T21:44:07.2948487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.2948876Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.2949251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.2949633Z return self.transformer( 2025-08-14T21:44:07.2950004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.2950387Z layer_outputs = layer_module( 2025-08-14T21:44:07.2950704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.2951043Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.2951429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.2951808Z sa_output = self.attention( 2025-08-14T21:44:07.2952180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:44:07.2952573Z attn_output = self.out_lin(attn_output) 2025-08-14T21:44:07.2952702Z 2025-08-14T21:44:07.2952805Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.2953185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.2953508Z return mod(**inputs) 2025-08-14T21:44:07.2953893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.2954304Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.2954702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.2955111Z return self.transformer( 2025-08-14T21:44:07.2955555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.2955966Z layer_outputs = layer_module( 2025-08-14T21:44:07.2956305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.2956653Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.2957065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:07.2957529Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:07.2957978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:07.2958544Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:07.2959066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:07.2959462Z return forward_fn(*input_tensors) 2025-08-14T21:44:07.2959869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:44:07.2960277Z x = self.lin1(input) 2025-08-14T21:44:07.2960391Z 2025-08-14T21:44:07.2960494Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.2960850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.2961171Z return mod(**inputs) 2025-08-14T21:44:07.2961550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.2961962Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.2962356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.2962763Z return self.transformer( 2025-08-14T21:44:07.2963155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.2963561Z layer_outputs = layer_module( 2025-08-14T21:44:07.2963900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.2964258Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.2964667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:07.2965108Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:07.2965640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:07.2966190Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:07.2966712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:07.2967107Z return forward_fn(*input_tensors) 2025-08-14T21:44:07.2967523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:44:07.2967954Z x = self.activation(x) 2025-08-14T21:44:07.2968282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:07.2968615Z return self.act(input) 2025-08-14T21:44:07.2968729Z 2025-08-14T21:44:07.2968831Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.2969191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.2970163Z return mod(**inputs) 2025-08-14T21:44:07.2970539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.2970953Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.2971363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.2971761Z return self.transformer( 2025-08-14T21:44:07.2972147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.2972560Z layer_outputs = layer_module( 2025-08-14T21:44:07.2972899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.2973261Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.2973664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:07.2974097Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:07.2974526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:07.2975034Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:07.2975535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:07.2975917Z return forward_fn(*input_tensors) 2025-08-14T21:44:07.2976305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:44:07.2976692Z x = self.lin2(x) 2025-08-14T21:44:07.2976795Z 2025-08-14T21:44:07.2976895Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.2977243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.2977547Z return mod(**inputs) 2025-08-14T21:44:07.2977916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.2978315Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.2978708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.2979098Z return self.transformer( 2025-08-14T21:44:07.2979479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.2979875Z layer_outputs = layer_module( 2025-08-14T21:44:07.2980203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.2980556Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.2980952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.2981350Z sa_output = self.attention( 2025-08-14T21:44:07.2981723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:44:07.2982198Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:44:07.2982372Z 2025-08-14T21:44:07.2982479Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.2982824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.2983130Z return mod(**inputs) 2025-08-14T21:44:07.2983510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.2983929Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.2984315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.2984712Z return self.transformer( 2025-08-14T21:44:07.2985090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.2985486Z layer_outputs = layer_module( 2025-08-14T21:44:07.2985816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.2986181Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.2986626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.2987042Z sa_output = self.attention( 2025-08-14T21:44:07.2987442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:44:07.2987898Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:07.2988071Z 2025-08-14T21:44:07.2988180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.2988539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.2988854Z return mod(**inputs) 2025-08-14T21:44:07.2989232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.2989632Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.2990032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.2990434Z return self.transformer( 2025-08-14T21:44:07.2990888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.2991292Z layer_outputs = layer_module( 2025-08-14T21:44:07.2991637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.2991994Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.2992405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.2992808Z sa_output = self.attention( 2025-08-14T21:44:07.2993206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:44:07.2993660Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:07.2993836Z 2025-08-14T21:44:07.2993924Z cudagraph partition due to non gpu ops 2025-08-14T21:44:07.2994158Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.2994511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.2994830Z return mod(**inputs) 2025-08-14T21:44:07.2995205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.2995613Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.2996037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.2996444Z return self.transformer( 2025-08-14T21:44:07.2996829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.2997229Z layer_outputs = layer_module( 2025-08-14T21:44:07.2997574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.2997946Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.2998360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.2998749Z sa_output = self.attention( 2025-08-14T21:44:07.2999111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:44:07.2999532Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:44:07.2999708Z 2025-08-14T21:44:07.2999802Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3000146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3000441Z return mod(**inputs) 2025-08-14T21:44:07.3000814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3001207Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3001583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3001959Z return self.transformer( 2025-08-14T21:44:07.3002327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3002709Z layer_outputs = layer_module( 2025-08-14T21:44:07.3003031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3003361Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3003749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.3004137Z sa_output = self.attention( 2025-08-14T21:44:07.3004503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:44:07.3004897Z attn_output = self.out_lin(attn_output) 2025-08-14T21:44:07.3005031Z 2025-08-14T21:44:07.3005130Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3005548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3005860Z return mod(**inputs) 2025-08-14T21:44:07.3006236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3006639Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3007035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3007434Z return self.transformer( 2025-08-14T21:44:07.3007808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3008204Z layer_outputs = layer_module( 2025-08-14T21:44:07.3008539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3008875Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3009269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:07.3009719Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:07.3010135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:07.3010643Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:07.3011134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:07.3011533Z return forward_fn(*input_tensors) 2025-08-14T21:44:07.3011923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:44:07.3012298Z x = self.lin1(input) 2025-08-14T21:44:07.3012395Z 2025-08-14T21:44:07.3012495Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3012822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3013126Z return mod(**inputs) 2025-08-14T21:44:07.3013507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3013903Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3014294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3014686Z return self.transformer( 2025-08-14T21:44:07.3015063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3015454Z layer_outputs = layer_module( 2025-08-14T21:44:07.3015775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3016117Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3016511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:07.3016926Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:07.3017347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:07.3017967Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:07.3018451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:07.3018819Z return forward_fn(*input_tensors) 2025-08-14T21:44:07.3019214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:44:07.3019603Z x = self.activation(x) 2025-08-14T21:44:07.3019914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:07.3020233Z return self.act(input) 2025-08-14T21:44:07.3020343Z 2025-08-14T21:44:07.3020442Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3020784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3021085Z return mod(**inputs) 2025-08-14T21:44:07.3021450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3021846Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3022232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3022614Z return self.transformer( 2025-08-14T21:44:07.3022990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3023401Z layer_outputs = layer_module( 2025-08-14T21:44:07.3023737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3024068Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3024465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:07.3024885Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:07.3025321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:07.3025828Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:07.3026307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:07.3026681Z return forward_fn(*input_tensors) 2025-08-14T21:44:07.3027083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:44:07.3027478Z x = self.lin2(x) 2025-08-14T21:44:07.3027579Z 2025-08-14T21:44:07.3027680Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3028026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3028323Z return mod(**inputs) 2025-08-14T21:44:07.3028687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3029080Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3029461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3029862Z return self.transformer( 2025-08-14T21:44:07.3030245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3030644Z layer_outputs = layer_module( 2025-08-14T21:44:07.3030972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3031322Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3031724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.3032124Z sa_output = self.attention( 2025-08-14T21:44:07.3032517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:44:07.3032968Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:44:07.3033140Z 2025-08-14T21:44:07.3033245Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3033587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3033906Z return mod(**inputs) 2025-08-14T21:44:07.3034268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3034664Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3035039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3035429Z return self.transformer( 2025-08-14T21:44:07.3035799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3036180Z layer_outputs = layer_module( 2025-08-14T21:44:07.3036506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3036858Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3037239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.3037750Z sa_output = self.attention( 2025-08-14T21:44:07.3038131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:44:07.3038554Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:07.3038765Z 2025-08-14T21:44:07.3038869Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3039194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3039496Z return mod(**inputs) 2025-08-14T21:44:07.3039860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3040249Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3040678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3041070Z return self.transformer( 2025-08-14T21:44:07.3041475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3041845Z layer_outputs = layer_module( 2025-08-14T21:44:07.3042167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3042497Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3042870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.3043258Z sa_output = self.attention( 2025-08-14T21:44:07.3043639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:44:07.3044079Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:07.3044251Z 2025-08-14T21:44:07.3044335Z cudagraph partition due to non gpu ops 2025-08-14T21:44:07.3044575Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3044929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3045256Z return mod(**inputs) 2025-08-14T21:44:07.3045695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3046118Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3046553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3046978Z return self.transformer( 2025-08-14T21:44:07.3047373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3047786Z layer_outputs = layer_module( 2025-08-14T21:44:07.3048127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3048483Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3048893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.3049303Z sa_output = self.attention( 2025-08-14T21:44:07.3049701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:44:07.3050161Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:44:07.3050386Z 2025-08-14T21:44:07.3050489Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3050849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3051168Z return mod(**inputs) 2025-08-14T21:44:07.3051554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3051970Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3052375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3052792Z return self.transformer( 2025-08-14T21:44:07.3053183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3053594Z layer_outputs = layer_module( 2025-08-14T21:44:07.3053932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3054290Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3054727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.3055136Z sa_output = self.attention( 2025-08-14T21:44:07.3055536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:44:07.3055954Z attn_output = self.out_lin(attn_output) 2025-08-14T21:44:07.3056091Z 2025-08-14T21:44:07.3056202Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3056553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3056866Z return mod(**inputs) 2025-08-14T21:44:07.3057230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3057620Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3057994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3058379Z return self.transformer( 2025-08-14T21:44:07.3058753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3059137Z layer_outputs = layer_module( 2025-08-14T21:44:07.3059456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3059797Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3060199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:07.3060639Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:07.3061050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:07.3061553Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:07.3062036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:07.3062402Z return forward_fn(*input_tensors) 2025-08-14T21:44:07.3062789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:44:07.3063172Z x = self.lin1(input) 2025-08-14T21:44:07.3063270Z 2025-08-14T21:44:07.3063373Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3063706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3064037Z return mod(**inputs) 2025-08-14T21:44:07.3064398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3064788Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3065164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3065547Z return self.transformer( 2025-08-14T21:44:07.3065917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3066320Z layer_outputs = layer_module( 2025-08-14T21:44:07.3066648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3066986Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3067377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:07.3067793Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:07.3068227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:07.3068733Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:07.3069242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:07.3069629Z return forward_fn(*input_tensors) 2025-08-14T21:44:07.3070043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:44:07.3070453Z x = self.activation(x) 2025-08-14T21:44:07.3070771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:07.3071130Z return self.act(input) 2025-08-14T21:44:07.3071252Z 2025-08-14T21:44:07.3071358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3071732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3072065Z return mod(**inputs) 2025-08-14T21:44:07.3072475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3072914Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3073360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3073788Z return self.transformer( 2025-08-14T21:44:07.3074279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3074714Z layer_outputs = layer_module( 2025-08-14T21:44:07.3075082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3075463Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3075972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:07.3076445Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:07.3076905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:07.3077473Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:07.3078016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:07.3078436Z return forward_fn(*input_tensors) 2025-08-14T21:44:07.3078900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:44:07.3079368Z x = self.lin2(x) 2025-08-14T21:44:07.3079475Z 2025-08-14T21:44:07.3079591Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3079963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3080315Z return mod(**inputs) 2025-08-14T21:44:07.3080721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3081177Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3081611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3082047Z return self.transformer( 2025-08-14T21:44:07.3082467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3082902Z layer_outputs = layer_module( 2025-08-14T21:44:07.3083299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3083702Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3084158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.3084591Z sa_output = self.attention( 2025-08-14T21:44:07.3085020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:44:07.3085598Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:44:07.3085806Z 2025-08-14T21:44:07.3085926Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3086319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3086678Z return mod(**inputs) 2025-08-14T21:44:07.3087103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3087533Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3087927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3088325Z return self.transformer( 2025-08-14T21:44:07.3088704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3089091Z layer_outputs = layer_module( 2025-08-14T21:44:07.3089421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3089765Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3090155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.3090550Z sa_output = self.attention( 2025-08-14T21:44:07.3090950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:44:07.3091440Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:07.3091622Z 2025-08-14T21:44:07.3091733Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3092116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3092458Z return mod(**inputs) 2025-08-14T21:44:07.3092871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3093261Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3093672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3094051Z return self.transformer( 2025-08-14T21:44:07.3094415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3094800Z layer_outputs = layer_module( 2025-08-14T21:44:07.3095128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3095490Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3095878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.3096266Z sa_output = self.attention( 2025-08-14T21:44:07.3096645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:44:07.3097080Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:07.3097254Z 2025-08-14T21:44:07.3097348Z cudagraph partition due to non gpu ops 2025-08-14T21:44:07.3097583Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3097919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3098272Z return mod(**inputs) 2025-08-14T21:44:07.3098648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3099028Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3099403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3099781Z return self.transformer( 2025-08-14T21:44:07.3100144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3100523Z layer_outputs = layer_module( 2025-08-14T21:44:07.3100836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3101167Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3101551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.3101929Z sa_output = self.attention( 2025-08-14T21:44:07.3102286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:44:07.3102719Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:44:07.3102892Z 2025-08-14T21:44:07.3102988Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3103317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3103615Z return mod(**inputs) 2025-08-14T21:44:07.3103981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3104375Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3104748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3105146Z return self.transformer( 2025-08-14T21:44:07.3105510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3105884Z layer_outputs = layer_module( 2025-08-14T21:44:07.3106195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3106523Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3106924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.3107309Z sa_output = self.attention( 2025-08-14T21:44:07.3107670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:44:07.3108060Z attn_output = self.out_lin(attn_output) 2025-08-14T21:44:07.3108188Z 2025-08-14T21:44:07.3108288Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3108635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3108933Z return mod(**inputs) 2025-08-14T21:44:07.3109290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3109680Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3110053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3110438Z return self.transformer( 2025-08-14T21:44:07.3110825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3111211Z layer_outputs = layer_module( 2025-08-14T21:44:07.3111546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3111882Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3112288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:07.3112709Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:07.3113133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:07.3113642Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:07.3114129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:07.3114510Z return forward_fn(*input_tensors) 2025-08-14T21:44:07.3114898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:44:07.3115280Z x = self.lin1(input) 2025-08-14T21:44:07.3115379Z 2025-08-14T21:44:07.3115485Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3115815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3116116Z return mod(**inputs) 2025-08-14T21:44:07.3116477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3116856Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3117234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3117613Z return self.transformer( 2025-08-14T21:44:07.3117980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3118356Z layer_outputs = layer_module( 2025-08-14T21:44:07.3118682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3119020Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3119404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:07.3119822Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:07.3120249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:07.3120749Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:07.3121221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:07.3121602Z return forward_fn(*input_tensors) 2025-08-14T21:44:07.3122002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:44:07.3122413Z x = self.activation(x) 2025-08-14T21:44:07.3122715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:07.3123031Z return self.act(input) 2025-08-14T21:44:07.3123134Z 2025-08-14T21:44:07.3123240Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3123568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3123875Z return mod(**inputs) 2025-08-14T21:44:07.3124262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3124661Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3125058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3125522Z return self.transformer( 2025-08-14T21:44:07.3125929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3126367Z layer_outputs = layer_module( 2025-08-14T21:44:07.3126738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3127139Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3127586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:07.3128054Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:07.3128537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:07.3129102Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:07.3129617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:07.3130007Z return forward_fn(*input_tensors) 2025-08-14T21:44:07.3130416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:44:07.3130820Z x = self.lin2(x) 2025-08-14T21:44:07.3130920Z 2025-08-14T21:44:07.3131033Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3131384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3131705Z return mod(**inputs) 2025-08-14T21:44:07.3132093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3132500Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3132912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3133342Z return self.transformer( 2025-08-14T21:44:07.3133761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3134209Z layer_outputs = layer_module( 2025-08-14T21:44:07.3134574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3134953Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3135385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.3135828Z sa_output = self.attention( 2025-08-14T21:44:07.3136266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:44:07.3136841Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:44:07.3137026Z 2025-08-14T21:44:07.3137133Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3137508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3138021Z return mod(**inputs) 2025-08-14T21:44:07.3138437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3138913Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3139334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3139737Z return self.transformer( 2025-08-14T21:44:07.3140208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3140645Z layer_outputs = layer_module( 2025-08-14T21:44:07.3141009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3141387Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3141817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.3142248Z sa_output = self.attention( 2025-08-14T21:44:07.3142671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:44:07.3143149Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:07.3143333Z 2025-08-14T21:44:07.3143442Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3143813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3144150Z return mod(**inputs) 2025-08-14T21:44:07.3144560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3145000Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3145423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3145835Z return self.transformer( 2025-08-14T21:44:07.3146200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3146585Z layer_outputs = layer_module( 2025-08-14T21:44:07.3146910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3147256Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3147640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.3148026Z sa_output = self.attention( 2025-08-14T21:44:07.3148401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:44:07.3148824Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:07.3149030Z 2025-08-14T21:44:07.3149110Z cudagraph partition due to non gpu ops 2025-08-14T21:44:07.3149338Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3149680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3149981Z return mod(**inputs) 2025-08-14T21:44:07.3150350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3150745Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3151147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3151532Z return self.transformer( 2025-08-14T21:44:07.3151902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3152293Z layer_outputs = layer_module( 2025-08-14T21:44:07.3152613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3152969Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3153367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.3153754Z sa_output = self.attention( 2025-08-14T21:44:07.3154141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:44:07.3154585Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:44:07.3154759Z 2025-08-14T21:44:07.3154863Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3155192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3155499Z return mod(**inputs) 2025-08-14T21:44:07.3155867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3156255Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3156630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3157016Z return self.transformer( 2025-08-14T21:44:07.3157390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3157778Z layer_outputs = layer_module( 2025-08-14T21:44:07.3158105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3158450Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3158844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.3159221Z sa_output = self.attention( 2025-08-14T21:44:07.3159601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:44:07.3159983Z attn_output = self.out_lin(attn_output) 2025-08-14T21:44:07.3160105Z 2025-08-14T21:44:07.3160205Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3160523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3160824Z return mod(**inputs) 2025-08-14T21:44:07.3161178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3161553Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3161924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3162333Z return self.transformer( 2025-08-14T21:44:07.3162697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3163071Z layer_outputs = layer_module( 2025-08-14T21:44:07.3163397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3163742Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3164134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:07.3164587Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:07.3165014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:07.3165607Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:07.3166108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:07.3166516Z return forward_fn(*input_tensors) 2025-08-14T21:44:07.3166926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:44:07.3167355Z x = self.lin1(input) 2025-08-14T21:44:07.3167458Z 2025-08-14T21:44:07.3167559Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3167910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3168220Z return mod(**inputs) 2025-08-14T21:44:07.3168584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3168971Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3169368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3169748Z return self.transformer( 2025-08-14T21:44:07.3170108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3170485Z layer_outputs = layer_module( 2025-08-14T21:44:07.3170810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3171141Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3171517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:07.3171929Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:07.3172347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:07.3172853Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:07.3173333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:07.3173711Z return forward_fn(*input_tensors) 2025-08-14T21:44:07.3174109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:44:07.3174502Z x = self.activation(x) 2025-08-14T21:44:07.3174822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:07.3175152Z return self.act(input) 2025-08-14T21:44:07.3175253Z 2025-08-14T21:44:07.3175358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3175693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3176022Z return mod(**inputs) 2025-08-14T21:44:07.3176396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3176787Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3177181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3177579Z return self.transformer( 2025-08-14T21:44:07.3177965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3178378Z layer_outputs = layer_module( 2025-08-14T21:44:07.3178715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3179060Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3179465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:07.3179886Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:07.3180330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:07.3180869Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:07.3181364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:07.3181740Z return forward_fn(*input_tensors) 2025-08-14T21:44:07.3182148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:44:07.3182554Z x = self.lin2(x) 2025-08-14T21:44:07.3182652Z 2025-08-14T21:44:07.3182756Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3183111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3183431Z return mod(**inputs) 2025-08-14T21:44:07.3183808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3184211Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3184616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3185025Z return self.transformer( 2025-08-14T21:44:07.3185413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3185820Z layer_outputs = layer_module( 2025-08-14T21:44:07.3186161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3186518Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3186923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.3187327Z sa_output = self.attention( 2025-08-14T21:44:07.3187723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:44:07.3188254Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:44:07.3188433Z 2025-08-14T21:44:07.3188536Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3188888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3189206Z return mod(**inputs) 2025-08-14T21:44:07.3189581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3190007Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3190400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3190784Z return self.transformer( 2025-08-14T21:44:07.3191147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3191546Z layer_outputs = layer_module( 2025-08-14T21:44:07.3191879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3192243Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3192655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.3193058Z sa_output = self.attention( 2025-08-14T21:44:07.3193447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:44:07.3193908Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:07.3194084Z 2025-08-14T21:44:07.3194184Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3194551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3194868Z return mod(**inputs) 2025-08-14T21:44:07.3195244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3195634Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3196021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3196398Z return self.transformer( 2025-08-14T21:44:07.3196775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3197168Z layer_outputs = layer_module( 2025-08-14T21:44:07.3197506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3197847Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3198251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.3198647Z sa_output = self.attention( 2025-08-14T21:44:07.3199030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:44:07.3199468Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:07.3199649Z 2025-08-14T21:44:07.3199727Z cudagraph partition due to non gpu ops 2025-08-14T21:44:07.3199962Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3200301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3200615Z return mod(**inputs) 2025-08-14T21:44:07.3200993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3201393Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3201784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3202200Z return self.transformer( 2025-08-14T21:44:07.3202593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3202999Z layer_outputs = layer_module( 2025-08-14T21:44:07.3203351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3203776Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3204215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.3204656Z sa_output = self.attention( 2025-08-14T21:44:07.3205072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:44:07.3205646Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:44:07.3205888Z 2025-08-14T21:44:07.3206019Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3206407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3206767Z return mod(**inputs) 2025-08-14T21:44:07.3207203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3207620Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3208025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3208415Z return self.transformer( 2025-08-14T21:44:07.3208882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3209323Z layer_outputs = layer_module( 2025-08-14T21:44:07.3209699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3210087Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3210551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:07.3210987Z sa_output = self.attention( 2025-08-14T21:44:07.3211413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:44:07.3211864Z attn_output = self.out_lin(attn_output) 2025-08-14T21:44:07.3212012Z 2025-08-14T21:44:07.3212123Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3212508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3212860Z return mod(**inputs) 2025-08-14T21:44:07.3213269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3213710Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3214153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3214591Z return self.transformer( 2025-08-14T21:44:07.3214990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3215383Z layer_outputs = layer_module( 2025-08-14T21:44:07.3215716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3216053Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3216435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:07.3216853Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:07.3217268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:07.3217768Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:07.3218247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:07.3218653Z return forward_fn(*input_tensors) 2025-08-14T21:44:07.3219048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:44:07.3219444Z x = self.lin1(input) 2025-08-14T21:44:07.3219544Z 2025-08-14T21:44:07.3219645Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3219994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3220327Z return mod(**inputs) 2025-08-14T21:44:07.3220687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3221088Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3221483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3221887Z return self.transformer( 2025-08-14T21:44:07.3222276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3222680Z layer_outputs = layer_module( 2025-08-14T21:44:07.3223019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3223382Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3223784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:07.3224217Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:07.3224643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:07.3225154Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:07.3225640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:07.3226008Z return forward_fn(*input_tensors) 2025-08-14T21:44:07.3226404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:44:07.3226797Z x = self.activation(x) 2025-08-14T21:44:07.3227112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:07.3227443Z return self.act(input) 2025-08-14T21:44:07.3227547Z 2025-08-14T21:44:07.3227646Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3227994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3228305Z return mod(**inputs) 2025-08-14T21:44:07.3228676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 826, in forward 2025-08-14T21:44:07.3229068Z dlbrt_output = self.distilbert( 2025-08-14T21:44:07.3229465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:07.3229850Z return self.transformer( 2025-08-14T21:44:07.3230221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:07.3230599Z layer_outputs = layer_module( 2025-08-14T21:44:07.3230926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:07.3231264Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:07.3231645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:07.3232099Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:07.3232529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:07.3233043Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:07.3233533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:07.3233914Z return forward_fn(*input_tensors) 2025-08-14T21:44:07.3234339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:44:07.3234733Z x = self.lin2(x) 2025-08-14T21:44:07.3234829Z 2025-08-14T21:44:07.3234928Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3235276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3235594Z return mod(**inputs) 2025-08-14T21:44:07.3235981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 836, in forward 2025-08-14T21:44:07.3236476Z prediction_logits = self.vocab_transform(hidden_states) # (bs, seq_length, dim) 2025-08-14T21:44:07.3236699Z 2025-08-14T21:44:07.3236817Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3237173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3237486Z return mod(**inputs) 2025-08-14T21:44:07.3237999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 839, in forward 2025-08-14T21:44:07.3238538Z prediction_logits = self.vocab_projector(prediction_logits) # (bs, seq_length, vocab_size) 2025-08-14T21:44:07.3238780Z 2025-08-14T21:44:07.3238893Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:07.3239243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:07.3239589Z return mod(**inputs) 2025-08-14T21:44:07.3240008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 843, in forward 2025-08-14T21:44:07.3240581Z mlm_loss = self.mlm_loss_fct(prediction_logits.view(-1, prediction_logits.size(-1)), labels.view(-1)) 2025-08-14T21:44:07.3240853Z 2025-08-14T21:44:13.9945992Z Compilation time (from dynamo_timed): 10.261062865 2025-08-14T21:44:13.9951844Z pass 2025-08-14T21:44:13.9952202Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:44:13.9953012Z TIMING: _recursive_pre_grad_passes:0.00478 _recursive_joint_graph_passes:0.23964 _recursive_post_grad_passes:0.05249 async_compile.wait:0.66511 code_gen:6.38005 inductor_compile:7.2889 backend_compile:8.95416 gc:0.00181 entire_frame_compile:10.26106 total_wall_time:10.26106 2025-08-14T21:44:13.9954061Z STATS: call_* op count: 153 | FakeTensorMode.__torch_dispatch__:6660 | FakeTensor.__torch_dispatch__:2532 | ProxyTorchDispatchMode.__torch_dispatch__:2359 2025-08-14T21:44:13.9954553Z Dynamo produced 1 graphs covering 153 ops with 0 graph breaks (0 unique) 2025-08-14T21:44:18.7479417Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:44:18.7481039Z from pkg_resources import resource_filename 2025-08-14T21:44:19.3105646Z 2025-08-14T21:44:19.8315312Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:44:19.8316133Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:44:19.8320131Z cpu eval DistilBertForQuestionAnswering 2025-08-14T21:44:20.1100292Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:44:20.1537588Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:44:20.1982499Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:44:24.7698978Z cudagraph partition due to non gpu ops 2025-08-14T21:44:24.7703435Z cudagraph partition due to non gpu ops 2025-08-14T21:44:24.7708278Z cudagraph partition due to non gpu ops 2025-08-14T21:44:24.7713697Z cudagraph partition due to non gpu ops 2025-08-14T21:44:24.7718699Z cudagraph partition due to non gpu ops 2025-08-14T21:44:24.7718958Z cudagraph partition due to non gpu ops 2025-08-14T21:44:24.7719685Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7720183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7720572Z return mod(**inputs) 2025-08-14T21:44:24.7721001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7721636Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7722082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7722573Z return self.transformer( 2025-08-14T21:44:24.7723025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7723486Z layer_outputs = layer_module( 2025-08-14T21:44:24.7723876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7724289Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7724711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7725120Z sa_output = self.attention( 2025-08-14T21:44:24.7725687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:44:24.7726202Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:44:24.7726408Z 2025-08-14T21:44:24.7726528Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7726915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7727270Z return mod(**inputs) 2025-08-14T21:44:24.7727694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7728143Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7728602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7729050Z return self.transformer( 2025-08-14T21:44:24.7729627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7730071Z layer_outputs = layer_module( 2025-08-14T21:44:24.7730445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7733415Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7733821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7734218Z sa_output = self.attention( 2025-08-14T21:44:24.7734589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:44:24.7735040Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:24.7735213Z 2025-08-14T21:44:24.7735315Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7735658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7735959Z return mod(**inputs) 2025-08-14T21:44:24.7736328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7736765Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7737189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7737581Z return self.transformer( 2025-08-14T21:44:24.7738304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7738698Z layer_outputs = layer_module( 2025-08-14T21:44:24.7739035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7739440Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7739838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7740224Z sa_output = self.attention( 2025-08-14T21:44:24.7740636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:44:24.7741079Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:24.7741249Z 2025-08-14T21:44:24.7741335Z cudagraph partition due to non gpu ops 2025-08-14T21:44:24.7741560Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7741904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7742214Z return mod(**inputs) 2025-08-14T21:44:24.7742580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7742976Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7743372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7743764Z return self.transformer( 2025-08-14T21:44:24.7744132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7744523Z layer_outputs = layer_module( 2025-08-14T21:44:24.7744848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7745188Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7745571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7745947Z sa_output = self.attention( 2025-08-14T21:44:24.7746318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:44:24.7746746Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:44:24.7746922Z 2025-08-14T21:44:24.7747020Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7747428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7747729Z return mod(**inputs) 2025-08-14T21:44:24.7748091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7748487Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7748879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7749253Z return self.transformer( 2025-08-14T21:44:24.7749625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7750011Z layer_outputs = layer_module( 2025-08-14T21:44:24.7750340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7750678Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7751091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7751467Z sa_output = self.attention( 2025-08-14T21:44:24.7751824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:44:24.7752212Z attn_output = self.out_lin(attn_output) 2025-08-14T21:44:24.7752345Z 2025-08-14T21:44:24.7752442Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7752794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7753090Z return mod(**inputs) 2025-08-14T21:44:24.7753474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7753866Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7754253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7754626Z return self.transformer( 2025-08-14T21:44:24.7754992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7755374Z layer_outputs = layer_module( 2025-08-14T21:44:24.7755690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7756021Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7756408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:24.7756825Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:24.7757231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:24.7757731Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:24.7758216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:24.7758586Z return forward_fn(*input_tensors) 2025-08-14T21:44:24.7758964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:44:24.7759342Z x = self.lin1(input) 2025-08-14T21:44:24.7759437Z 2025-08-14T21:44:24.7759542Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7759863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7760167Z return mod(**inputs) 2025-08-14T21:44:24.7760524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7760953Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7761328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7761706Z return self.transformer( 2025-08-14T21:44:24.7762067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7762445Z layer_outputs = layer_module( 2025-08-14T21:44:24.7762758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7763089Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7763479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:24.7763896Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:24.7764382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:24.7764872Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:24.7765351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:24.7765909Z return forward_fn(*input_tensors) 2025-08-14T21:44:24.7766376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:44:24.7766799Z x = self.activation(x) 2025-08-14T21:44:24.7767145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:24.7767486Z return self.act(input) 2025-08-14T21:44:24.7767600Z 2025-08-14T21:44:24.7767702Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7768042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7768342Z return mod(**inputs) 2025-08-14T21:44:24.7768737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7769160Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7769578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7769984Z return self.transformer( 2025-08-14T21:44:24.7770383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7770800Z layer_outputs = layer_module( 2025-08-14T21:44:24.7771155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7771519Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7771938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:24.7772388Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:24.7772828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:24.7773369Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:24.7773886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:24.7774289Z return forward_fn(*input_tensors) 2025-08-14T21:44:24.7774700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:44:24.7775157Z x = self.lin2(x) 2025-08-14T21:44:24.7775255Z 2025-08-14T21:44:24.7775366Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7775722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7776040Z return mod(**inputs) 2025-08-14T21:44:24.7776425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7776823Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7777198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7777572Z return self.transformer( 2025-08-14T21:44:24.7777939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7778344Z layer_outputs = layer_module( 2025-08-14T21:44:24.7778658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7778993Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7779372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7779740Z sa_output = self.attention( 2025-08-14T21:44:24.7780147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:44:24.7780580Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:44:24.7780743Z 2025-08-14T21:44:24.7780845Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7781186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7781481Z return mod(**inputs) 2025-08-14T21:44:24.7781840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7782224Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7782597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7782972Z return self.transformer( 2025-08-14T21:44:24.7783331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7783710Z layer_outputs = layer_module( 2025-08-14T21:44:24.7784020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7784354Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7784733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7785103Z sa_output = self.attention( 2025-08-14T21:44:24.7785467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:44:24.7785882Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:24.7786038Z 2025-08-14T21:44:24.7786140Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7786458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7786756Z return mod(**inputs) 2025-08-14T21:44:24.7787114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7787504Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7787885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7788279Z return self.transformer( 2025-08-14T21:44:24.7788638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7789012Z layer_outputs = layer_module( 2025-08-14T21:44:24.7789331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7789662Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7790046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7790418Z sa_output = self.attention( 2025-08-14T21:44:24.7790785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:44:24.7791217Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:24.7791453Z 2025-08-14T21:44:24.7791542Z cudagraph partition due to non gpu ops 2025-08-14T21:44:24.7791759Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7792089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7792395Z return mod(**inputs) 2025-08-14T21:44:24.7792750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7793145Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7793548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7793936Z return self.transformer( 2025-08-14T21:44:24.7794312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7794697Z layer_outputs = layer_module( 2025-08-14T21:44:24.7795020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7795353Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7795739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7796114Z sa_output = self.attention( 2025-08-14T21:44:24.7796483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:44:24.7796913Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:44:24.7797088Z 2025-08-14T21:44:24.7797181Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7797511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7797813Z return mod(**inputs) 2025-08-14T21:44:24.7798160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7798542Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7798921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7799289Z return self.transformer( 2025-08-14T21:44:24.7799655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7800030Z layer_outputs = layer_module( 2025-08-14T21:44:24.7800345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7800671Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7801071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7801452Z sa_output = self.attention( 2025-08-14T21:44:24.7801808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:44:24.7802194Z attn_output = self.out_lin(attn_output) 2025-08-14T21:44:24.7802326Z 2025-08-14T21:44:24.7802419Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7802743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7803035Z return mod(**inputs) 2025-08-14T21:44:24.7803390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7803775Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7804156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7804549Z return self.transformer( 2025-08-14T21:44:24.7804923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7805311Z layer_outputs = layer_module( 2025-08-14T21:44:24.7805759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7806121Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7806548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:24.7806995Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:24.7807432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:24.7807943Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:24.7808432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:24.7808806Z return forward_fn(*input_tensors) 2025-08-14T21:44:24.7809193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:44:24.7809575Z x = self.lin1(input) 2025-08-14T21:44:24.7809673Z 2025-08-14T21:44:24.7809779Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7810113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7810420Z return mod(**inputs) 2025-08-14T21:44:24.7810786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7811185Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7811568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7811956Z return self.transformer( 2025-08-14T21:44:24.7812324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7812712Z layer_outputs = layer_module( 2025-08-14T21:44:24.7813030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7813368Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7813756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:24.7814168Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:24.7814624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:24.7815125Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:24.7815607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:24.7816014Z return forward_fn(*input_tensors) 2025-08-14T21:44:24.7816486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:44:24.7816897Z x = self.activation(x) 2025-08-14T21:44:24.7817215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:24.7817534Z return self.act(input) 2025-08-14T21:44:24.7817647Z 2025-08-14T21:44:24.7817746Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7818107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7818406Z return mod(**inputs) 2025-08-14T21:44:24.7818815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7819221Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7819619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7820003Z return self.transformer( 2025-08-14T21:44:24.7820397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7820790Z layer_outputs = layer_module( 2025-08-14T21:44:24.7821134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7821474Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7821864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:24.7822283Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:24.7822688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:24.7823180Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:24.7823655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:24.7824020Z return forward_fn(*input_tensors) 2025-08-14T21:44:24.7824393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:44:24.7824772Z x = self.lin2(x) 2025-08-14T21:44:24.7824862Z 2025-08-14T21:44:24.7824963Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7825298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7825593Z return mod(**inputs) 2025-08-14T21:44:24.7825950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7826342Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7826722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7827101Z return self.transformer( 2025-08-14T21:44:24.7827471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7827859Z layer_outputs = layer_module( 2025-08-14T21:44:24.7828206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7828551Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7828948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7829337Z sa_output = self.attention( 2025-08-14T21:44:24.7829703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:44:24.7830154Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:44:24.7830318Z 2025-08-14T21:44:24.7830421Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7830746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7831050Z return mod(**inputs) 2025-08-14T21:44:24.7831413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7831828Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7832221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7832604Z return self.transformer( 2025-08-14T21:44:24.7832979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7833363Z layer_outputs = layer_module( 2025-08-14T21:44:24.7833698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7834038Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7834437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7834823Z sa_output = self.attention( 2025-08-14T21:44:24.7835190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:44:24.7835609Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:24.7835765Z 2025-08-14T21:44:24.7835870Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7836189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7836483Z return mod(**inputs) 2025-08-14T21:44:24.7836841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7837224Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7837810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7838208Z return self.transformer( 2025-08-14T21:44:24.7838581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7838955Z layer_outputs = layer_module( 2025-08-14T21:44:24.7839276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7839615Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7840001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7840419Z sa_output = self.attention( 2025-08-14T21:44:24.7840866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:44:24.7841382Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:24.7841634Z 2025-08-14T21:44:24.7841723Z cudagraph partition due to non gpu ops 2025-08-14T21:44:24.7841982Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7842371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7842724Z return mod(**inputs) 2025-08-14T21:44:24.7843134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7843668Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7844122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7844571Z return self.transformer( 2025-08-14T21:44:24.7845006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7845523Z layer_outputs = layer_module( 2025-08-14T21:44:24.7845976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7846379Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7846841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7847289Z sa_output = self.attention( 2025-08-14T21:44:24.7847736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:44:24.7848284Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:44:24.7848507Z 2025-08-14T21:44:24.7848602Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7848956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7849253Z return mod(**inputs) 2025-08-14T21:44:24.7849615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7850030Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7850420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7850796Z return self.transformer( 2025-08-14T21:44:24.7851174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7851552Z layer_outputs = layer_module( 2025-08-14T21:44:24.7851865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7852194Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7852574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7852963Z sa_output = self.attention( 2025-08-14T21:44:24.7853331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:44:24.7853729Z attn_output = self.out_lin(attn_output) 2025-08-14T21:44:24.7853865Z 2025-08-14T21:44:24.7853968Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7854321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7854640Z return mod(**inputs) 2025-08-14T21:44:24.7855028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7855461Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7855891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7856347Z return self.transformer( 2025-08-14T21:44:24.7856735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7857121Z layer_outputs = layer_module( 2025-08-14T21:44:24.7857442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7857785Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7858220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:24.7858690Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:24.7859163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:24.7859699Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:24.7860208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:24.7860570Z return forward_fn(*input_tensors) 2025-08-14T21:44:24.7860957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:44:24.7861343Z x = self.lin1(input) 2025-08-14T21:44:24.7861443Z 2025-08-14T21:44:24.7861547Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7861893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7862212Z return mod(**inputs) 2025-08-14T21:44:24.7862596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7862986Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7863366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7863750Z return self.transformer( 2025-08-14T21:44:24.7864112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7864481Z layer_outputs = layer_module( 2025-08-14T21:44:24.7864805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7865144Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7865531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:24.7865932Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:24.7866342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:24.7866835Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:24.7867308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:24.7867666Z return forward_fn(*input_tensors) 2025-08-14T21:44:24.7868052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:44:24.7868444Z x = self.activation(x) 2025-08-14T21:44:24.7868744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:24.7869060Z return self.act(input) 2025-08-14T21:44:24.7869167Z 2025-08-14T21:44:24.7869262Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7869602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7869915Z return mod(**inputs) 2025-08-14T21:44:24.7870278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7870679Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7871066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7871456Z return self.transformer( 2025-08-14T21:44:24.7871830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7872219Z layer_outputs = layer_module( 2025-08-14T21:44:24.7872536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7872877Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7873273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:24.7873713Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:24.7874127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:24.7874628Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:24.7875135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:24.7875511Z return forward_fn(*input_tensors) 2025-08-14T21:44:24.7875913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:44:24.7876297Z x = self.lin2(x) 2025-08-14T21:44:24.7876392Z 2025-08-14T21:44:24.7876496Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7876831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7877139Z return mod(**inputs) 2025-08-14T21:44:24.7877510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7877905Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7878290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7878678Z return self.transformer( 2025-08-14T21:44:24.7879051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7879431Z layer_outputs = layer_module( 2025-08-14T21:44:24.7879757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7880102Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7880499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7880893Z sa_output = self.attention( 2025-08-14T21:44:24.7881282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:44:24.7881732Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:44:24.7881904Z 2025-08-14T21:44:24.7882014Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7882355Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7882678Z return mod(**inputs) 2025-08-14T21:44:24.7883059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7883490Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7883898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7884302Z return self.transformer( 2025-08-14T21:44:24.7884690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7885091Z layer_outputs = layer_module( 2025-08-14T21:44:24.7885503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7885902Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7886333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7886751Z sa_output = self.attention( 2025-08-14T21:44:24.7887185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:44:24.7887658Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:24.7887832Z 2025-08-14T21:44:24.7887940Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7888297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7888620Z return mod(**inputs) 2025-08-14T21:44:24.7889073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7889491Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7889922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7890338Z return self.transformer( 2025-08-14T21:44:24.7890732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7891143Z layer_outputs = layer_module( 2025-08-14T21:44:24.7891490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7891850Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7892256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7892749Z sa_output = self.attention( 2025-08-14T21:44:24.7893147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:44:24.7893604Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:24.7893780Z 2025-08-14T21:44:24.7893862Z cudagraph partition due to non gpu ops 2025-08-14T21:44:24.7894097Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7894451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7894765Z return mod(**inputs) 2025-08-14T21:44:24.7895157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7895577Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7895989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7896392Z return self.transformer( 2025-08-14T21:44:24.7896782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7897191Z layer_outputs = layer_module( 2025-08-14T21:44:24.7897531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7897904Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7898319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7898726Z sa_output = self.attention( 2025-08-14T21:44:24.7899112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:44:24.7899588Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:44:24.7899773Z 2025-08-14T21:44:24.7899874Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7900217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7900526Z return mod(**inputs) 2025-08-14T21:44:24.7900899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7901340Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7901738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7902143Z return self.transformer( 2025-08-14T21:44:24.7902521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7902918Z layer_outputs = layer_module( 2025-08-14T21:44:24.7903262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7903613Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7904031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7904428Z sa_output = self.attention( 2025-08-14T21:44:24.7904807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:44:24.7905217Z attn_output = self.out_lin(attn_output) 2025-08-14T21:44:24.7905348Z 2025-08-14T21:44:24.7905455Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7905796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7906112Z return mod(**inputs) 2025-08-14T21:44:24.7906493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7906903Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7907303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7907702Z return self.transformer( 2025-08-14T21:44:24.7908090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7908492Z layer_outputs = layer_module( 2025-08-14T21:44:24.7908820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7909166Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7909565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:24.7909969Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:24.7910382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:24.7910885Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:24.7911384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:24.7911752Z return forward_fn(*input_tensors) 2025-08-14T21:44:24.7912144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:44:24.7912531Z x = self.lin1(input) 2025-08-14T21:44:24.7912627Z 2025-08-14T21:44:24.7912730Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7913057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7913363Z return mod(**inputs) 2025-08-14T21:44:24.7913733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7914117Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7914501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7914898Z return self.transformer( 2025-08-14T21:44:24.7915257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7915631Z layer_outputs = layer_module( 2025-08-14T21:44:24.7915958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7916295Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7916708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:24.7917131Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:24.7917557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:24.7918060Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:24.7918528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:24.7918897Z return forward_fn(*input_tensors) 2025-08-14T21:44:24.7919278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:44:24.7919658Z x = self.activation(x) 2025-08-14T21:44:24.7919956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:24.7920276Z return self.act(input) 2025-08-14T21:44:24.7920377Z 2025-08-14T21:44:24.7920484Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7920842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7921144Z return mod(**inputs) 2025-08-14T21:44:24.7921509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7921904Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7922292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7922682Z return self.transformer( 2025-08-14T21:44:24.7923041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7923421Z layer_outputs = layer_module( 2025-08-14T21:44:24.7923730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7924064Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7924456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:24.7924892Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:24.7925334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:24.7926013Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:24.7926571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:24.7926992Z return forward_fn(*input_tensors) 2025-08-14T21:44:24.7927437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:44:24.7927954Z x = self.lin2(x) 2025-08-14T21:44:24.7928060Z 2025-08-14T21:44:24.7928176Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7928561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7928950Z return mod(**inputs) 2025-08-14T21:44:24.7929378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7929838Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7930279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7930728Z return self.transformer( 2025-08-14T21:44:24.7931167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7931610Z layer_outputs = layer_module( 2025-08-14T21:44:24.7931997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7932393Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7932834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7933278Z sa_output = self.attention( 2025-08-14T21:44:24.7933697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:44:24.7934183Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:44:24.7934371Z 2025-08-14T21:44:24.7934488Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7934862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7935204Z return mod(**inputs) 2025-08-14T21:44:24.7935617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7936052Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7936492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7936927Z return self.transformer( 2025-08-14T21:44:24.7937345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7937982Z layer_outputs = layer_module( 2025-08-14T21:44:24.7938355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7938746Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7939184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7939624Z sa_output = self.attention( 2025-08-14T21:44:24.7940048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:44:24.7940587Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:24.7940773Z 2025-08-14T21:44:24.7940881Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7941264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7941608Z return mod(**inputs) 2025-08-14T21:44:24.7942005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7942406Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7942810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7943215Z return self.transformer( 2025-08-14T21:44:24.7943589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7944019Z layer_outputs = layer_module( 2025-08-14T21:44:24.7944358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7944701Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7945090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7945484Z sa_output = self.attention( 2025-08-14T21:44:24.7945894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:44:24.7946331Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:24.7946508Z 2025-08-14T21:44:24.7946610Z cudagraph partition due to non gpu ops 2025-08-14T21:44:24.7946844Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7947195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7947504Z return mod(**inputs) 2025-08-14T21:44:24.7947883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7948291Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7948691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7949082Z return self.transformer( 2025-08-14T21:44:24.7949466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7949865Z layer_outputs = layer_module( 2025-08-14T21:44:24.7950191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7950546Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7950951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7951354Z sa_output = self.attention( 2025-08-14T21:44:24.7951730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:44:24.7952184Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:44:24.7952366Z 2025-08-14T21:44:24.7952481Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7952828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7953134Z return mod(**inputs) 2025-08-14T21:44:24.7953514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7953943Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7954343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7954727Z return self.transformer( 2025-08-14T21:44:24.7955097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7955484Z layer_outputs = layer_module( 2025-08-14T21:44:24.7955792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7956124Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7956498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7956867Z sa_output = self.attention( 2025-08-14T21:44:24.7957233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:44:24.7957644Z attn_output = self.out_lin(attn_output) 2025-08-14T21:44:24.7957767Z 2025-08-14T21:44:24.7957867Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7958187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7958485Z return mod(**inputs) 2025-08-14T21:44:24.7958840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7959242Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7959614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7959989Z return self.transformer( 2025-08-14T21:44:24.7960372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7960747Z layer_outputs = layer_module( 2025-08-14T21:44:24.7961067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7961405Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7961793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:24.7962205Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:24.7962628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:24.7963132Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:24.7963613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:24.7963980Z return forward_fn(*input_tensors) 2025-08-14T21:44:24.7964365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:44:24.7964747Z x = self.lin1(input) 2025-08-14T21:44:24.7964845Z 2025-08-14T21:44:24.7964941Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7965277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7965657Z return mod(**inputs) 2025-08-14T21:44:24.7966058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7966479Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7966897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7967361Z return self.transformer( 2025-08-14T21:44:24.7967737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7968124Z layer_outputs = layer_module( 2025-08-14T21:44:24.7968464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7968796Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7969171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:24.7969583Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:24.7969995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:24.7970509Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:24.7971021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:24.7971398Z return forward_fn(*input_tensors) 2025-08-14T21:44:24.7971790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:44:24.7972188Z x = self.activation(x) 2025-08-14T21:44:24.7972483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:24.7972803Z return self.act(input) 2025-08-14T21:44:24.7972925Z 2025-08-14T21:44:24.7973033Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7973367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7973691Z return mod(**inputs) 2025-08-14T21:44:24.7974062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7974461Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7974846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7975231Z return self.transformer( 2025-08-14T21:44:24.7975600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7975979Z layer_outputs = layer_module( 2025-08-14T21:44:24.7976307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7976642Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7977028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:24.7977443Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:24.7977863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:24.7978366Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:24.7978845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:24.7979212Z return forward_fn(*input_tensors) 2025-08-14T21:44:24.7979602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:44:24.7979979Z x = self.lin2(x) 2025-08-14T21:44:24.7980073Z 2025-08-14T21:44:24.7980179Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7980507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7980838Z return mod(**inputs) 2025-08-14T21:44:24.7981206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7981599Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7981996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7982383Z return self.transformer( 2025-08-14T21:44:24.7982759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7983138Z layer_outputs = layer_module( 2025-08-14T21:44:24.7983465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7983805Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7984193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7984597Z sa_output = self.attention( 2025-08-14T21:44:24.7984971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 390, in forward 2025-08-14T21:44:24.7985409Z q = shape(self.q_lin(query)) # (bs, n_heads, q_length, dim_per_head) 2025-08-14T21:44:24.7985584Z 2025-08-14T21:44:24.7985680Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7986020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7986318Z return mod(**inputs) 2025-08-14T21:44:24.7986670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7987101Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7987488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7987872Z return self.transformer( 2025-08-14T21:44:24.7988233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7988610Z layer_outputs = layer_module( 2025-08-14T21:44:24.7988932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7989261Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7989638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7990027Z sa_output = self.attention( 2025-08-14T21:44:24.7990405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 391, in forward 2025-08-14T21:44:24.7990855Z k = shape(self.k_lin(key)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:24.7991018Z 2025-08-14T21:44:24.7991115Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7991453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7991800Z return mod(**inputs) 2025-08-14T21:44:24.7992153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7992556Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7992967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7993352Z return self.transformer( 2025-08-14T21:44:24.7993718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7994128Z layer_outputs = layer_module( 2025-08-14T21:44:24.7994452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.7994784Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.7995217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.7995591Z sa_output = self.attention( 2025-08-14T21:44:24.7995957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 392, in forward 2025-08-14T21:44:24.7996371Z v = shape(self.v_lin(value)) # (bs, n_heads, k_length, dim_per_head) 2025-08-14T21:44:24.7996543Z 2025-08-14T21:44:24.7996618Z cudagraph partition due to non gpu ops 2025-08-14T21:44:24.7996838Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.7997166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.7997479Z return mod(**inputs) 2025-08-14T21:44:24.7997839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.7998227Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.7998598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.7998979Z return self.transformer( 2025-08-14T21:44:24.7999352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.7999729Z layer_outputs = layer_module( 2025-08-14T21:44:24.8000054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.8000389Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.8000780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.8001157Z sa_output = self.attention( 2025-08-14T21:44:24.8001530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 402, in forward 2025-08-14T21:44:24.8001974Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:44:24.8002142Z 2025-08-14T21:44:24.8002245Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.8002564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.8002869Z return mod(**inputs) 2025-08-14T21:44:24.8003238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.8003633Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.8004015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.8004404Z return self.transformer( 2025-08-14T21:44:24.8004776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.8005155Z layer_outputs = layer_module( 2025-08-14T21:44:24.8005548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.8005910Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.8006319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 476, in forward 2025-08-14T21:44:24.8006745Z sa_output = self.attention( 2025-08-14T21:44:24.8007148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 412, in forward 2025-08-14T21:44:24.8007572Z attn_output = self.out_lin(attn_output) 2025-08-14T21:44:24.8007702Z 2025-08-14T21:44:24.8007807Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.8008140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.8008447Z return mod(**inputs) 2025-08-14T21:44:24.8008815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.8009201Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.8009597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.8009980Z return self.transformer( 2025-08-14T21:44:24.8010348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.8010729Z layer_outputs = layer_module( 2025-08-14T21:44:24.8011076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.8011425Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.8011821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:24.8012240Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:24.8012661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:24.8013196Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:24.8013708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:24.8014096Z return forward_fn(*input_tensors) 2025-08-14T21:44:24.8014492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 431, in ff_chunk 2025-08-14T21:44:24.8014888Z x = self.lin1(input) 2025-08-14T21:44:24.8014990Z 2025-08-14T21:44:24.8015091Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.8015449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.8015757Z return mod(**inputs) 2025-08-14T21:44:24.8016115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.8016508Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.8016900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.8017282Z return self.transformer( 2025-08-14T21:44:24.8017648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.8018034Z layer_outputs = layer_module( 2025-08-14T21:44:24.8018357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.8018694Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.8019087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:24.8019492Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:24.8019899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:24.8020387Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:24.8020855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:24.8021264Z return forward_fn(*input_tensors) 2025-08-14T21:44:24.8021652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 432, in ff_chunk 2025-08-14T21:44:24.8022030Z x = self.activation(x) 2025-08-14T21:44:24.8022348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:44:24.8022663Z return self.act(input) 2025-08-14T21:44:24.8022764Z 2025-08-14T21:44:24.8022868Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.8023201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.8023512Z return mod(**inputs) 2025-08-14T21:44:24.8023878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1031, in forward 2025-08-14T21:44:24.8024285Z distilbert_output = self.distilbert( 2025-08-14T21:44:24.8024682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 736, in forward 2025-08-14T21:44:24.8025057Z return self.transformer( 2025-08-14T21:44:24.8025415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 541, in forward 2025-08-14T21:44:24.8025787Z layer_outputs = layer_module( 2025-08-14T21:44:24.8026113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:24.8026468Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:24.8026862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 494, in forward 2025-08-14T21:44:24.8027294Z ffn_output = self.ffn(sa_output) # (bs, seq_length, dim) 2025-08-14T21:44:24.8027719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 428, in forward 2025-08-14T21:44:24.8028223Z return apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, input) 2025-08-14T21:44:24.8028684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:44:24.8029051Z return forward_fn(*input_tensors) 2025-08-14T21:44:24.8029439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 433, in ff_chunk 2025-08-14T21:44:24.8029823Z x = self.lin2(x) 2025-08-14T21:44:24.8029917Z 2025-08-14T21:44:24.8030014Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.8030361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.8030679Z return mod(**inputs) 2025-08-14T21:44:24.8031061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1043, in forward 2025-08-14T21:44:24.8031494Z logits = self.qa_outputs(hidden_states) # (bs, max_query_len, 2) 2025-08-14T21:44:24.8031666Z 2025-08-14T21:44:24.8031761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.8032100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.8032399Z return mod(**inputs) 2025-08-14T21:44:24.8032770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1061, in forward 2025-08-14T21:44:24.8033185Z start_loss = loss_fct(start_logits, start_positions) 2025-08-14T21:44:24.8033329Z 2025-08-14T21:44:24.8033428Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:24.8033754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:44:24.8034082Z return mod(**inputs) 2025-08-14T21:44:24.8034446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py", line 1062, in forward 2025-08-14T21:44:24.8034852Z end_loss = loss_fct(end_logits, end_positions) 2025-08-14T21:44:24.8034991Z 2025-08-14T21:44:31.3733099Z Compilation time (from dynamo_timed): 10.151792472 2025-08-14T21:44:31.3735137Z pass 2025-08-14T21:44:31.3736057Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:44:31.3740726Z TIMING: _recursive_pre_grad_passes:0.00475 _recursive_joint_graph_passes:0.23291 _recursive_post_grad_passes:0.05728 async_compile.wait:0.6553 code_gen:6.29783 inductor_compile:7.18859 backend_compile:8.85641 gc:0.00173 entire_frame_compile:10.15179 total_wall_time:10.15179 2025-08-14T21:44:31.3742283Z STATS: call_* op count: 161 | FakeTensorMode.__torch_dispatch__:6705 | FakeTensor.__torch_dispatch__:2556 | ProxyTorchDispatchMode.__torch_dispatch__:2400 2025-08-14T21:44:31.3743553Z Dynamo produced 1 graphs covering 161 ops with 0 graph breaks (0 unique) 2025-08-14T21:44:36.0243790Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:44:36.0244701Z from pkg_resources import resource_filename 2025-08-14T21:44:36.6002025Z 2025-08-14T21:44:38.5666849Z loading model: 0it [00:00, ?it/s]`loss_type=None` was set in the config but it is unrecognised.Using the default loss: `ForCausalLMLoss`. 2025-08-14T21:44:38.5668968Z WARNING:transformers.modeling_utils:`loss_type=None` was set in the config but it is unrecognised.Using the default loss: `ForCausalLMLoss`. 2025-08-14T21:44:38.5950778Z 2025-08-14T21:44:38.5956019Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:44:38.5959203Z cpu eval DistillGPT2 2025-08-14T21:44:38.9864356Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:44:39.1624676Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:44:39.3527222Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:44:45.0390761Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0391074Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0391302Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0391499Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0391686Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0391881Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0392123Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0392566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0393033Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0393437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0394470Z outputs = block( 2025-08-14T21:44:45.0394802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0395160Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0395581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0395941Z return func(*args, **kwargs) 2025-08-14T21:44:45.0396354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0396751Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0397419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0397885Z return func(*args, **kwargs) 2025-08-14T21:44:45.0398254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:44:45.0398726Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:44:45.0399169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0399575Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0399759Z 2025-08-14T21:44:45.0399840Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0400049Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0400243Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0400444Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0400745Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0401147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0401516Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0401886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0402240Z outputs = block( 2025-08-14T21:44:45.0402593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0402943Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0403300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0403702Z return func(*args, **kwargs) 2025-08-14T21:44:45.0404063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0404580Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0404972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0405364Z return func(*args, **kwargs) 2025-08-14T21:44:45.0405925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:44:45.0406331Z attn_output, attn_weights = attention_interface( 2025-08-14T21:44:45.0406770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:44:45.0407244Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:44:45.0407445Z 2025-08-14T21:44:45.0407551Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0407963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0408342Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0408707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0409064Z outputs = block( 2025-08-14T21:44:45.0409377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0409721Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0410096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0410450Z return func(*args, **kwargs) 2025-08-14T21:44:45.0410802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0411196Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0411561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0411913Z return func(*args, **kwargs) 2025-08-14T21:44:45.0412263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:44:45.0412645Z attn_output, attn_weights = attention_interface( 2025-08-14T21:44:45.0413070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:44:45.0413505Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:44:45.0413657Z 2025-08-14T21:44:45.0413887Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0414270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0414673Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0415041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0415391Z outputs = block( 2025-08-14T21:44:45.0415697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0416041Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0416402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0416801Z return func(*args, **kwargs) 2025-08-14T21:44:45.0417156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0417548Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0417907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0418256Z return func(*args, **kwargs) 2025-08-14T21:44:45.0418601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:44:45.0418964Z attn_output = self.c_proj(attn_output) 2025-08-14T21:44:45.0419295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0419672Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0419835Z 2025-08-14T21:44:45.0419944Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0420333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0420694Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0421059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0421408Z outputs = block( 2025-08-14T21:44:45.0421698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0422036Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0422387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0422734Z return func(*args, **kwargs) 2025-08-14T21:44:45.0423065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:44:45.0423448Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:44:45.0423830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:44:45.0424186Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:44:45.0424543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0424917Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0425080Z 2025-08-14T21:44:45.0425188Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0425568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0425942Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0426305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0426656Z outputs = block( 2025-08-14T21:44:45.0426953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0427293Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0427652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0428015Z return func(*args, **kwargs) 2025-08-14T21:44:45.0428363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:44:45.0428759Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:44:45.0429154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:44:45.0429517Z hidden_states = self.act(hidden_states) 2025-08-14T21:44:45.0429868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:44:45.0430317Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:44:45.0430581Z 2025-08-14T21:44:45.0430691Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0431088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0431475Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0431849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0432202Z outputs = block( 2025-08-14T21:44:45.0432513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0432866Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0433233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0433589Z return func(*args, **kwargs) 2025-08-14T21:44:45.0433951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:44:45.0434355Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:44:45.0434743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:44:45.0435127Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:44:45.0435479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0435866Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0436032Z 2025-08-14T21:44:45.0436133Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0436538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0436924Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0437300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0438096Z outputs = block( 2025-08-14T21:44:45.0438419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0438774Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0439135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0439497Z return func(*args, **kwargs) 2025-08-14T21:44:45.0439853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0440238Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0440604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0440965Z return func(*args, **kwargs) 2025-08-14T21:44:45.0441319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:44:45.0441852Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:44:45.0442291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0442680Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0442847Z 2025-08-14T21:44:45.0442934Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0443135Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0443348Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0443571Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0443795Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0444208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0444591Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0444964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0445311Z outputs = block( 2025-08-14T21:44:45.0445617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0446104Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0446508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0446891Z return func(*args, **kwargs) 2025-08-14T21:44:45.0447293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0447667Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0448029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0448393Z return func(*args, **kwargs) 2025-08-14T21:44:45.0448742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:44:45.0449123Z attn_output, attn_weights = attention_interface( 2025-08-14T21:44:45.0449546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:44:45.0450013Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:44:45.0450195Z 2025-08-14T21:44:45.0450307Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0450723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0451144Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0451513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0452738Z outputs = block( 2025-08-14T21:44:45.0453036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0453383Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0453746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0454102Z return func(*args, **kwargs) 2025-08-14T21:44:45.0454448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0454830Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0455198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0455546Z return func(*args, **kwargs) 2025-08-14T21:44:45.0455895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:44:45.0456301Z attn_output, attn_weights = attention_interface( 2025-08-14T21:44:45.0456717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:44:45.0457146Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:44:45.0457305Z 2025-08-14T21:44:45.0457403Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0457812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0458183Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0458539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0458900Z outputs = block( 2025-08-14T21:44:45.0459208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0459545Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0459899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0460248Z return func(*args, **kwargs) 2025-08-14T21:44:45.0460594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0460956Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0461321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0461669Z return func(*args, **kwargs) 2025-08-14T21:44:45.0462016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:44:45.0462386Z attn_output = self.c_proj(attn_output) 2025-08-14T21:44:45.0462736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0463124Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0463289Z 2025-08-14T21:44:45.0463389Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0463788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0464157Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0464519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0464871Z outputs = block( 2025-08-14T21:44:45.0465181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0465534Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0465935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0466307Z return func(*args, **kwargs) 2025-08-14T21:44:45.0466672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:44:45.0467074Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:44:45.0467464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:44:45.0467845Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:44:45.0468195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0468583Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0468753Z 2025-08-14T21:44:45.0468857Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0469296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0469685Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0470053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0470418Z outputs = block( 2025-08-14T21:44:45.0470732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0471083Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0471459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0471825Z return func(*args, **kwargs) 2025-08-14T21:44:45.0472197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:44:45.0472608Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:44:45.0473005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:44:45.0473393Z hidden_states = self.act(hidden_states) 2025-08-14T21:44:45.0473740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:44:45.0474182Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:44:45.0474422Z 2025-08-14T21:44:45.0474526Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0474942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0475335Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0475716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0476089Z outputs = block( 2025-08-14T21:44:45.0476408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0476771Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0477142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0477566Z return func(*args, **kwargs) 2025-08-14T21:44:45.0477923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:44:45.0478312Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:44:45.0478708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:44:45.0479088Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:44:45.0479446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0479812Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0479982Z 2025-08-14T21:44:45.0480080Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0480469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0480841Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0481199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0481550Z outputs = block( 2025-08-14T21:44:45.0481853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0482188Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0482546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0482924Z return func(*args, **kwargs) 2025-08-14T21:44:45.0483281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0483657Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0484025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0484384Z return func(*args, **kwargs) 2025-08-14T21:44:45.0484753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:44:45.0485220Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:44:45.0485788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0486229Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0486409Z 2025-08-14T21:44:45.0486496Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0486724Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0486944Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0487135Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0487356Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0487751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0488134Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0488491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0488845Z outputs = block( 2025-08-14T21:44:45.0489154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0489502Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0489845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0490197Z return func(*args, **kwargs) 2025-08-14T21:44:45.0490543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0490909Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0491270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0491618Z return func(*args, **kwargs) 2025-08-14T21:44:45.0491960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:44:45.0492336Z attn_output, attn_weights = attention_interface( 2025-08-14T21:44:45.0492772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:44:45.0493225Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:44:45.0493404Z 2025-08-14T21:44:45.0493499Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0493878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0494240Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0494596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0494928Z outputs = block( 2025-08-14T21:44:45.0495226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0495559Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0495908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0496296Z return func(*args, **kwargs) 2025-08-14T21:44:45.0496637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0497001Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0497350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0497693Z return func(*args, **kwargs) 2025-08-14T21:44:45.0498060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:44:45.0498434Z attn_output, attn_weights = attention_interface( 2025-08-14T21:44:45.0498852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:44:45.0499275Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:44:45.0499421Z 2025-08-14T21:44:45.0499523Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0499901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0500255Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0500606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0500946Z outputs = block( 2025-08-14T21:44:45.0501231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0501564Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0501920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0502282Z return func(*args, **kwargs) 2025-08-14T21:44:45.0502613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0502978Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0503333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0503663Z return func(*args, **kwargs) 2025-08-14T21:44:45.0504000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:44:45.0504356Z attn_output = self.c_proj(attn_output) 2025-08-14T21:44:45.0504683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0505046Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0505214Z 2025-08-14T21:44:45.0505329Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0505720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0506105Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0506451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0506791Z outputs = block( 2025-08-14T21:44:45.0507086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0507415Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0507762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0508098Z return func(*args, **kwargs) 2025-08-14T21:44:45.0508432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:44:45.0508816Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:44:45.0509189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:44:45.0509543Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:44:45.0509860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0510221Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0510385Z 2025-08-14T21:44:45.0510495Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0510879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0511254Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0511607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0511949Z outputs = block( 2025-08-14T21:44:45.0512241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0512563Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0512909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0513257Z return func(*args, **kwargs) 2025-08-14T21:44:45.0513597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:44:45.0513981Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:44:45.0514362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:44:45.0514733Z hidden_states = self.act(hidden_states) 2025-08-14T21:44:45.0515046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:44:45.0515466Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:44:45.0515690Z 2025-08-14T21:44:45.0515786Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0516171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0516531Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0516889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0517233Z outputs = block( 2025-08-14T21:44:45.0517523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0517864Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0518254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0518608Z return func(*args, **kwargs) 2025-08-14T21:44:45.0518949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:44:45.0519336Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:44:45.0519714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:44:45.0520085Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:44:45.0520422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0520802Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0520965Z 2025-08-14T21:44:45.0521069Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0521472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0521841Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0522201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0522550Z outputs = block( 2025-08-14T21:44:45.0522845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0523188Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0523557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0523909Z return func(*args, **kwargs) 2025-08-14T21:44:45.0524278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-08-14T21:44:45.0524678Z hidden_states = residual + feed_forward_hidden_states 2025-08-14T21:44:45.0524838Z 2025-08-14T21:44:45.0524939Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0525315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0525762Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0526151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0526519Z outputs = block( 2025-08-14T21:44:45.0526831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0527191Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0527572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0527917Z return func(*args, **kwargs) 2025-08-14T21:44:45.0528271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0528652Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0529017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0529355Z return func(*args, **kwargs) 2025-08-14T21:44:45.0529700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:44:45.0530160Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:44:45.0530593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0530959Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0531174Z 2025-08-14T21:44:45.0531255Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0531459Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0531648Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0531846Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0532077Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0532464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0532829Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0533194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0533546Z outputs = block( 2025-08-14T21:44:45.0533842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0534184Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0534545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0534925Z return func(*args, **kwargs) 2025-08-14T21:44:45.0535272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0535653Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0536035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0536377Z return func(*args, **kwargs) 2025-08-14T21:44:45.0536742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:44:45.0537137Z attn_output, attn_weights = attention_interface( 2025-08-14T21:44:45.0537582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:44:45.0538214Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:44:45.0538403Z 2025-08-14T21:44:45.0538506Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0538914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0539304Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0539673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0540031Z outputs = block( 2025-08-14T21:44:45.0540348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0540691Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0541061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0541425Z return func(*args, **kwargs) 2025-08-14T21:44:45.0541778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0542154Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0542528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0542886Z return func(*args, **kwargs) 2025-08-14T21:44:45.0543235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:44:45.0543630Z attn_output, attn_weights = attention_interface( 2025-08-14T21:44:45.0544061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:44:45.0544506Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:44:45.0544710Z 2025-08-14T21:44:45.0544810Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0545209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0545589Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0545964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0546313Z outputs = block( 2025-08-14T21:44:45.0546626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0546979Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0547337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0547698Z return func(*args, **kwargs) 2025-08-14T21:44:45.0548055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0548471Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0548829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0549186Z return func(*args, **kwargs) 2025-08-14T21:44:45.0549545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:44:45.0549926Z attn_output = self.c_proj(attn_output) 2025-08-14T21:44:45.0550277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0550642Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0550801Z 2025-08-14T21:44:45.0550923Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0551295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0551661Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0552015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0552356Z outputs = block( 2025-08-14T21:44:45.0552642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0552978Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0553329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0553672Z return func(*args, **kwargs) 2025-08-14T21:44:45.0554017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:44:45.0554402Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:44:45.0554782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:44:45.0555151Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:44:45.0555475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0555837Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0555994Z 2025-08-14T21:44:45.0556094Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0556466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0556824Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0557178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0557507Z outputs = block( 2025-08-14T21:44:45.0557822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0558160Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0558506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0558854Z return func(*args, **kwargs) 2025-08-14T21:44:45.0559205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:44:45.0559604Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:44:45.0559981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:44:45.0560332Z hidden_states = self.act(hidden_states) 2025-08-14T21:44:45.0560661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:44:45.0561100Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:44:45.0561332Z 2025-08-14T21:44:45.0561427Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0561807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0562173Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0562525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0562860Z outputs = block( 2025-08-14T21:44:45.0563174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0563510Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0563885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0564222Z return func(*args, **kwargs) 2025-08-14T21:44:45.0564564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:44:45.0564948Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:44:45.0565320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:44:45.0565751Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:44:45.0566101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0566473Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0566632Z 2025-08-14T21:44:45.0566730Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0567130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0581570Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0582171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0582535Z outputs = block( 2025-08-14T21:44:45.0582841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0583193Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0583558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0583925Z return func(*args, **kwargs) 2025-08-14T21:44:45.0584278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0584665Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0585036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0585474Z return func(*args, **kwargs) 2025-08-14T21:44:45.0585837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:44:45.0586296Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:44:45.0586725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0587090Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0587267Z 2025-08-14T21:44:45.0587348Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0587554Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0587740Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0587931Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0588154Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0588586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0588953Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0589317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0589661Z outputs = block( 2025-08-14T21:44:45.0589953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0590291Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0590673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0591022Z return func(*args, **kwargs) 2025-08-14T21:44:45.0591390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0591760Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0592117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0592467Z return func(*args, **kwargs) 2025-08-14T21:44:45.0592821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:44:45.0593191Z attn_output, attn_weights = attention_interface( 2025-08-14T21:44:45.0593595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:44:45.0594055Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:44:45.0594227Z 2025-08-14T21:44:45.0594335Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0594735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0595105Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0595472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0595824Z outputs = block( 2025-08-14T21:44:45.0596120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0596462Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0596820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0597170Z return func(*args, **kwargs) 2025-08-14T21:44:45.0597507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0597895Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0598276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0598623Z return func(*args, **kwargs) 2025-08-14T21:44:45.0598967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:44:45.0599349Z attn_output, attn_weights = attention_interface( 2025-08-14T21:44:45.0599765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:44:45.0600190Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:44:45.0600354Z 2025-08-14T21:44:45.0600455Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0600847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0601230Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0601590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0601960Z outputs = block( 2025-08-14T21:44:45.0602262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0602593Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0602943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0603285Z return func(*args, **kwargs) 2025-08-14T21:44:45.0603647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0604010Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0604396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0604771Z return func(*args, **kwargs) 2025-08-14T21:44:45.0605147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:44:45.0605576Z attn_output = self.c_proj(attn_output) 2025-08-14T21:44:45.0606120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0606548Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0606731Z 2025-08-14T21:44:45.0606850Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0607246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0607624Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0607991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0608340Z outputs = block( 2025-08-14T21:44:45.0608648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0608993Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0609342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0609694Z return func(*args, **kwargs) 2025-08-14T21:44:45.0610045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:44:45.0610424Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:44:45.0610793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:44:45.0611147Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:44:45.0611475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0611859Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0612017Z 2025-08-14T21:44:45.0612110Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0612485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0612844Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0613189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0613535Z outputs = block( 2025-08-14T21:44:45.0613839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0614175Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0614525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0614895Z return func(*args, **kwargs) 2025-08-14T21:44:45.0615254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:44:45.0615629Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:44:45.0616012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:44:45.0616381Z hidden_states = self.act(hidden_states) 2025-08-14T21:44:45.0616711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:44:45.0617155Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:44:45.0617382Z 2025-08-14T21:44:45.0617508Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0617899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0618276Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0618630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0618981Z outputs = block( 2025-08-14T21:44:45.0619284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0619616Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0619972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0620326Z return func(*args, **kwargs) 2025-08-14T21:44:45.0620672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:44:45.0621052Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:44:45.0621434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:44:45.0621806Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:44:45.0622146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0622509Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0622676Z 2025-08-14T21:44:45.0622774Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0623163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0623526Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0623889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0624238Z outputs = block( 2025-08-14T21:44:45.0624561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0624895Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0625250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0625605Z return func(*args, **kwargs) 2025-08-14T21:44:45.0625954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-08-14T21:44:45.0626334Z hidden_states = residual + feed_forward_hidden_states 2025-08-14T21:44:45.0626492Z 2025-08-14T21:44:45.0626590Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0626974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0627340Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0627706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0628072Z outputs = block( 2025-08-14T21:44:45.0628373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0628703Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0629061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0629399Z return func(*args, **kwargs) 2025-08-14T21:44:45.0629751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0630112Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0630473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0630813Z return func(*args, **kwargs) 2025-08-14T21:44:45.0631146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:44:45.0631595Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:44:45.0632018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0632386Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0632544Z 2025-08-14T21:44:45.0632621Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0632824Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0633024Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0633219Z cudagraph partition due to non gpu ops 2025-08-14T21:44:45.0633434Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0633820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0634189Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0634538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0634878Z outputs = block( 2025-08-14T21:44:45.0635178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0635502Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0635851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0636195Z return func(*args, **kwargs) 2025-08-14T21:44:45.0636532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0636888Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0637264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0637811Z return func(*args, **kwargs) 2025-08-14T21:44:45.0638158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:44:45.0638535Z attn_output, attn_weights = attention_interface( 2025-08-14T21:44:45.0638943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:44:45.0639391Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:44:45.0639562Z 2025-08-14T21:44:45.0639660Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0640044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0640414Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0640779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0641179Z outputs = block( 2025-08-14T21:44:45.0641486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0641827Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0642174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0642525Z return func(*args, **kwargs) 2025-08-14T21:44:45.0642905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0643291Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0643685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0644045Z return func(*args, **kwargs) 2025-08-14T21:44:45.0644396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:44:45.0644787Z attn_output, attn_weights = attention_interface( 2025-08-14T21:44:45.0645203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:44:45.0645714Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:44:45.0645898Z 2025-08-14T21:44:45.0646017Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0646455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0646874Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0647285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0647655Z outputs = block( 2025-08-14T21:44:45.0647969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0648325Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0648693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0649048Z return func(*args, **kwargs) 2025-08-14T21:44:45.0649405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:44:45.0649789Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:44:45.0650158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0650519Z return func(*args, **kwargs) 2025-08-14T21:44:45.0650880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:44:45.0651288Z attn_output = self.c_proj(attn_output) 2025-08-14T21:44:45.0651624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0652006Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0652169Z 2025-08-14T21:44:45.0652276Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0652672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0653049Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0653421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0653782Z outputs = block( 2025-08-14T21:44:45.0654084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0654459Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0654820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0655175Z return func(*args, **kwargs) 2025-08-14T21:44:45.0655501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:44:45.0655876Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:44:45.0656264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:44:45.0656613Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:44:45.0656940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0657316Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0657478Z 2025-08-14T21:44:45.0657583Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0657963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0658328Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0658683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0659024Z outputs = block( 2025-08-14T21:44:45.0659312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0659645Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0659991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0660330Z return func(*args, **kwargs) 2025-08-14T21:44:45.0660670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:44:45.0661051Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:44:45.0661429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:44:45.0661778Z hidden_states = self.act(hidden_states) 2025-08-14T21:44:45.0662098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:44:45.0662520Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:44:45.0662740Z 2025-08-14T21:44:45.0662844Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0663219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1183, in forward 2025-08-14T21:44:45.0663584Z transformer_outputs = self.transformer( 2025-08-14T21:44:45.0663959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:44:45.0664289Z outputs = block( 2025-08-14T21:44:45.0664584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:44:45.0664911Z return super().__call__(*args, **kwargs) 2025-08-14T21:44:45.0665254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:44:45.0665585Z return func(*args, **kwargs) 2025-08-14T21:44:45.0665924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:44:45.0666296Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:44:45.0666662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:44:45.0667041Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:44:45.0667373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:44:45.0667740Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:44:45.0667902Z 2025-08-14T21:44:45.0667998Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:45.0668379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1207, in forward 2025-08-14T21:44:45.0668772Z logits = self.lm_head(hidden_states[:, slice_indices, :]) 2025-08-14T21:44:45.0668949Z 2025-08-14T21:44:52.6016359Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:44:52.6017276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-08-14T21:44:52.6017774Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-08-14T21:44:52.6018265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-08-14T21:44:52.6018752Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-08-14T21:44:52.6018994Z 2025-08-14T21:44:53.6773923Z Compilation time (from dynamo_timed): 13.02341879 2025-08-14T21:44:53.6881170Z pass 2025-08-14T21:44:53.6881636Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:44:53.6882471Z TIMING: gc:0.00466 entire_frame_compile:13.02342 _recursive_pre_grad_passes:0.00679 _recursive_joint_graph_passes:0.20538 _recursive_post_grad_passes:0.05444 async_compile.wait:1.33041 code_gen:8.04719 inductor_compile:8.68717 backend_compile:10.25745 total_wall_time:13.02342 2025-08-14T21:44:53.6883529Z STATS: call_* op count: 299 | FakeTensorMode.__torch_dispatch__:7245 | FakeTensor.__torch_dispatch__:2465 | ProxyTorchDispatchMode.__torch_dispatch__:2190 2025-08-14T21:44:53.6884047Z Dynamo produced 2 graphs covering 299 ops with 2 graph breaks (1 unique) 2025-08-14T21:44:58.6979499Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:44:58.6980811Z from pkg_resources import resource_filename 2025-08-14T21:44:59.4886435Z 2025-08-14T21:44:59.4902833Z loading model: 0it [00:00, ?it/s]If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2025-08-14T21:44:59.4903446Z WARNING:transformers.models.electra.modeling_electra:If you want to use `ElectraForCausalLM` as a standalone, add `is_decoder=True.` 2025-08-14T21:44:59.8789194Z 2025-08-14T21:44:59.8790016Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:44:59.8798594Z cpu eval ElectraForCausalLM 2025-08-14T21:45:00.0742957Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:45:00.1784740Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:45:00.2806604Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:45:08.0225063Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0225504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0228605Z return mod(**inputs) 2025-08-14T21:45:08.0229734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0233049Z outputs = self.electra( 2025-08-14T21:45:08.0233534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 797, in forward 2025-08-14T21:45:08.0234444Z hidden_states = self.embeddings_project(hidden_states) 2025-08-14T21:45:08.0234659Z 2025-08-14T21:45:08.0234777Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0235186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0235538Z return mod(**inputs) 2025-08-14T21:45:08.0235920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0236324Z outputs = self.electra( 2025-08-14T21:45:08.0236774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0237183Z hidden_states = self.encoder( 2025-08-14T21:45:08.0237821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0238242Z layer_outputs = layer_module( 2025-08-14T21:45:08.0238612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0239009Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0239456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0239914Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0240340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0240719Z return func(*args, **kwargs) 2025-08-14T21:45:08.0241151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0241607Z self_outputs = self.self( 2025-08-14T21:45:08.0242009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0242433Z return func(*args, **kwargs) 2025-08-14T21:45:08.0242858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:08.0243305Z query_layer = self.query(hidden_states) 2025-08-14T21:45:08.0243456Z 2025-08-14T21:45:08.0243568Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0243953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0244307Z return mod(**inputs) 2025-08-14T21:45:08.0244709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0245138Z outputs = self.electra( 2025-08-14T21:45:08.0245539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0246124Z hidden_states = self.encoder( 2025-08-14T21:45:08.0246529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0246947Z layer_outputs = layer_module( 2025-08-14T21:45:08.0247311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0247697Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0248104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0248508Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0248891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0249283Z return func(*args, **kwargs) 2025-08-14T21:45:08.0249696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0250146Z self_outputs = self.self( 2025-08-14T21:45:08.0250524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0250943Z return func(*args, **kwargs) 2025-08-14T21:45:08.0251335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:08.0251760Z key_layer = self.key(current_states) 2025-08-14T21:45:08.0251911Z 2025-08-14T21:45:08.0252050Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0252443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0252798Z return mod(**inputs) 2025-08-14T21:45:08.0253219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0253641Z outputs = self.electra( 2025-08-14T21:45:08.0254032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0254441Z hidden_states = self.encoder( 2025-08-14T21:45:08.0254848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0255260Z layer_outputs = layer_module( 2025-08-14T21:45:08.0255617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0255996Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0256421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0256846Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0257242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0257631Z return func(*args, **kwargs) 2025-08-14T21:45:08.0258031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0258435Z self_outputs = self.self( 2025-08-14T21:45:08.0258809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0259201Z return func(*args, **kwargs) 2025-08-14T21:45:08.0259600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:08.0259988Z value_layer = self.value(current_states) 2025-08-14T21:45:08.0260129Z 2025-08-14T21:45:08.0260214Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0260427Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0260676Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0261041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0261363Z return mod(**inputs) 2025-08-14T21:45:08.0261737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0262123Z outputs = self.electra( 2025-08-14T21:45:08.0262493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0262889Z hidden_states = self.encoder( 2025-08-14T21:45:08.0263280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0263667Z layer_outputs = layer_module( 2025-08-14T21:45:08.0264018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0264410Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0264803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0265213Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0265595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0265969Z return func(*args, **kwargs) 2025-08-14T21:45:08.0266357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:08.0266811Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:08.0267272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:08.0267682Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0267823Z 2025-08-14T21:45:08.0267927Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0268285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0268606Z return mod(**inputs) 2025-08-14T21:45:08.0268969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0269367Z outputs = self.electra( 2025-08-14T21:45:08.0269742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0270138Z hidden_states = self.encoder( 2025-08-14T21:45:08.0270511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0270880Z layer_outputs = layer_module( 2025-08-14T21:45:08.0271210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0271544Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0271923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0272317Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0272705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0273080Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0273496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0273959Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0274390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:08.0274799Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0274939Z 2025-08-14T21:45:08.0275039Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0275383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0275697Z return mod(**inputs) 2025-08-14T21:45:08.0276075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0276460Z outputs = self.electra( 2025-08-14T21:45:08.0276828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0277216Z hidden_states = self.encoder( 2025-08-14T21:45:08.0277617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0278030Z layer_outputs = layer_module( 2025-08-14T21:45:08.0278412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0278791Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0279232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0279672Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0280086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0280801Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0281259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0281778Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0282265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:08.0282727Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:08.0283132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:08.0283513Z return self.act(input) 2025-08-14T21:45:08.0283631Z 2025-08-14T21:45:08.0283743Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0284125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0284468Z return mod(**inputs) 2025-08-14T21:45:08.0284858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0285277Z outputs = self.electra( 2025-08-14T21:45:08.0285763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0286201Z hidden_states = self.encoder( 2025-08-14T21:45:08.0286612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0287035Z layer_outputs = layer_module( 2025-08-14T21:45:08.0287401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0287772Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0288194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0288623Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0289043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0289447Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0289925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:08.0290435Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:08.0290912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:08.0291336Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0291492Z 2025-08-14T21:45:08.0291607Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0291993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0292326Z return mod(**inputs) 2025-08-14T21:45:08.0292683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0293068Z outputs = self.electra( 2025-08-14T21:45:08.0293438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0293837Z hidden_states = self.encoder( 2025-08-14T21:45:08.0294211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0294583Z layer_outputs = layer_module( 2025-08-14T21:45:08.0294907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0295242Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0295658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0296048Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0296425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0296795Z return func(*args, **kwargs) 2025-08-14T21:45:08.0297187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0297564Z self_outputs = self.self( 2025-08-14T21:45:08.0297905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0298258Z return func(*args, **kwargs) 2025-08-14T21:45:08.0298623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:08.0299004Z query_layer = self.query(hidden_states) 2025-08-14T21:45:08.0299145Z 2025-08-14T21:45:08.0299243Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0299590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0299900Z return mod(**inputs) 2025-08-14T21:45:08.0300260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0300640Z outputs = self.electra( 2025-08-14T21:45:08.0300997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0301373Z hidden_states = self.encoder( 2025-08-14T21:45:08.0301730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0302102Z layer_outputs = layer_module( 2025-08-14T21:45:08.0302427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0302762Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0303141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0303541Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0303902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0304248Z return func(*args, **kwargs) 2025-08-14T21:45:08.0304604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0304972Z self_outputs = self.self( 2025-08-14T21:45:08.0305301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0305661Z return func(*args, **kwargs) 2025-08-14T21:45:08.0306026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:08.0306412Z key_layer = self.key(current_states) 2025-08-14T21:45:08.0306549Z 2025-08-14T21:45:08.0306650Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0307013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0307318Z return mod(**inputs) 2025-08-14T21:45:08.0307664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0308043Z outputs = self.electra( 2025-08-14T21:45:08.0308407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0308784Z hidden_states = self.encoder( 2025-08-14T21:45:08.0309167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0309549Z layer_outputs = layer_module( 2025-08-14T21:45:08.0309902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0310262Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0310642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0311033Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0311402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0311761Z return func(*args, **kwargs) 2025-08-14T21:45:08.0312131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0312517Z self_outputs = self.self( 2025-08-14T21:45:08.0312867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0313219Z return func(*args, **kwargs) 2025-08-14T21:45:08.0313595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:08.0313993Z value_layer = self.value(current_states) 2025-08-14T21:45:08.0314124Z 2025-08-14T21:45:08.0314203Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0314414Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0314643Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0314993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0315308Z return mod(**inputs) 2025-08-14T21:45:08.0315677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0316064Z outputs = self.electra( 2025-08-14T21:45:08.0316429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0316822Z hidden_states = self.encoder( 2025-08-14T21:45:08.0317258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0317647Z layer_outputs = layer_module( 2025-08-14T21:45:08.0317974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0318323Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0318705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0319091Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0319452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0319812Z return func(*args, **kwargs) 2025-08-14T21:45:08.0320177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:08.0320632Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:08.0321066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:08.0321466Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0321599Z 2025-08-14T21:45:08.0321707Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0322052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0322378Z return mod(**inputs) 2025-08-14T21:45:08.0322758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0323137Z outputs = self.electra( 2025-08-14T21:45:08.0323516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0323907Z hidden_states = self.encoder( 2025-08-14T21:45:08.0324294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0324682Z layer_outputs = layer_module( 2025-08-14T21:45:08.0325026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0325385Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0325884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0326302Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0326709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0327096Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0327513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0327989Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0328430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:08.0328833Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0328969Z 2025-08-14T21:45:08.0329073Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0329432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0329763Z return mod(**inputs) 2025-08-14T21:45:08.0330135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0330520Z outputs = self.electra( 2025-08-14T21:45:08.0330896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0331318Z hidden_states = self.encoder( 2025-08-14T21:45:08.0331696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0332091Z layer_outputs = layer_module( 2025-08-14T21:45:08.0332433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0332800Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0333199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0333611Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0334011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0334398Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0334835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0335331Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0335769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:08.0336191Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:08.0336570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:08.0336935Z return self.act(input) 2025-08-14T21:45:08.0337043Z 2025-08-14T21:45:08.0337149Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0337489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0337981Z return mod(**inputs) 2025-08-14T21:45:08.0338369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0338763Z outputs = self.electra( 2025-08-14T21:45:08.0339143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0339537Z hidden_states = self.encoder( 2025-08-14T21:45:08.0339920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0340307Z layer_outputs = layer_module( 2025-08-14T21:45:08.0340666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0341019Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0341402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0341804Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0342193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0342577Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0342978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:08.0343452Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:08.0343889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:08.0344284Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0344419Z 2025-08-14T21:45:08.0344521Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0344874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0345222Z return mod(**inputs) 2025-08-14T21:45:08.0345568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0345945Z outputs = self.electra( 2025-08-14T21:45:08.0346303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0346670Z hidden_states = self.encoder( 2025-08-14T21:45:08.0347026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0347399Z layer_outputs = layer_module( 2025-08-14T21:45:08.0347727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0348070Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0348445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0348880Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0349242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0349592Z return func(*args, **kwargs) 2025-08-14T21:45:08.0349950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0350325Z self_outputs = self.self( 2025-08-14T21:45:08.0350691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0351041Z return func(*args, **kwargs) 2025-08-14T21:45:08.0351419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:08.0351807Z query_layer = self.query(hidden_states) 2025-08-14T21:45:08.0351937Z 2025-08-14T21:45:08.0352045Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0352379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0352693Z return mod(**inputs) 2025-08-14T21:45:08.0353049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0353417Z outputs = self.electra( 2025-08-14T21:45:08.0353773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0354154Z hidden_states = self.encoder( 2025-08-14T21:45:08.0354521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0354894Z layer_outputs = layer_module( 2025-08-14T21:45:08.0355229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0355570Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0355936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0356321Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0356686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0357042Z return func(*args, **kwargs) 2025-08-14T21:45:08.0357399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0357790Z self_outputs = self.self( 2025-08-14T21:45:08.0358124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0358462Z return func(*args, **kwargs) 2025-08-14T21:45:08.0358832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:08.0359203Z key_layer = self.key(current_states) 2025-08-14T21:45:08.0359328Z 2025-08-14T21:45:08.0359428Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0359750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0360048Z return mod(**inputs) 2025-08-14T21:45:08.0360394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0360760Z outputs = self.electra( 2025-08-14T21:45:08.0361098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0361456Z hidden_states = self.encoder( 2025-08-14T21:45:08.0361820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0362212Z layer_outputs = layer_module( 2025-08-14T21:45:08.0362545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0362894Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0363276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0363656Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0364038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0364404Z return func(*args, **kwargs) 2025-08-14T21:45:08.0364787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0365174Z self_outputs = self.self( 2025-08-14T21:45:08.0365547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0366032Z return func(*args, **kwargs) 2025-08-14T21:45:08.0366432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:08.0366844Z value_layer = self.value(current_states) 2025-08-14T21:45:08.0366972Z 2025-08-14T21:45:08.0367059Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0367260Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0367493Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0367827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0368129Z return mod(**inputs) 2025-08-14T21:45:08.0368468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0368839Z outputs = self.electra( 2025-08-14T21:45:08.0369192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0369557Z hidden_states = self.encoder( 2025-08-14T21:45:08.0369904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0370266Z layer_outputs = layer_module( 2025-08-14T21:45:08.0370589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0370914Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0371279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0371661Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0372017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0372401Z return func(*args, **kwargs) 2025-08-14T21:45:08.0372770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:08.0373207Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:08.0373631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:08.0374021Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0374155Z 2025-08-14T21:45:08.0374253Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0374603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0374918Z return mod(**inputs) 2025-08-14T21:45:08.0375284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0376583Z outputs = self.electra( 2025-08-14T21:45:08.0376957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0377332Z hidden_states = self.encoder( 2025-08-14T21:45:08.0377704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0378086Z layer_outputs = layer_module( 2025-08-14T21:45:08.0378434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0378794Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0379203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0379603Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0379987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0380370Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0380778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0381236Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0381658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:08.0382055Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0382189Z 2025-08-14T21:45:08.0382296Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0382640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0382970Z return mod(**inputs) 2025-08-14T21:45:08.0383330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0383706Z outputs = self.electra( 2025-08-14T21:45:08.0384054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0384431Z hidden_states = self.encoder( 2025-08-14T21:45:08.0384788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0385151Z layer_outputs = layer_module( 2025-08-14T21:45:08.0385481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0385823Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0386199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0386598Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0386978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0387369Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0387762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0388194Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0388599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:08.0388998Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:08.0389350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:08.0389682Z return self.act(input) 2025-08-14T21:45:08.0389792Z 2025-08-14T21:45:08.0389907Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0390237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0390532Z return mod(**inputs) 2025-08-14T21:45:08.0390885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0391262Z outputs = self.electra( 2025-08-14T21:45:08.0391614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0392016Z hidden_states = self.encoder( 2025-08-14T21:45:08.0392391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0392770Z layer_outputs = layer_module( 2025-08-14T21:45:08.0393118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0393467Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0393849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0394237Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0394598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0394958Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0395347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:08.0395784Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:08.0396201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:08.0396623Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0396753Z 2025-08-14T21:45:08.0396858Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0397189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0397497Z return mod(**inputs) 2025-08-14T21:45:08.0397848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0398220Z outputs = self.electra( 2025-08-14T21:45:08.0398565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0398939Z hidden_states = self.encoder( 2025-08-14T21:45:08.0399304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0399673Z layer_outputs = layer_module( 2025-08-14T21:45:08.0400037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0400388Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0400768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0401149Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0401513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0401882Z return func(*args, **kwargs) 2025-08-14T21:45:08.0402262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0402663Z self_outputs = self.self( 2025-08-14T21:45:08.0403039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0403484Z return func(*args, **kwargs) 2025-08-14T21:45:08.0403878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:08.0404311Z query_layer = self.query(hidden_states) 2025-08-14T21:45:08.0404447Z 2025-08-14T21:45:08.0404559Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0404916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0405235Z return mod(**inputs) 2025-08-14T21:45:08.0405726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0406163Z outputs = self.electra( 2025-08-14T21:45:08.0406581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0406989Z hidden_states = self.encoder( 2025-08-14T21:45:08.0407359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0407731Z layer_outputs = layer_module( 2025-08-14T21:45:08.0408052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0408395Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0408781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0409164Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0409530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0409881Z return func(*args, **kwargs) 2025-08-14T21:45:08.0410248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0410619Z self_outputs = self.self( 2025-08-14T21:45:08.0410961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0411316Z return func(*args, **kwargs) 2025-08-14T21:45:08.0411679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:08.0412052Z key_layer = self.key(current_states) 2025-08-14T21:45:08.0412185Z 2025-08-14T21:45:08.0412282Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0412623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0412928Z return mod(**inputs) 2025-08-14T21:45:08.0413283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0413656Z outputs = self.electra( 2025-08-14T21:45:08.0414033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0414403Z hidden_states = self.encoder( 2025-08-14T21:45:08.0414778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0415167Z layer_outputs = layer_module( 2025-08-14T21:45:08.0415497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0415856Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0416256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0416649Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0417011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0417409Z return func(*args, **kwargs) 2025-08-14T21:45:08.0417777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0418152Z self_outputs = self.self( 2025-08-14T21:45:08.0418491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0418851Z return func(*args, **kwargs) 2025-08-14T21:45:08.0419221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:08.0419620Z value_layer = self.value(current_states) 2025-08-14T21:45:08.0419763Z 2025-08-14T21:45:08.0419843Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0420052Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0420299Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0420647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0420972Z return mod(**inputs) 2025-08-14T21:45:08.0421341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0421717Z outputs = self.electra( 2025-08-14T21:45:08.0422088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0422467Z hidden_states = self.encoder( 2025-08-14T21:45:08.0422842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0423216Z layer_outputs = layer_module( 2025-08-14T21:45:08.0423553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0423904Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0424283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0424676Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0425044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0425401Z return func(*args, **kwargs) 2025-08-14T21:45:08.0425766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:08.0426205Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:08.0426639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:08.0427034Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0427178Z 2025-08-14T21:45:08.0427272Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0427628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0427927Z return mod(**inputs) 2025-08-14T21:45:08.0428265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0428627Z outputs = self.electra( 2025-08-14T21:45:08.0428973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0429334Z hidden_states = self.encoder( 2025-08-14T21:45:08.0429685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0430051Z layer_outputs = layer_module( 2025-08-14T21:45:08.0430372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0430705Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0431096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0431480Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0431855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0432219Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0432633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0433082Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0433521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:08.0433893Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0434031Z 2025-08-14T21:45:08.0434128Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0434473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0434783Z return mod(**inputs) 2025-08-14T21:45:08.0435128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0435496Z outputs = self.electra( 2025-08-14T21:45:08.0435840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0436213Z hidden_states = self.encoder( 2025-08-14T21:45:08.0436566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0436931Z layer_outputs = layer_module( 2025-08-14T21:45:08.0437240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0437578Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0438075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0438454Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0438827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0439211Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0439607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0440049Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0440450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:08.0440895Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:08.0441245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:08.0441559Z return self.act(input) 2025-08-14T21:45:08.0441671Z 2025-08-14T21:45:08.0441768Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0442114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0442431Z return mod(**inputs) 2025-08-14T21:45:08.0442789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0443177Z outputs = self.electra( 2025-08-14T21:45:08.0443543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0443927Z hidden_states = self.encoder( 2025-08-14T21:45:08.0444306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0444715Z layer_outputs = layer_module( 2025-08-14T21:45:08.0445048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0445390Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0445847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0446267Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0446704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0447078Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0447523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:08.0448005Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:08.0448441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:08.0448833Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0448971Z 2025-08-14T21:45:08.0449072Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0449417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0449726Z return mod(**inputs) 2025-08-14T21:45:08.0450088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0450466Z outputs = self.electra( 2025-08-14T21:45:08.0450832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0451209Z hidden_states = self.encoder( 2025-08-14T21:45:08.0451587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0451981Z layer_outputs = layer_module( 2025-08-14T21:45:08.0452307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0452659Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0453039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0453432Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0453794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0454161Z return func(*args, **kwargs) 2025-08-14T21:45:08.0454530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0454946Z self_outputs = self.self( 2025-08-14T21:45:08.0455297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0455677Z return func(*args, **kwargs) 2025-08-14T21:45:08.0456055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:08.0456439Z query_layer = self.query(hidden_states) 2025-08-14T21:45:08.0456571Z 2025-08-14T21:45:08.0456667Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0456999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0457299Z return mod(**inputs) 2025-08-14T21:45:08.0457636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0458020Z outputs = self.electra( 2025-08-14T21:45:08.0458367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0458720Z hidden_states = self.encoder( 2025-08-14T21:45:08.0459075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0459435Z layer_outputs = layer_module( 2025-08-14T21:45:08.0459751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0460092Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0460461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0460849Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0461207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0461544Z return func(*args, **kwargs) 2025-08-14T21:45:08.0461890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0462248Z self_outputs = self.self( 2025-08-14T21:45:08.0462581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0462931Z return func(*args, **kwargs) 2025-08-14T21:45:08.0463287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:08.0463666Z key_layer = self.key(current_states) 2025-08-14T21:45:08.0463786Z 2025-08-14T21:45:08.0463881Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0464208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0464505Z return mod(**inputs) 2025-08-14T21:45:08.0464839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0465201Z outputs = self.electra( 2025-08-14T21:45:08.0465543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0465897Z hidden_states = self.encoder( 2025-08-14T21:45:08.0466241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0466595Z layer_outputs = layer_module( 2025-08-14T21:45:08.0466911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0467233Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0467601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0467996Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0468352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0468693Z return func(*args, **kwargs) 2025-08-14T21:45:08.0469047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0469412Z self_outputs = self.self( 2025-08-14T21:45:08.0469746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0470086Z return func(*args, **kwargs) 2025-08-14T21:45:08.0470441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:08.0470818Z value_layer = self.value(current_states) 2025-08-14T21:45:08.0470960Z 2025-08-14T21:45:08.0471036Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0471234Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0471457Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0471805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0472173Z return mod(**inputs) 2025-08-14T21:45:08.0472523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0472922Z outputs = self.electra( 2025-08-14T21:45:08.0473299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0473681Z hidden_states = self.encoder( 2025-08-14T21:45:08.0474064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0474452Z layer_outputs = layer_module( 2025-08-14T21:45:08.0474777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0475129Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0475514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0475903Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0476281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0476647Z return func(*args, **kwargs) 2025-08-14T21:45:08.0477020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:08.0477455Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:08.0477895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:08.0478293Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0478427Z 2025-08-14T21:45:08.0478534Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0478884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0479202Z return mod(**inputs) 2025-08-14T21:45:08.0479561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0479940Z outputs = self.electra( 2025-08-14T21:45:08.0480302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0480683Z hidden_states = self.encoder( 2025-08-14T21:45:08.0481057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0481457Z layer_outputs = layer_module( 2025-08-14T21:45:08.0481788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0482137Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0482528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0482932Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0483332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0483726Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0484135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0484593Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0485077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:08.0485478Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0485698Z 2025-08-14T21:45:08.0485807Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0486173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0486508Z return mod(**inputs) 2025-08-14T21:45:08.0486885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0487272Z outputs = self.electra( 2025-08-14T21:45:08.0487653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0488041Z hidden_states = self.encoder( 2025-08-14T21:45:08.0488411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0488800Z layer_outputs = layer_module( 2025-08-14T21:45:08.0489137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0489489Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0489947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0490350Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0490740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0491115Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0491531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0491992Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0492447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:08.0492903Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:08.0493300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:08.0493678Z return self.act(input) 2025-08-14T21:45:08.0493797Z 2025-08-14T21:45:08.0493915Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0494287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0494634Z return mod(**inputs) 2025-08-14T21:45:08.0495007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0495409Z outputs = self.electra( 2025-08-14T21:45:08.0495760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0496125Z hidden_states = self.encoder( 2025-08-14T21:45:08.0496480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0496863Z layer_outputs = layer_module( 2025-08-14T21:45:08.0497210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0497576Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0497976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0498387Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0498778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0499156Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0499536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:08.0499974Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:08.0500392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:08.0500773Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0500936Z 2025-08-14T21:45:08.0501046Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0501441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0501820Z return mod(**inputs) 2025-08-14T21:45:08.0502217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0502648Z outputs = self.electra( 2025-08-14T21:45:08.0503055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0503487Z hidden_states = self.encoder( 2025-08-14T21:45:08.0503907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0504330Z layer_outputs = layer_module( 2025-08-14T21:45:08.0504697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0505069Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0505456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0505834Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0506193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0506535Z return func(*args, **kwargs) 2025-08-14T21:45:08.0506892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0507262Z self_outputs = self.self( 2025-08-14T21:45:08.0507599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0507942Z return func(*args, **kwargs) 2025-08-14T21:45:08.0508298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:08.0508676Z query_layer = self.query(hidden_states) 2025-08-14T21:45:08.0508803Z 2025-08-14T21:45:08.0508901Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0509271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0509596Z return mod(**inputs) 2025-08-14T21:45:08.0509968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0510353Z outputs = self.electra( 2025-08-14T21:45:08.0510723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0511117Z hidden_states = self.encoder( 2025-08-14T21:45:08.0511474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0511845Z layer_outputs = layer_module( 2025-08-14T21:45:08.0512180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0512542Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0512952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0513360Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0513740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0514093Z return func(*args, **kwargs) 2025-08-14T21:45:08.0514448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0514840Z self_outputs = self.self( 2025-08-14T21:45:08.0515188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0515540Z return func(*args, **kwargs) 2025-08-14T21:45:08.0515921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:08.0516313Z key_layer = self.key(current_states) 2025-08-14T21:45:08.0516441Z 2025-08-14T21:45:08.0516547Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0516898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0517224Z return mod(**inputs) 2025-08-14T21:45:08.0517603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0517990Z outputs = self.electra( 2025-08-14T21:45:08.0518373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0518745Z hidden_states = self.encoder( 2025-08-14T21:45:08.0519110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0519480Z layer_outputs = layer_module( 2025-08-14T21:45:08.0519814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0520168Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0520559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0520950Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0521322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0521685Z return func(*args, **kwargs) 2025-08-14T21:45:08.0522047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0522429Z self_outputs = self.self( 2025-08-14T21:45:08.0522787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0523174Z return func(*args, **kwargs) 2025-08-14T21:45:08.0523550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:08.0523938Z value_layer = self.value(current_states) 2025-08-14T21:45:08.0524073Z 2025-08-14T21:45:08.0524159Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0524358Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0524584Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0524926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0525236Z return mod(**inputs) 2025-08-14T21:45:08.0525664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0526095Z outputs = self.electra( 2025-08-14T21:45:08.0526488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0526918Z hidden_states = self.encoder( 2025-08-14T21:45:08.0527327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0527753Z layer_outputs = layer_module( 2025-08-14T21:45:08.0528116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0528488Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0528945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0529380Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0529796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0530187Z return func(*args, **kwargs) 2025-08-14T21:45:08.0530589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:08.0531069Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:08.0531561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:08.0532011Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0532169Z 2025-08-14T21:45:08.0532282Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0532680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0533015Z return mod(**inputs) 2025-08-14T21:45:08.0533407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0533821Z outputs = self.electra( 2025-08-14T21:45:08.0534205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0534594Z hidden_states = self.encoder( 2025-08-14T21:45:08.0535000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0535411Z layer_outputs = layer_module( 2025-08-14T21:45:08.0535764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0536150Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0536573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0536974Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0537365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0537947Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0538380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0538855Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0539318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:08.0539719Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0539853Z 2025-08-14T21:45:08.0539965Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0540311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0540635Z return mod(**inputs) 2025-08-14T21:45:08.0541003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0541433Z outputs = self.electra( 2025-08-14T21:45:08.0541786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0542149Z hidden_states = self.encoder( 2025-08-14T21:45:08.0542508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0542871Z layer_outputs = layer_module( 2025-08-14T21:45:08.0543203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0543570Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0543954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0544335Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0544709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0545072Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0545455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0545891Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0546293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:08.0546691Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:08.0547031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:08.0547348Z return self.act(input) 2025-08-14T21:45:08.0547450Z 2025-08-14T21:45:08.0547553Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0547886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0548181Z return mod(**inputs) 2025-08-14T21:45:08.0548523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0548884Z outputs = self.electra( 2025-08-14T21:45:08.0549219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0549580Z hidden_states = self.encoder( 2025-08-14T21:45:08.0549933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0550295Z layer_outputs = layer_module( 2025-08-14T21:45:08.0550610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0550950Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0551355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0551731Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0552108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0552480Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0552876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:08.0553327Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:08.0553753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:08.0554140Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0554269Z 2025-08-14T21:45:08.0554374Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0554734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0555042Z return mod(**inputs) 2025-08-14T21:45:08.0555397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0555761Z outputs = self.electra( 2025-08-14T21:45:08.0556118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0556506Z hidden_states = self.encoder( 2025-08-14T21:45:08.0556872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0557238Z layer_outputs = layer_module( 2025-08-14T21:45:08.0557571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0557918Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0558300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0558666Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0559016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0559359Z return func(*args, **kwargs) 2025-08-14T21:45:08.0559705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0560068Z self_outputs = self.self( 2025-08-14T21:45:08.0560401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0560744Z return func(*args, **kwargs) 2025-08-14T21:45:08.0561091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:08.0561467Z query_layer = self.query(hidden_states) 2025-08-14T21:45:08.0561593Z 2025-08-14T21:45:08.0561698Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0562033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0562340Z return mod(**inputs) 2025-08-14T21:45:08.0562697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0563075Z outputs = self.electra( 2025-08-14T21:45:08.0563421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0563795Z hidden_states = self.encoder( 2025-08-14T21:45:08.0564154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0564534Z layer_outputs = layer_module( 2025-08-14T21:45:08.0564840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0565169Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0565531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0566014Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0566431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0566821Z return func(*args, **kwargs) 2025-08-14T21:45:08.0567205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0567585Z self_outputs = self.self( 2025-08-14T21:45:08.0567937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0568295Z return func(*args, **kwargs) 2025-08-14T21:45:08.0568682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:08.0569105Z key_layer = self.key(current_states) 2025-08-14T21:45:08.0569253Z 2025-08-14T21:45:08.0569362Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0569735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0570082Z return mod(**inputs) 2025-08-14T21:45:08.0570479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0570897Z outputs = self.electra( 2025-08-14T21:45:08.0571303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0571721Z hidden_states = self.encoder( 2025-08-14T21:45:08.0572125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0572537Z layer_outputs = layer_module( 2025-08-14T21:45:08.0572890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0573272Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0573689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0574112Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0574504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0574893Z return func(*args, **kwargs) 2025-08-14T21:45:08.0575293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0575697Z self_outputs = self.self( 2025-08-14T21:45:08.0576074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0576408Z return func(*args, **kwargs) 2025-08-14T21:45:08.0576751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:08.0577107Z value_layer = self.value(current_states) 2025-08-14T21:45:08.0577236Z 2025-08-14T21:45:08.0577311Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0577510Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0577718Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0578048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0578374Z return mod(**inputs) 2025-08-14T21:45:08.0578718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0579075Z outputs = self.electra( 2025-08-14T21:45:08.0579417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0579782Z hidden_states = self.encoder( 2025-08-14T21:45:08.0580126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0580489Z layer_outputs = layer_module( 2025-08-14T21:45:08.0580807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0581137Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0581498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0581899Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0582259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0582613Z return func(*args, **kwargs) 2025-08-14T21:45:08.0582962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:08.0583385Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:08.0583826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:08.0584200Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0584333Z 2025-08-14T21:45:08.0584445Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0584777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0585076Z return mod(**inputs) 2025-08-14T21:45:08.0585417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0585791Z outputs = self.electra( 2025-08-14T21:45:08.0586146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0586506Z hidden_states = self.encoder( 2025-08-14T21:45:08.0586855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0587213Z layer_outputs = layer_module( 2025-08-14T21:45:08.0587527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0587854Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0588222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0588599Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0588979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0589346Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0589746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0590195Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0590593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:08.0590965Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0591100Z 2025-08-14T21:45:08.0591194Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0591546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0591839Z return mod(**inputs) 2025-08-14T21:45:08.0592178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0592547Z outputs = self.electra( 2025-08-14T21:45:08.0592896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0593258Z hidden_states = self.encoder( 2025-08-14T21:45:08.0593620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0593988Z layer_outputs = layer_module( 2025-08-14T21:45:08.0594303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0594647Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0595039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0595422Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0595787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0596157Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0596559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0597023Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0597433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:08.0597857Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:08.0598220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:08.0598534Z return self.act(input) 2025-08-14T21:45:08.0598645Z 2025-08-14T21:45:08.0598743Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0599086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0599392Z return mod(**inputs) 2025-08-14T21:45:08.0599732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0600104Z outputs = self.electra( 2025-08-14T21:45:08.0600451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0600816Z hidden_states = self.encoder( 2025-08-14T21:45:08.0601178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0601547Z layer_outputs = layer_module( 2025-08-14T21:45:08.0601867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0602198Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0602617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0603011Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0603397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0603765Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0604172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:08.0604642Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:08.0605102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:08.0605497Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0605712Z 2025-08-14T21:45:08.0605818Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0606216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0606566Z return mod(**inputs) 2025-08-14T21:45:08.0606969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0607350Z outputs = self.electra( 2025-08-14T21:45:08.0607710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0608078Z hidden_states = self.encoder( 2025-08-14T21:45:08.0608442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0608837Z layer_outputs = layer_module( 2025-08-14T21:45:08.0609157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0609505Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0609891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0610278Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0610652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0611006Z return func(*args, **kwargs) 2025-08-14T21:45:08.0611383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0611754Z self_outputs = self.self( 2025-08-14T21:45:08.0611990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0612056Z return func(*args, **kwargs) 2025-08-14T21:45:08.0612317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:08.0612393Z query_layer = self.query(hidden_states) 2025-08-14T21:45:08.0612397Z 2025-08-14T21:45:08.0612492Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0612683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0612744Z return mod(**inputs) 2025-08-14T21:45:08.0612997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0613063Z outputs = self.electra( 2025-08-14T21:45:08.0613302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0613376Z hidden_states = self.encoder( 2025-08-14T21:45:08.0613615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0613679Z layer_outputs = layer_module( 2025-08-14T21:45:08.0613888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0613961Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0614209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0614286Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0614508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0614602Z return func(*args, **kwargs) 2025-08-14T21:45:08.0614839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0614903Z self_outputs = self.self( 2025-08-14T21:45:08.0615130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0615194Z return func(*args, **kwargs) 2025-08-14T21:45:08.0615441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:08.0615514Z key_layer = self.key(current_states) 2025-08-14T21:45:08.0615518Z 2025-08-14T21:45:08.0615613Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0615802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0615862Z return mod(**inputs) 2025-08-14T21:45:08.0616109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0616189Z outputs = self.electra( 2025-08-14T21:45:08.0616427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0616498Z hidden_states = self.encoder( 2025-08-14T21:45:08.0616735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0616800Z layer_outputs = layer_module( 2025-08-14T21:45:08.0617024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0617095Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0617354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0617435Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0617656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0617727Z return func(*args, **kwargs) 2025-08-14T21:45:08.0617966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0618029Z self_outputs = self.self( 2025-08-14T21:45:08.0618252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0618317Z return func(*args, **kwargs) 2025-08-14T21:45:08.0618567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:08.0618642Z value_layer = self.value(current_states) 2025-08-14T21:45:08.0618645Z 2025-08-14T21:45:08.0618721Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0618804Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0618897Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0619089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0619151Z return mod(**inputs) 2025-08-14T21:45:08.0619396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0619467Z outputs = self.electra( 2025-08-14T21:45:08.0619709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0619773Z hidden_states = self.encoder( 2025-08-14T21:45:08.0620024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0620088Z layer_outputs = layer_module( 2025-08-14T21:45:08.0620319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0620391Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0620626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0620708Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0620927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0620991Z return func(*args, **kwargs) 2025-08-14T21:45:08.0621230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:08.0621347Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:08.0621590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:08.0621684Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0621688Z 2025-08-14T21:45:08.0621781Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0621970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0622032Z return mod(**inputs) 2025-08-14T21:45:08.0622279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0622342Z outputs = self.electra( 2025-08-14T21:45:08.0622596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0622671Z hidden_states = self.encoder( 2025-08-14T21:45:08.0622929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0622997Z layer_outputs = layer_module( 2025-08-14T21:45:08.0623208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0623278Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0623521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0623597Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0623830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0623909Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0624183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0624300Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0624538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:08.0624614Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0624617Z 2025-08-14T21:45:08.0624715Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0624896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0624956Z return mod(**inputs) 2025-08-14T21:45:08.0625202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0625266Z outputs = self.electra( 2025-08-14T21:45:08.0625509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0625575Z hidden_states = self.encoder( 2025-08-14T21:45:08.0625816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0625907Z layer_outputs = layer_module( 2025-08-14T21:45:08.0626111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0626190Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0626426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0626502Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0626745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0626817Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0627085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0627204Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0627456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:08.0627566Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:08.0627757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:08.0627823Z return self.act(input) 2025-08-14T21:45:08.0627826Z 2025-08-14T21:45:08.0627928Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0628127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0628195Z return mod(**inputs) 2025-08-14T21:45:08.0628459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0628524Z outputs = self.electra( 2025-08-14T21:45:08.0628769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0628836Z hidden_states = self.encoder( 2025-08-14T21:45:08.0629073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0629145Z layer_outputs = layer_module( 2025-08-14T21:45:08.0629342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0629417Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0629654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0629730Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0629969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0630039Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0630303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:08.0630430Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:08.0630665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:08.0630746Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0630749Z 2025-08-14T21:45:08.0630842Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0631024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0631090Z return mod(**inputs) 2025-08-14T21:45:08.0631329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0631417Z outputs = self.electra( 2025-08-14T21:45:08.0631659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0631723Z hidden_states = self.encoder( 2025-08-14T21:45:08.0631973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0632037Z layer_outputs = layer_module( 2025-08-14T21:45:08.0632243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0632322Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0632563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0632644Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0632867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0632949Z return func(*args, **kwargs) 2025-08-14T21:45:08.0633192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0633257Z self_outputs = self.self( 2025-08-14T21:45:08.0633484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0633548Z return func(*args, **kwargs) 2025-08-14T21:45:08.0633806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:08.0633892Z query_layer = self.query(hidden_states) 2025-08-14T21:45:08.0633896Z 2025-08-14T21:45:08.0633989Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0634188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0634260Z return mod(**inputs) 2025-08-14T21:45:08.0634502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0634571Z outputs = self.electra( 2025-08-14T21:45:08.0634808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0634872Z hidden_states = self.encoder( 2025-08-14T21:45:08.0635118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0635183Z layer_outputs = layer_module( 2025-08-14T21:45:08.0635395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0635467Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0635703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0635788Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0636006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0636069Z return func(*args, **kwargs) 2025-08-14T21:45:08.0636312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0636378Z self_outputs = self.self( 2025-08-14T21:45:08.0636610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0636675Z return func(*args, **kwargs) 2025-08-14T21:45:08.0636918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:08.0636999Z key_layer = self.key(current_states) 2025-08-14T21:45:08.0637026Z 2025-08-14T21:45:08.0637126Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0637312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0637382Z return mod(**inputs) 2025-08-14T21:45:08.0637760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0637853Z outputs = self.electra( 2025-08-14T21:45:08.0638129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0638218Z hidden_states = self.encoder( 2025-08-14T21:45:08.0638474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0638544Z layer_outputs = layer_module( 2025-08-14T21:45:08.0638761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0638879Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0639132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0639221Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0639461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0639527Z return func(*args, **kwargs) 2025-08-14T21:45:08.0639809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0639875Z self_outputs = self.self( 2025-08-14T21:45:08.0640138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0640206Z return func(*args, **kwargs) 2025-08-14T21:45:08.0640458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:08.0640543Z value_layer = self.value(current_states) 2025-08-14T21:45:08.0640547Z 2025-08-14T21:45:08.0640623Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0640697Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0640803Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0640991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0641060Z return mod(**inputs) 2025-08-14T21:45:08.0641314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0641379Z outputs = self.electra( 2025-08-14T21:45:08.0641642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0641713Z hidden_states = self.encoder( 2025-08-14T21:45:08.0641967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0642042Z layer_outputs = layer_module( 2025-08-14T21:45:08.0642255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0642335Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0642587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0642666Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0642906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0642973Z return func(*args, **kwargs) 2025-08-14T21:45:08.0643242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:08.0643397Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:08.0643686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:08.0643773Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0643776Z 2025-08-14T21:45:08.0643876Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0644073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0644137Z return mod(**inputs) 2025-08-14T21:45:08.0644392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0644466Z outputs = self.electra( 2025-08-14T21:45:08.0644717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0644802Z hidden_states = self.encoder( 2025-08-14T21:45:08.0645064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0645131Z layer_outputs = layer_module( 2025-08-14T21:45:08.0645351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0645423Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0645746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0645847Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0646110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0646185Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0646475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0646588Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0646852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:08.0646929Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0646932Z 2025-08-14T21:45:08.0647028Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0647222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0647285Z return mod(**inputs) 2025-08-14T21:45:08.0647538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0647602Z outputs = self.electra( 2025-08-14T21:45:08.0647844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0647917Z hidden_states = self.encoder( 2025-08-14T21:45:08.0648159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0648226Z layer_outputs = layer_module( 2025-08-14T21:45:08.0648438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0648509Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0648760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0648837Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0649079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0649204Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0649479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0649597Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0649841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:08.0649945Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:08.0650151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:08.0650217Z return self.act(input) 2025-08-14T21:45:08.0650221Z 2025-08-14T21:45:08.0650317Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0650512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0650594Z return mod(**inputs) 2025-08-14T21:45:08.0650853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0650917Z outputs = self.electra( 2025-08-14T21:45:08.0651164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0651240Z hidden_states = self.encoder( 2025-08-14T21:45:08.0651493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0651595Z layer_outputs = layer_module( 2025-08-14T21:45:08.0651809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0651899Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0652158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0652242Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0652487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0652583Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0652864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:08.0652993Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:08.0653245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:08.0653324Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0653327Z 2025-08-14T21:45:08.0653431Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0653618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0653690Z return mod(**inputs) 2025-08-14T21:45:08.0653934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0653999Z outputs = self.electra( 2025-08-14T21:45:08.0654246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0654311Z hidden_states = self.encoder( 2025-08-14T21:45:08.0654554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0654627Z layer_outputs = layer_module( 2025-08-14T21:45:08.0654834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0654914Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0655176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0655252Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0655483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0655548Z return func(*args, **kwargs) 2025-08-14T21:45:08.0655791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0655864Z self_outputs = self.self( 2025-08-14T21:45:08.0656089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0656163Z return func(*args, **kwargs) 2025-08-14T21:45:08.0656409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:08.0656506Z query_layer = self.query(hidden_states) 2025-08-14T21:45:08.0656510Z 2025-08-14T21:45:08.0656624Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0656806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0656872Z return mod(**inputs) 2025-08-14T21:45:08.0657109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0657170Z outputs = self.electra( 2025-08-14T21:45:08.0657424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0657489Z hidden_states = self.encoder( 2025-08-14T21:45:08.0657745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0657819Z layer_outputs = layer_module( 2025-08-14T21:45:08.0658023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0658100Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0658337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0658412Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0658635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0658699Z return func(*args, **kwargs) 2025-08-14T21:45:08.0658939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0659011Z self_outputs = self.self( 2025-08-14T21:45:08.0659229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0659300Z return func(*args, **kwargs) 2025-08-14T21:45:08.0659537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:08.0659608Z key_layer = self.key(current_states) 2025-08-14T21:45:08.0659611Z 2025-08-14T21:45:08.0659711Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0659895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0659962Z return mod(**inputs) 2025-08-14T21:45:08.0660204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0660266Z outputs = self.electra( 2025-08-14T21:45:08.0660508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0660592Z hidden_states = self.encoder( 2025-08-14T21:45:08.0660834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0660906Z layer_outputs = layer_module( 2025-08-14T21:45:08.0661109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0661186Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0661426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0661504Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0661732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0661797Z return func(*args, **kwargs) 2025-08-14T21:45:08.0662042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0662124Z self_outputs = self.self( 2025-08-14T21:45:08.0662348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0662421Z return func(*args, **kwargs) 2025-08-14T21:45:08.0662670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:08.0662747Z value_layer = self.value(current_states) 2025-08-14T21:45:08.0662751Z 2025-08-14T21:45:08.0662836Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0662933Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0663038Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0663225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0663302Z return mod(**inputs) 2025-08-14T21:45:08.0663559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0663627Z outputs = self.electra( 2025-08-14T21:45:08.0663877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0663949Z hidden_states = self.encoder( 2025-08-14T21:45:08.0664186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0664258Z layer_outputs = layer_module( 2025-08-14T21:45:08.0664463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0664534Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0664783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0664858Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0665079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0665150Z return func(*args, **kwargs) 2025-08-14T21:45:08.0665388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:08.0665510Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:08.0665747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:08.0665824Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0665827Z 2025-08-14T21:45:08.0665929Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0666113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0666179Z return mod(**inputs) 2025-08-14T21:45:08.0666441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0666504Z outputs = self.electra( 2025-08-14T21:45:08.0666747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0666810Z hidden_states = self.encoder( 2025-08-14T21:45:08.0667047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0667117Z layer_outputs = layer_module( 2025-08-14T21:45:08.0667320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0667395Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0667636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0667731Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0667973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0668043Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0668319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0668427Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0668678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:08.0668762Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0668766Z 2025-08-14T21:45:08.0668858Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0669063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0669128Z return mod(**inputs) 2025-08-14T21:45:08.0669371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0669441Z outputs = self.electra( 2025-08-14T21:45:08.0669679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0669743Z hidden_states = self.encoder( 2025-08-14T21:45:08.0669985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0670050Z layer_outputs = layer_module( 2025-08-14T21:45:08.0670261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0670334Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0670570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0670654Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0670888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0670956Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0671230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0671338Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0671580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:08.0671682Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:08.0671874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:08.0671971Z return self.act(input) 2025-08-14T21:45:08.0671974Z 2025-08-14T21:45:08.0672069Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0672254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0672314Z return mod(**inputs) 2025-08-14T21:45:08.0672551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0672619Z outputs = self.electra( 2025-08-14T21:45:08.0672854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0672918Z hidden_states = self.encoder( 2025-08-14T21:45:08.0673164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0673227Z layer_outputs = layer_module( 2025-08-14T21:45:08.0673461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0673533Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0673768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0673848Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0674078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0674153Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0674430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:08.0674566Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:08.0674815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:08.0674897Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0674900Z 2025-08-14T21:45:08.0674994Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0675189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0675249Z return mod(**inputs) 2025-08-14T21:45:08.0675506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0675567Z outputs = self.electra( 2025-08-14T21:45:08.0675804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0675875Z hidden_states = self.encoder( 2025-08-14T21:45:08.0676113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0676188Z layer_outputs = layer_module( 2025-08-14T21:45:08.0676388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0676458Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0676699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0676773Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0676991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0677063Z return func(*args, **kwargs) 2025-08-14T21:45:08.0677299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0677370Z self_outputs = self.self( 2025-08-14T21:45:08.0677588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0677672Z return func(*args, **kwargs) 2025-08-14T21:45:08.0677916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:08.0677989Z query_layer = self.query(hidden_states) 2025-08-14T21:45:08.0677992Z 2025-08-14T21:45:08.0678085Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0678273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0678333Z return mod(**inputs) 2025-08-14T21:45:08.0678581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0678646Z outputs = self.electra( 2025-08-14T21:45:08.0678881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0678971Z hidden_states = self.encoder( 2025-08-14T21:45:08.0679214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0679287Z layer_outputs = layer_module( 2025-08-14T21:45:08.0679491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0679563Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0679825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0679901Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0680118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0680207Z return func(*args, **kwargs) 2025-08-14T21:45:08.0680446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0680522Z self_outputs = self.self( 2025-08-14T21:45:08.0680737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0680800Z return func(*args, **kwargs) 2025-08-14T21:45:08.0681042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:08.0681114Z key_layer = self.key(current_states) 2025-08-14T21:45:08.0681118Z 2025-08-14T21:45:08.0681218Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0681405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0681467Z return mod(**inputs) 2025-08-14T21:45:08.0681727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0681794Z outputs = self.electra( 2025-08-14T21:45:08.0682037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0682110Z hidden_states = self.encoder( 2025-08-14T21:45:08.0682353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0682428Z layer_outputs = layer_module( 2025-08-14T21:45:08.0682637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0682718Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0682963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0683039Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0683278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0683352Z return func(*args, **kwargs) 2025-08-14T21:45:08.0683596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0683666Z self_outputs = self.self( 2025-08-14T21:45:08.0683890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0683955Z return func(*args, **kwargs) 2025-08-14T21:45:08.0684209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:08.0684283Z value_layer = self.value(current_states) 2025-08-14T21:45:08.0684286Z 2025-08-14T21:45:08.0684368Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0684444Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0684541Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0684751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0684812Z return mod(**inputs) 2025-08-14T21:45:08.0685061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0685131Z outputs = self.electra( 2025-08-14T21:45:08.0685378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0685466Z hidden_states = self.encoder( 2025-08-14T21:45:08.0685796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0685873Z layer_outputs = layer_module( 2025-08-14T21:45:08.0686123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0686204Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0686464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0686551Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0686801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0686875Z return func(*args, **kwargs) 2025-08-14T21:45:08.0687118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:08.0687237Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:08.0687490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:08.0687569Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0687575Z 2025-08-14T21:45:08.0687679Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0687864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0687926Z return mod(**inputs) 2025-08-14T21:45:08.0688176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0688239Z outputs = self.electra( 2025-08-14T21:45:08.0688482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0688554Z hidden_states = self.encoder( 2025-08-14T21:45:08.0688798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0688873Z layer_outputs = layer_module( 2025-08-14T21:45:08.0689077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0689168Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0689419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0689497Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0689738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0689815Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0690089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0690206Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0690451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:08.0690548Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0690553Z 2025-08-14T21:45:08.0690655Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0690843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0690913Z return mod(**inputs) 2025-08-14T21:45:08.0691157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0691220Z outputs = self.electra( 2025-08-14T21:45:08.0691488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0691557Z hidden_states = self.encoder( 2025-08-14T21:45:08.0691821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0691898Z layer_outputs = layer_module( 2025-08-14T21:45:08.0692105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0692186Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0692429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0692506Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0692756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0692830Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0693111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0693221Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0693467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:08.0693582Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:08.0693777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:08.0693843Z return self.act(input) 2025-08-14T21:45:08.0693854Z 2025-08-14T21:45:08.0693949Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0694134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0694204Z return mod(**inputs) 2025-08-14T21:45:08.0694450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0694514Z outputs = self.electra( 2025-08-14T21:45:08.0694762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0694856Z hidden_states = self.encoder( 2025-08-14T21:45:08.0695105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0695171Z layer_outputs = layer_module( 2025-08-14T21:45:08.0695378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0695460Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0695704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0695782Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0696030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0696102Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0696385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:08.0696526Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:08.0696771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:08.0696856Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0696859Z 2025-08-14T21:45:08.0696955Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0697194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0697257Z return mod(**inputs) 2025-08-14T21:45:08.0697505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0697590Z outputs = self.electra( 2025-08-14T21:45:08.0697836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0697906Z hidden_states = self.encoder( 2025-08-14T21:45:08.0698159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0698226Z layer_outputs = layer_module( 2025-08-14T21:45:08.0698439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0698514Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0698760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0698846Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0699072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0699140Z return func(*args, **kwargs) 2025-08-14T21:45:08.0699393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0699459Z self_outputs = self.self( 2025-08-14T21:45:08.0699692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0699757Z return func(*args, **kwargs) 2025-08-14T21:45:08.0700002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:08.0700089Z query_layer = self.query(hidden_states) 2025-08-14T21:45:08.0700092Z 2025-08-14T21:45:08.0700188Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0700378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0700440Z return mod(**inputs) 2025-08-14T21:45:08.0700704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0700776Z outputs = self.electra( 2025-08-14T21:45:08.0701015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0701080Z hidden_states = self.encoder( 2025-08-14T21:45:08.0701328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0701393Z layer_outputs = layer_module( 2025-08-14T21:45:08.0701607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0701679Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0701924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0702004Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0702250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0702320Z return func(*args, **kwargs) 2025-08-14T21:45:08.0702564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0702629Z self_outputs = self.self( 2025-08-14T21:45:08.0702860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0702939Z return func(*args, **kwargs) 2025-08-14T21:45:08.0703184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:08.0703266Z key_layer = self.key(current_states) 2025-08-14T21:45:08.0703288Z 2025-08-14T21:45:08.0703384Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0703579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0703642Z return mod(**inputs) 2025-08-14T21:45:08.0703893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0703964Z outputs = self.electra( 2025-08-14T21:45:08.0704211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0704275Z hidden_states = self.encoder( 2025-08-14T21:45:08.0704533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0704599Z layer_outputs = layer_module( 2025-08-14T21:45:08.0704814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0704888Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0705137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0705222Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0705451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0705522Z return func(*args, **kwargs) 2025-08-14T21:45:08.0705774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:08.0705841Z self_outputs = self.self( 2025-08-14T21:45:08.0706078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0706143Z return func(*args, **kwargs) 2025-08-14T21:45:08.0706397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:08.0706500Z value_layer = self.value(current_states) 2025-08-14T21:45:08.0706503Z 2025-08-14T21:45:08.0706578Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0706658Z cudagraph partition due to non gpu ops 2025-08-14T21:45:08.0706752Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0706934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0707006Z return mod(**inputs) 2025-08-14T21:45:08.0707253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0707317Z outputs = self.electra( 2025-08-14T21:45:08.0707567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0707633Z hidden_states = self.encoder( 2025-08-14T21:45:08.0707883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0707968Z layer_outputs = layer_module( 2025-08-14T21:45:08.0708173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0708254Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0708497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:08.0708580Z self_attention_outputs = self.attention( 2025-08-14T21:45:08.0708824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:08.0708899Z return func(*args, **kwargs) 2025-08-14T21:45:08.0709168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:08.0709291Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:08.0709537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:08.0709623Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0709627Z 2025-08-14T21:45:08.0709772Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0709960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0710023Z return mod(**inputs) 2025-08-14T21:45:08.0710278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0710341Z outputs = self.electra( 2025-08-14T21:45:08.0710588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0710662Z hidden_states = self.encoder( 2025-08-14T21:45:08.0710911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0710986Z layer_outputs = layer_module( 2025-08-14T21:45:08.0711195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0711266Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0711518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0711599Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0711848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0711930Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0712201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0712337Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0712579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:08.0712655Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0712658Z 2025-08-14T21:45:08.0712761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0712942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0713012Z return mod(**inputs) 2025-08-14T21:45:08.0713254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0713317Z outputs = self.electra( 2025-08-14T21:45:08.0713562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0713646Z hidden_states = self.encoder( 2025-08-14T21:45:08.0713885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0713957Z layer_outputs = layer_module( 2025-08-14T21:45:08.0714159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0714237Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0714489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0714567Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0714828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0714898Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0715175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:08.0715285Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:08.0715520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:08.0715630Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:08.0715821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:08.0715887Z return self.act(input) 2025-08-14T21:45:08.0715897Z 2025-08-14T21:45:08.0715990Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0716173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0716241Z return mod(**inputs) 2025-08-14T21:45:08.0716487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1543, in forward 2025-08-14T21:45:08.0716553Z outputs = self.electra( 2025-08-14T21:45:08.0716803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:08.0716869Z hidden_states = self.encoder( 2025-08-14T21:45:08.0717117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:08.0717182Z layer_outputs = layer_module( 2025-08-14T21:45:08.0717388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:08.0717467Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:08.0717710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:08.0717808Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:08.0718058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:08.0718129Z return forward_fn(*input_tensors) 2025-08-14T21:45:08.0718413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:08.0718539Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:08.0718785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:08.0718870Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:08.0718873Z 2025-08-14T21:45:08.0718969Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0719167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0719230Z return mod(**inputs) 2025-08-14T21:45:08.0719496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1560, in forward 2025-08-14T21:45:08.0719673Z prediction_scores = self.generator_lm_head(self.generator_predictions(sequence_output)) 2025-08-14T21:45:08.0719916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 640, in forward 2025-08-14T21:45:08.0720012Z hidden_states = self.dense(generator_hidden_states) 2025-08-14T21:45:08.0720024Z 2025-08-14T21:45:08.0720120Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0720323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0720393Z return mod(**inputs) 2025-08-14T21:45:08.0720670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1560, in forward 2025-08-14T21:45:08.0720841Z prediction_scores = self.generator_lm_head(self.generator_predictions(sequence_output)) 2025-08-14T21:45:08.0720845Z 2025-08-14T21:45:08.0720948Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:08.0721132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:08.0721202Z return mod(**inputs) 2025-08-14T21:45:08.0721448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1564, in forward 2025-08-14T21:45:08.0721517Z lm_loss = self.loss_function( 2025-08-14T21:45:08.0721751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-08-14T21:45:08.0721914Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-08-14T21:45:08.0722158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-08-14T21:45:08.0722349Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-08-14T21:45:08.0722353Z 2025-08-14T21:45:15.8046444Z Compilation time (from dynamo_timed): 14.384871842 2025-08-14T21:45:15.8159925Z pass 2025-08-14T21:45:15.8160593Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:45:15.8161516Z TIMING: _recursive_pre_grad_passes:0.00669 _recursive_joint_graph_passes:0.42897 _recursive_post_grad_passes:0.07618 async_compile.wait:0.7521 code_gen:7.15719 inductor_compile:8.26951 backend_compile:11.64017 gc:0.00018 entire_frame_compile:14.38487 total_wall_time:14.38487 2025-08-14T21:45:15.8162582Z STATS: call_* op count: 377 | FakeTensorMode.__torch_dispatch__:15041 | FakeTensor.__torch_dispatch__:4687 | ProxyTorchDispatchMode.__torch_dispatch__:5671 2025-08-14T21:45:15.8163081Z Dynamo produced 1 graphs covering 377 ops with 0 graph breaks (0 unique) 2025-08-14T21:45:20.6828144Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:45:20.6829067Z from pkg_resources import resource_filename 2025-08-14T21:45:21.3317494Z 2025-08-14T21:45:21.6849715Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:45:21.6853930Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:45:21.6860493Z cpu eval ElectraForQuestionAnswering 2025-08-14T21:45:21.8003564Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:45:21.8592881Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:45:21.9165390Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:45:29.5378965Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5379598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5380536Z return mod(**inputs) 2025-08-14T21:45:29.5381101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5381651Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5382485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 797, in forward 2025-08-14T21:45:29.5382978Z hidden_states = self.embeddings_project(hidden_states) 2025-08-14T21:45:29.5383193Z 2025-08-14T21:45:29.5383317Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5383777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5384162Z return mod(**inputs) 2025-08-14T21:45:29.5384574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5385030Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5385490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5385897Z hidden_states = self.encoder( 2025-08-14T21:45:29.5386270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5386653Z layer_outputs = layer_module( 2025-08-14T21:45:29.5386992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5387402Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5387782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5388171Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5388583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5388940Z return func(*args, **kwargs) 2025-08-14T21:45:29.5389314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5389703Z self_outputs = self.self( 2025-08-14T21:45:29.5390060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5390409Z return func(*args, **kwargs) 2025-08-14T21:45:29.5390785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:29.5391184Z query_layer = self.query(hidden_states) 2025-08-14T21:45:29.5391380Z 2025-08-14T21:45:29.5391494Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5391842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5392168Z return mod(**inputs) 2025-08-14T21:45:29.5392545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5392951Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5393355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5393755Z hidden_states = self.encoder( 2025-08-14T21:45:29.5394122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5394497Z layer_outputs = layer_module( 2025-08-14T21:45:29.5394833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5395239Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5395637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5396025Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5396386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5396738Z return func(*args, **kwargs) 2025-08-14T21:45:29.5397114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5397492Z self_outputs = self.self( 2025-08-14T21:45:29.5397863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5398213Z return func(*args, **kwargs) 2025-08-14T21:45:29.5398576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:29.5398961Z key_layer = self.key(current_states) 2025-08-14T21:45:29.5399086Z 2025-08-14T21:45:29.5399193Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5399529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5399843Z return mod(**inputs) 2025-08-14T21:45:29.5400205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5400599Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5400982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5401367Z hidden_states = self.encoder( 2025-08-14T21:45:29.5401743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5402115Z layer_outputs = layer_module( 2025-08-14T21:45:29.5402447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5402807Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5403236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5403664Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5404070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5404476Z return func(*args, **kwargs) 2025-08-14T21:45:29.5404897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5405347Z self_outputs = self.self( 2025-08-14T21:45:29.5405923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5406353Z return func(*args, **kwargs) 2025-08-14T21:45:29.5406744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:29.5407142Z value_layer = self.value(current_states) 2025-08-14T21:45:29.5407284Z 2025-08-14T21:45:29.5407369Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5407582Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5407823Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5408168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5408486Z return mod(**inputs) 2025-08-14T21:45:29.5408844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5409269Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5409658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5410040Z hidden_states = self.encoder( 2025-08-14T21:45:29.5410402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5410782Z layer_outputs = layer_module( 2025-08-14T21:45:29.5411127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5411477Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5411893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5412294Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5412671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5413032Z return func(*args, **kwargs) 2025-08-14T21:45:29.5413413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:29.5413868Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:29.5414301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:29.5414689Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5414829Z 2025-08-14T21:45:29.5414928Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5415272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5415576Z return mod(**inputs) 2025-08-14T21:45:29.5415939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5416361Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5416753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5417125Z hidden_states = self.encoder( 2025-08-14T21:45:29.5417493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5417870Z layer_outputs = layer_module( 2025-08-14T21:45:29.5418199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5418537Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5418917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5419336Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5419717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5420100Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5420512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5420971Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5421393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:29.5421785Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5421921Z 2025-08-14T21:45:29.5422028Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5422379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5422713Z return mod(**inputs) 2025-08-14T21:45:29.5423075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5423472Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5423869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5424243Z hidden_states = self.encoder( 2025-08-14T21:45:29.5424642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5425012Z layer_outputs = layer_module( 2025-08-14T21:45:29.5425327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5425682Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5426060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5426439Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5426815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5427183Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5427589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5428024Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5428445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:29.5428867Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:29.5429236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:29.5429565Z return self.act(input) 2025-08-14T21:45:29.5429680Z 2025-08-14T21:45:29.5429780Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5430130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5430437Z return mod(**inputs) 2025-08-14T21:45:29.5430802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5431202Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5431596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5431968Z hidden_states = self.encoder( 2025-08-14T21:45:29.5432340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5432750Z layer_outputs = layer_module( 2025-08-14T21:45:29.5433091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5433444Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5433844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5434236Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5434611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5434991Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5435398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:29.5435862Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:29.5436291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:29.5436708Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5436840Z 2025-08-14T21:45:29.5436946Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5437290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5437601Z return mod(**inputs) 2025-08-14T21:45:29.5438271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5438848Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5439242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5439655Z hidden_states = self.encoder( 2025-08-14T21:45:29.5440066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5440464Z layer_outputs = layer_module( 2025-08-14T21:45:29.5440792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5441144Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5441525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5441910Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5442298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5442682Z return func(*args, **kwargs) 2025-08-14T21:45:29.5443082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5443489Z self_outputs = self.self( 2025-08-14T21:45:29.5443873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5444241Z return func(*args, **kwargs) 2025-08-14T21:45:29.5444615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:29.5445009Z query_layer = self.query(hidden_states) 2025-08-14T21:45:29.5445153Z 2025-08-14T21:45:29.5445256Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5445671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5446006Z return mod(**inputs) 2025-08-14T21:45:29.5446397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5446835Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5447290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5447669Z hidden_states = self.encoder( 2025-08-14T21:45:29.5448043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5448422Z layer_outputs = layer_module( 2025-08-14T21:45:29.5448751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5449099Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5449489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5449879Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5450242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5450610Z return func(*args, **kwargs) 2025-08-14T21:45:29.5451004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5451387Z self_outputs = self.self( 2025-08-14T21:45:29.5451726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5452085Z return func(*args, **kwargs) 2025-08-14T21:45:29.5452457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:29.5452843Z key_layer = self.key(current_states) 2025-08-14T21:45:29.5452977Z 2025-08-14T21:45:29.5453076Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5453426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5453736Z return mod(**inputs) 2025-08-14T21:45:29.5454085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5454478Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5454859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5455221Z hidden_states = self.encoder( 2025-08-14T21:45:29.5455584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5455957Z layer_outputs = layer_module( 2025-08-14T21:45:29.5456279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5456616Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5456991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5457379Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5457744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5458080Z return func(*args, **kwargs) 2025-08-14T21:45:29.5458665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5459028Z self_outputs = self.self( 2025-08-14T21:45:29.5459353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5459702Z return func(*args, **kwargs) 2025-08-14T21:45:29.5460056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:29.5460430Z value_layer = self.value(current_states) 2025-08-14T21:45:29.5460552Z 2025-08-14T21:45:29.5460645Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5460848Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5461067Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5461393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5461700Z return mod(**inputs) 2025-08-14T21:45:29.5462046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5462438Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5462817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5463191Z hidden_states = self.encoder( 2025-08-14T21:45:29.5463556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5463940Z layer_outputs = layer_module( 2025-08-14T21:45:29.5464271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5464606Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5464974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5465345Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5465695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5466038Z return func(*args, **kwargs) 2025-08-14T21:45:29.5466411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:29.5466842Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:29.5467270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:29.5467658Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5467786Z 2025-08-14T21:45:29.5467884Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5468226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5468532Z return mod(**inputs) 2025-08-14T21:45:29.5468888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5469263Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5469653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5470023Z hidden_states = self.encoder( 2025-08-14T21:45:29.5470387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5470761Z layer_outputs = layer_module( 2025-08-14T21:45:29.5471085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5471426Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5471790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5472176Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5472556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5472927Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5473324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5473774Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5474224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:29.5474608Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5474738Z 2025-08-14T21:45:29.5474835Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5475175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5475483Z return mod(**inputs) 2025-08-14T21:45:29.5475829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5476223Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5476604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5476985Z hidden_states = self.encoder( 2025-08-14T21:45:29.5477341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5477752Z layer_outputs = layer_module( 2025-08-14T21:45:29.5478075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5478407Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5478780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5479162Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5479558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5479929Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5480347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5480818Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5481238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:29.5481646Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:29.5482006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:29.5482333Z return self.act(input) 2025-08-14T21:45:29.5482439Z 2025-08-14T21:45:29.5482537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5482878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5483189Z return mod(**inputs) 2025-08-14T21:45:29.5483541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5483923Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5484315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5484704Z hidden_states = self.encoder( 2025-08-14T21:45:29.5485069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5485446Z layer_outputs = layer_module( 2025-08-14T21:45:29.5485861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5486231Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5486629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5487049Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5487455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5487867Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5488282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:29.5488766Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:29.5489216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:29.5489613Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5489745Z 2025-08-14T21:45:29.5489846Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5490196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5490514Z return mod(**inputs) 2025-08-14T21:45:29.5490869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5491291Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5491683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5492062Z hidden_states = self.encoder( 2025-08-14T21:45:29.5492425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5492802Z layer_outputs = layer_module( 2025-08-14T21:45:29.5493154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5493504Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5493905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5494307Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5494682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5495034Z return func(*args, **kwargs) 2025-08-14T21:45:29.5495405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5495783Z self_outputs = self.self( 2025-08-14T21:45:29.5496130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5496481Z return func(*args, **kwargs) 2025-08-14T21:45:29.5496857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:29.5497245Z query_layer = self.query(hidden_states) 2025-08-14T21:45:29.5497376Z 2025-08-14T21:45:29.5497476Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5497827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5498142Z return mod(**inputs) 2025-08-14T21:45:29.5498496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5498870Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5499242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5499605Z hidden_states = self.encoder( 2025-08-14T21:45:29.5499951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5500313Z layer_outputs = layer_module( 2025-08-14T21:45:29.5500630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5500986Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5501350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5501730Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5502085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5502432Z return func(*args, **kwargs) 2025-08-14T21:45:29.5502790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5503173Z self_outputs = self.self( 2025-08-14T21:45:29.5503526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5503864Z return func(*args, **kwargs) 2025-08-14T21:45:29.5504221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:29.5504611Z key_layer = self.key(current_states) 2025-08-14T21:45:29.5504733Z 2025-08-14T21:45:29.5504834Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5505158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5505454Z return mod(**inputs) 2025-08-14T21:45:29.5505797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5506176Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5506566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5506930Z hidden_states = self.encoder( 2025-08-14T21:45:29.5507308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5507672Z layer_outputs = layer_module( 2025-08-14T21:45:29.5507998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5508342Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5508715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5509092Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5509449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5509804Z return func(*args, **kwargs) 2025-08-14T21:45:29.5510158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5510534Z self_outputs = self.self( 2025-08-14T21:45:29.5510880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5511233Z return func(*args, **kwargs) 2025-08-14T21:45:29.5511587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:29.5511979Z value_layer = self.value(current_states) 2025-08-14T21:45:29.5512110Z 2025-08-14T21:45:29.5512197Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5512402Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5512636Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5512995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5513298Z return mod(**inputs) 2025-08-14T21:45:29.5513647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5514037Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5514432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5514786Z hidden_states = self.encoder( 2025-08-14T21:45:29.5515145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5515511Z layer_outputs = layer_module( 2025-08-14T21:45:29.5515830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5516157Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5516524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5516899Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5517254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5517614Z return func(*args, **kwargs) 2025-08-14T21:45:29.5517967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:29.5518384Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:29.5518789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:29.5519166Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5519303Z 2025-08-14T21:45:29.5519417Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5519752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5520044Z return mod(**inputs) 2025-08-14T21:45:29.5520402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5520791Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5521169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5521534Z hidden_states = self.encoder( 2025-08-14T21:45:29.5521891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5522264Z layer_outputs = layer_module( 2025-08-14T21:45:29.5522586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5522933Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5523314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5523707Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5524083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5524461Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5524869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5525310Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5525799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:29.5526212Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5526354Z 2025-08-14T21:45:29.5526466Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5526818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5527127Z return mod(**inputs) 2025-08-14T21:45:29.5527517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5527909Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5528286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5528659Z hidden_states = self.encoder( 2025-08-14T21:45:29.5529022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5529392Z layer_outputs = layer_module( 2025-08-14T21:45:29.5529717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5530062Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5530440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5530827Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5531224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5531595Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5531994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5532434Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5532861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:29.5533270Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:29.5533637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:29.5533963Z return self.act(input) 2025-08-14T21:45:29.5534077Z 2025-08-14T21:45:29.5534177Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5534518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5534820Z return mod(**inputs) 2025-08-14T21:45:29.5535169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5535563Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5535933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5536308Z hidden_states = self.encoder( 2025-08-14T21:45:29.5536668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5537040Z layer_outputs = layer_module( 2025-08-14T21:45:29.5537354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5537807Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5538191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5538578Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5538946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5539317Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5539726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:29.5540175Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:29.5540602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:29.5541053Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5541182Z 2025-08-14T21:45:29.5541286Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5541620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5541933Z return mod(**inputs) 2025-08-14T21:45:29.5542288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5542680Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5543057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5543432Z hidden_states = self.encoder( 2025-08-14T21:45:29.5543796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5544162Z layer_outputs = layer_module( 2025-08-14T21:45:29.5544518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5544858Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5545232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5545608Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5545972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5546352Z return func(*args, **kwargs) 2025-08-14T21:45:29.5546728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5547089Z self_outputs = self.self( 2025-08-14T21:45:29.5547445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5547793Z return func(*args, **kwargs) 2025-08-14T21:45:29.5548142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:29.5548519Z query_layer = self.query(hidden_states) 2025-08-14T21:45:29.5548653Z 2025-08-14T21:45:29.5548750Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5549094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5549403Z return mod(**inputs) 2025-08-14T21:45:29.5549760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5550154Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5550532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5550911Z hidden_states = self.encoder( 2025-08-14T21:45:29.5551270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5551634Z layer_outputs = layer_module( 2025-08-14T21:45:29.5552029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5552382Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5552774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5553172Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5553540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5553892Z return func(*args, **kwargs) 2025-08-14T21:45:29.5554282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5554665Z self_outputs = self.self( 2025-08-14T21:45:29.5554997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5555338Z return func(*args, **kwargs) 2025-08-14T21:45:29.5555683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:29.5556040Z key_layer = self.key(current_states) 2025-08-14T21:45:29.5556169Z 2025-08-14T21:45:29.5556265Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5556595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5556891Z return mod(**inputs) 2025-08-14T21:45:29.5557235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5557634Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5558006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5558363Z hidden_states = self.encoder( 2025-08-14T21:45:29.5558716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5559079Z layer_outputs = layer_module( 2025-08-14T21:45:29.5559395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5559737Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5560103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5560487Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5560830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5561177Z return func(*args, **kwargs) 2025-08-14T21:45:29.5561539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5561916Z self_outputs = self.self( 2025-08-14T21:45:29.5562253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5562612Z return func(*args, **kwargs) 2025-08-14T21:45:29.5562986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:29.5563371Z value_layer = self.value(current_states) 2025-08-14T21:45:29.5563512Z 2025-08-14T21:45:29.5563589Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5563800Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5564031Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5564380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5564689Z return mod(**inputs) 2025-08-14T21:45:29.5565047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5565431Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5565876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5566272Z hidden_states = self.encoder( 2025-08-14T21:45:29.5566657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5567020Z layer_outputs = layer_module( 2025-08-14T21:45:29.5567346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5567722Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5568091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5568478Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5568836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5569186Z return func(*args, **kwargs) 2025-08-14T21:45:29.5569540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:29.5569966Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:29.5570384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:29.5570762Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5570912Z 2025-08-14T21:45:29.5571009Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5571346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5571653Z return mod(**inputs) 2025-08-14T21:45:29.5571996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5572384Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5572820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5573197Z hidden_states = self.encoder( 2025-08-14T21:45:29.5573554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5573950Z layer_outputs = layer_module( 2025-08-14T21:45:29.5574283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5574629Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5574996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5575383Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5575762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5576123Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5576526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5576975Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5577390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:29.5577770Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5577917Z 2025-08-14T21:45:29.5578012Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5578342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5578641Z return mod(**inputs) 2025-08-14T21:45:29.5578975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5579353Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5579722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5580078Z hidden_states = self.encoder( 2025-08-14T21:45:29.5580437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5580813Z layer_outputs = layer_module( 2025-08-14T21:45:29.5581130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5581455Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5581816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5582198Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5582559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5582923Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5583312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5583746Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5584144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:29.5584562Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:29.5584911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:29.5585220Z return self.act(input) 2025-08-14T21:45:29.5585323Z 2025-08-14T21:45:29.5585419Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5585757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5586075Z return mod(**inputs) 2025-08-14T21:45:29.5586416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5586813Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5587190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5587556Z hidden_states = self.encoder( 2025-08-14T21:45:29.5587904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5588272Z layer_outputs = layer_module( 2025-08-14T21:45:29.5588589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5588919Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5589284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5589662Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5590031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5590391Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5590786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:29.5591234Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:29.5591656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:29.5592024Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5592157Z 2025-08-14T21:45:29.5592251Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5592585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5592885Z return mod(**inputs) 2025-08-14T21:45:29.5593221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5593605Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5593992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5594344Z hidden_states = self.encoder( 2025-08-14T21:45:29.5594698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5595060Z layer_outputs = layer_module( 2025-08-14T21:45:29.5595375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5595698Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5596064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5596438Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5596787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5597159Z return func(*args, **kwargs) 2025-08-14T21:45:29.5597512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5597879Z self_outputs = self.self( 2025-08-14T21:45:29.5598211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5598555Z return func(*args, **kwargs) 2025-08-14T21:45:29.5598928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:29.5599298Z query_layer = self.query(hidden_states) 2025-08-14T21:45:29.5599422Z 2025-08-14T21:45:29.5599518Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5599869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5600173Z return mod(**inputs) 2025-08-14T21:45:29.5600512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5600898Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5601270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5601630Z hidden_states = self.encoder( 2025-08-14T21:45:29.5601976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5602353Z layer_outputs = layer_module( 2025-08-14T21:45:29.5602679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5603013Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5603389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5603780Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5604138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5604482Z return func(*args, **kwargs) 2025-08-14T21:45:29.5604844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5605222Z self_outputs = self.self( 2025-08-14T21:45:29.5605637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5606044Z return func(*args, **kwargs) 2025-08-14T21:45:29.5606447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:29.5606869Z key_layer = self.key(current_states) 2025-08-14T21:45:29.5607044Z 2025-08-14T21:45:29.5607146Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5607488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5607802Z return mod(**inputs) 2025-08-14T21:45:29.5608157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5608544Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5608922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5609295Z hidden_states = self.encoder( 2025-08-14T21:45:29.5609648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5610015Z layer_outputs = layer_module( 2025-08-14T21:45:29.5610337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5610696Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5611060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5611440Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5611799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5612148Z return func(*args, **kwargs) 2025-08-14T21:45:29.5612514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5612889Z self_outputs = self.self( 2025-08-14T21:45:29.5613255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5613675Z return func(*args, **kwargs) 2025-08-14T21:45:29.5614036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:29.5614421Z value_layer = self.value(current_states) 2025-08-14T21:45:29.5614549Z 2025-08-14T21:45:29.5614632Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5614828Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5615049Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5615386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5615686Z return mod(**inputs) 2025-08-14T21:45:29.5616042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5616430Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5616813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5617178Z hidden_states = self.encoder( 2025-08-14T21:45:29.5617540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5617911Z layer_outputs = layer_module( 2025-08-14T21:45:29.5618226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5618566Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5618951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5619332Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5619703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5620068Z return func(*args, **kwargs) 2025-08-14T21:45:29.5620467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:29.5620917Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:29.5621352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:29.5621773Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5621917Z 2025-08-14T21:45:29.5622032Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5622401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5622744Z return mod(**inputs) 2025-08-14T21:45:29.5623140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5623582Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5624008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5624424Z hidden_states = self.encoder( 2025-08-14T21:45:29.5624808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5625207Z layer_outputs = layer_module( 2025-08-14T21:45:29.5625549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5625914Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5626342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5626738Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5627157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5627596Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5628045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5628544Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5629014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:29.5629452Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5629594Z 2025-08-14T21:45:29.5629709Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5630091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5630444Z return mod(**inputs) 2025-08-14T21:45:29.5630837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5631267Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5631695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5632111Z hidden_states = self.encoder( 2025-08-14T21:45:29.5632514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5632919Z layer_outputs = layer_module( 2025-08-14T21:45:29.5633280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5633658Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5634078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5634514Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5634930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5635365Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5635807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5636315Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5636774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:29.5637233Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:29.5637750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:29.5638132Z return self.act(input) 2025-08-14T21:45:29.5638250Z 2025-08-14T21:45:29.5638371Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5638742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5639139Z return mod(**inputs) 2025-08-14T21:45:29.5639522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5639955Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5640401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5640828Z hidden_states = self.encoder( 2025-08-14T21:45:29.5641264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5641687Z layer_outputs = layer_module( 2025-08-14T21:45:29.5674309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5674840Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5675265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5675675Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5676123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5676600Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5677061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:29.5677528Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:29.5677951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:29.5678343Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5678487Z 2025-08-14T21:45:29.5678595Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5678951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5679264Z return mod(**inputs) 2025-08-14T21:45:29.5679631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5680029Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5680421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5680816Z hidden_states = self.encoder( 2025-08-14T21:45:29.5681186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5681560Z layer_outputs = layer_module( 2025-08-14T21:45:29.5681888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5682305Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5682717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5683115Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5683498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5683878Z return func(*args, **kwargs) 2025-08-14T21:45:29.5684251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5684630Z self_outputs = self.self( 2025-08-14T21:45:29.5684993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5685370Z return func(*args, **kwargs) 2025-08-14T21:45:29.5685836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:29.5686309Z query_layer = self.query(hidden_states) 2025-08-14T21:45:29.5686455Z 2025-08-14T21:45:29.5686578Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5686948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5687299Z return mod(**inputs) 2025-08-14T21:45:29.5687669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5688112Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5688492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5688882Z hidden_states = self.encoder( 2025-08-14T21:45:29.5689243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5689611Z layer_outputs = layer_module( 2025-08-14T21:45:29.5689929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5690270Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5690643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5691020Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5691380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5691734Z return func(*args, **kwargs) 2025-08-14T21:45:29.5692082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5692443Z self_outputs = self.self( 2025-08-14T21:45:29.5692783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5693127Z return func(*args, **kwargs) 2025-08-14T21:45:29.5693477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:29.5693856Z key_layer = self.key(current_states) 2025-08-14T21:45:29.5693982Z 2025-08-14T21:45:29.5694091Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5694427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5694728Z return mod(**inputs) 2025-08-14T21:45:29.5695079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5695471Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5695848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5696241Z hidden_states = self.encoder( 2025-08-14T21:45:29.5696603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5696972Z layer_outputs = layer_module( 2025-08-14T21:45:29.5697289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5697627Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5698004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5698377Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5698735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5699088Z return func(*args, **kwargs) 2025-08-14T21:45:29.5699467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5699836Z self_outputs = self.self( 2025-08-14T21:45:29.5700178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5700530Z return func(*args, **kwargs) 2025-08-14T21:45:29.5700891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:29.5701286Z value_layer = self.value(current_states) 2025-08-14T21:45:29.5701422Z 2025-08-14T21:45:29.5701501Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5701705Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5701942Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5702295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5702615Z return mod(**inputs) 2025-08-14T21:45:29.5702976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5703372Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5703815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5704197Z hidden_states = self.encoder( 2025-08-14T21:45:29.5704564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5704947Z layer_outputs = layer_module( 2025-08-14T21:45:29.5705279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5705627Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5706006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5706397Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5706760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5707113Z return func(*args, **kwargs) 2025-08-14T21:45:29.5707481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:29.5707916Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:29.5708345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:29.5708733Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5708875Z 2025-08-14T21:45:29.5708977Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5709346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5709659Z return mod(**inputs) 2025-08-14T21:45:29.5710017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5710422Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5710812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5711186Z hidden_states = self.encoder( 2025-08-14T21:45:29.5711558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5711934Z layer_outputs = layer_module( 2025-08-14T21:45:29.5712267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5712610Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5713026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5713425Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5713814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5714191Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5714607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5715085Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5715514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:29.5715963Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5716109Z 2025-08-14T21:45:29.5716213Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5716561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5716870Z return mod(**inputs) 2025-08-14T21:45:29.5717231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5717630Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5718012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5718393Z hidden_states = self.encoder( 2025-08-14T21:45:29.5718762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5719143Z layer_outputs = layer_module( 2025-08-14T21:45:29.5719468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5719825Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5720215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5720604Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5720986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5721380Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5721800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5722264Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5722697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:29.5723168Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:29.5723552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:29.5723890Z return self.act(input) 2025-08-14T21:45:29.5724011Z 2025-08-14T21:45:29.5724115Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5724473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5724799Z return mod(**inputs) 2025-08-14T21:45:29.5725169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5725663Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5726124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5726543Z hidden_states = self.encoder( 2025-08-14T21:45:29.5726980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5727372Z layer_outputs = layer_module( 2025-08-14T21:45:29.5727715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5728066Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5728463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5728889Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5729284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5729676Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5730117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:29.5730605Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:29.5731048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:29.5731455Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5731597Z 2025-08-14T21:45:29.5731701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5732057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5732374Z return mod(**inputs) 2025-08-14T21:45:29.5732749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5733162Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5733558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5733949Z hidden_states = self.encoder( 2025-08-14T21:45:29.5734331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5734721Z layer_outputs = layer_module( 2025-08-14T21:45:29.5735052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5735406Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5735803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5736201Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5736575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5736948Z return func(*args, **kwargs) 2025-08-14T21:45:29.5737348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5737916Z self_outputs = self.self( 2025-08-14T21:45:29.5738286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5738676Z return func(*args, **kwargs) 2025-08-14T21:45:29.5739078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:29.5739495Z query_layer = self.query(hidden_states) 2025-08-14T21:45:29.5739652Z 2025-08-14T21:45:29.5739766Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5740138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5740454Z return mod(**inputs) 2025-08-14T21:45:29.5740833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5741324Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5741752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5742157Z hidden_states = self.encoder( 2025-08-14T21:45:29.5742567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5742984Z layer_outputs = layer_module( 2025-08-14T21:45:29.5743386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5743760Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5744200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5744630Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5745025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5745413Z return func(*args, **kwargs) 2025-08-14T21:45:29.5745776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5746132Z self_outputs = self.self( 2025-08-14T21:45:29.5746458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5746800Z return func(*args, **kwargs) 2025-08-14T21:45:29.5747151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:29.5747508Z key_layer = self.key(current_states) 2025-08-14T21:45:29.5747637Z 2025-08-14T21:45:29.5747732Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5748063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5748361Z return mod(**inputs) 2025-08-14T21:45:29.5748693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5749071Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5749442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5749804Z hidden_states = self.encoder( 2025-08-14T21:45:29.5750150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5750509Z layer_outputs = layer_module( 2025-08-14T21:45:29.5750826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5751176Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5751542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5751911Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5752257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5752589Z return func(*args, **kwargs) 2025-08-14T21:45:29.5752936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5753300Z self_outputs = self.self( 2025-08-14T21:45:29.5753622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5753965Z return func(*args, **kwargs) 2025-08-14T21:45:29.5754322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:29.5754716Z value_layer = self.value(current_states) 2025-08-14T21:45:29.5754838Z 2025-08-14T21:45:29.5754914Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5755118Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5755333Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5755659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5755946Z return mod(**inputs) 2025-08-14T21:45:29.5756301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5756685Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5757067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5757430Z hidden_states = self.encoder( 2025-08-14T21:45:29.5757785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5758141Z layer_outputs = layer_module( 2025-08-14T21:45:29.5758447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5758776Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5759142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5759514Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5759856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5760199Z return func(*args, **kwargs) 2025-08-14T21:45:29.5760552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:29.5760960Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:29.5761372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:29.5761746Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5761873Z 2025-08-14T21:45:29.5761975Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5762297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5762598Z return mod(**inputs) 2025-08-14T21:45:29.5762940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5763319Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5763695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5764083Z hidden_states = self.encoder( 2025-08-14T21:45:29.5764437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5764808Z layer_outputs = layer_module( 2025-08-14T21:45:29.5765136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5765478Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5765923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5766310Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5766716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5767117Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5767534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5768035Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5768455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:29.5768844Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5768974Z 2025-08-14T21:45:29.5769074Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5769415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5769740Z return mod(**inputs) 2025-08-14T21:45:29.5770095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5770500Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5770886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5771262Z hidden_states = self.encoder( 2025-08-14T21:45:29.5771617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5771990Z layer_outputs = layer_module( 2025-08-14T21:45:29.5772314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5772657Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5773023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5773405Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5773782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5774153Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5774546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5774988Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5775401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:29.5775805Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:29.5776168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:29.5776494Z return self.act(input) 2025-08-14T21:45:29.5776600Z 2025-08-14T21:45:29.5776704Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5777040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5777347Z return mod(**inputs) 2025-08-14T21:45:29.5777720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5778102Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5778486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5778859Z hidden_states = self.encoder( 2025-08-14T21:45:29.5779223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5779596Z layer_outputs = layer_module( 2025-08-14T21:45:29.5779907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5780239Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5780607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5780994Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5781363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5781726Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5782109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:29.5782564Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:29.5782995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:29.5783371Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5783500Z 2025-08-14T21:45:29.5783613Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5783949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5784258Z return mod(**inputs) 2025-08-14T21:45:29.5784605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5784984Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5785365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5785732Z hidden_states = self.encoder( 2025-08-14T21:45:29.5786084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5786454Z layer_outputs = layer_module( 2025-08-14T21:45:29.5786774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5787110Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5787470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5787847Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5788200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5788543Z return func(*args, **kwargs) 2025-08-14T21:45:29.5788897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5789263Z self_outputs = self.self( 2025-08-14T21:45:29.5789485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5789558Z return func(*args, **kwargs) 2025-08-14T21:45:29.5789800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:29.5789893Z query_layer = self.query(hidden_states) 2025-08-14T21:45:29.5789903Z 2025-08-14T21:45:29.5789998Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5790182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5790249Z return mod(**inputs) 2025-08-14T21:45:29.5790492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5790574Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5790822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5790888Z hidden_states = self.encoder( 2025-08-14T21:45:29.5791133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5791201Z layer_outputs = layer_module( 2025-08-14T21:45:29.5791431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5791510Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5791748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5791824Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5792049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5792114Z return func(*args, **kwargs) 2025-08-14T21:45:29.5792378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5792447Z self_outputs = self.self( 2025-08-14T21:45:29.5792691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5792770Z return func(*args, **kwargs) 2025-08-14T21:45:29.5793014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:29.5793088Z key_layer = self.key(current_states) 2025-08-14T21:45:29.5793099Z 2025-08-14T21:45:29.5793197Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5793383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5793456Z return mod(**inputs) 2025-08-14T21:45:29.5793713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5793794Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5794044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5794112Z hidden_states = self.encoder( 2025-08-14T21:45:29.5794360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5794425Z layer_outputs = layer_module( 2025-08-14T21:45:29.5794629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5794708Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5794945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5795021Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5795249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5795314Z return func(*args, **kwargs) 2025-08-14T21:45:29.5795557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5795643Z self_outputs = self.self( 2025-08-14T21:45:29.5795863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5795936Z return func(*args, **kwargs) 2025-08-14T21:45:29.5796178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:29.5796260Z value_layer = self.value(current_states) 2025-08-14T21:45:29.5796263Z 2025-08-14T21:45:29.5796337Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5796413Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5796515Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5796699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5796760Z return mod(**inputs) 2025-08-14T21:45:29.5797010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5797122Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5797368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5797433Z hidden_states = self.encoder( 2025-08-14T21:45:29.5797671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5797742Z layer_outputs = layer_module( 2025-08-14T21:45:29.5797960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5798032Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5798294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5798373Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5798597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5798662Z return func(*args, **kwargs) 2025-08-14T21:45:29.5798899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:29.5799024Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:29.5799263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:29.5799348Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5799351Z 2025-08-14T21:45:29.5799445Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5799627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5799697Z return mod(**inputs) 2025-08-14T21:45:29.5799940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5800022Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5800275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5800343Z hidden_states = self.encoder( 2025-08-14T21:45:29.5800593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5800661Z layer_outputs = layer_module( 2025-08-14T21:45:29.5800866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5800947Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5801191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5801298Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5801548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5801623Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5801913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5802028Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5802283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:29.5802370Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5802373Z 2025-08-14T21:45:29.5802473Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5802677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5802759Z return mod(**inputs) 2025-08-14T21:45:29.5803018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5803109Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5803361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5803436Z hidden_states = self.encoder( 2025-08-14T21:45:29.5803706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5803777Z layer_outputs = layer_module( 2025-08-14T21:45:29.5804016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5804094Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5804352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5804440Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5804689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5804771Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5805062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5805184Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5805450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:29.5805635Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:29.5805868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:29.5805945Z return self.act(input) 2025-08-14T21:45:29.5805949Z 2025-08-14T21:45:29.5806057Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5806278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5806347Z return mod(**inputs) 2025-08-14T21:45:29.5806626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5806719Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5806969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5807047Z hidden_states = self.encoder( 2025-08-14T21:45:29.5807308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5807398Z layer_outputs = layer_module( 2025-08-14T21:45:29.5807614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5807688Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5807934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5808019Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5808280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5808350Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5808630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:29.5808764Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:29.5809031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:29.5809114Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5809118Z 2025-08-14T21:45:29.5809214Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5809402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5809470Z return mod(**inputs) 2025-08-14T21:45:29.5809733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5809823Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5810081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5810152Z hidden_states = self.encoder( 2025-08-14T21:45:29.5810404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5810472Z layer_outputs = layer_module( 2025-08-14T21:45:29.5810680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5810761Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5811008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5811093Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5811321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5811388Z return func(*args, **kwargs) 2025-08-14T21:45:29.5811640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5811710Z self_outputs = self.self( 2025-08-14T21:45:29.5811937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5812011Z return func(*args, **kwargs) 2025-08-14T21:45:29.5812252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:29.5812336Z query_layer = self.query(hidden_states) 2025-08-14T21:45:29.5812340Z 2025-08-14T21:45:29.5812435Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5812624Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5812692Z return mod(**inputs) 2025-08-14T21:45:29.5812939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5813027Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5813293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5813361Z hidden_states = self.encoder( 2025-08-14T21:45:29.5813610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5813675Z layer_outputs = layer_module( 2025-08-14T21:45:29.5813879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5813959Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5814202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5814282Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5814506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5814589Z return func(*args, **kwargs) 2025-08-14T21:45:29.5814840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5814908Z self_outputs = self.self( 2025-08-14T21:45:29.5815134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5815201Z return func(*args, **kwargs) 2025-08-14T21:45:29.5815459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:29.5815543Z key_layer = self.key(current_states) 2025-08-14T21:45:29.5815546Z 2025-08-14T21:45:29.5815640Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5815842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5815913Z return mod(**inputs) 2025-08-14T21:45:29.5816164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5816250Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5816495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5816561Z hidden_states = self.encoder( 2025-08-14T21:45:29.5816815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5816884Z layer_outputs = layer_module( 2025-08-14T21:45:29.5817092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5817170Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5817417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5817502Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5817723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5817790Z return func(*args, **kwargs) 2025-08-14T21:45:29.5818039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5818107Z self_outputs = self.self( 2025-08-14T21:45:29.5818340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5818405Z return func(*args, **kwargs) 2025-08-14T21:45:29.5818651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:29.5818733Z value_layer = self.value(current_states) 2025-08-14T21:45:29.5818757Z 2025-08-14T21:45:29.5818836Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5818910Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5819014Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5819205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5819274Z return mod(**inputs) 2025-08-14T21:45:29.5819523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5819615Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5819863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5819928Z hidden_states = self.encoder( 2025-08-14T21:45:29.5820173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5820265Z layer_outputs = layer_module( 2025-08-14T21:45:29.5820468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5820545Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5820787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5820860Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5821088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5821166Z return func(*args, **kwargs) 2025-08-14T21:45:29.5821414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:29.5821546Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:29.5821789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:29.5821874Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5821877Z 2025-08-14T21:45:29.5821972Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5822154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5822221Z return mod(**inputs) 2025-08-14T21:45:29.5822461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5822549Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5822790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5822856Z hidden_states = self.encoder( 2025-08-14T21:45:29.5823105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5823172Z layer_outputs = layer_module( 2025-08-14T21:45:29.5823379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5823450Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5823689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5823772Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5824007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5824075Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5824354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5824483Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5824729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:29.5824803Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5824806Z 2025-08-14T21:45:29.5824898Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5825087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5825149Z return mod(**inputs) 2025-08-14T21:45:29.5825398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5825479Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5825716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5825790Z hidden_states = self.encoder( 2025-08-14T21:45:29.5826045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5826110Z layer_outputs = layer_module( 2025-08-14T21:45:29.5826320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5826390Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5826636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5826728Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5826971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5827050Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5827381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5827503Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5827743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:29.5827848Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:29.5828050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:29.5828115Z return self.act(input) 2025-08-14T21:45:29.5828118Z 2025-08-14T21:45:29.5828212Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5828403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5828465Z return mod(**inputs) 2025-08-14T21:45:29.5828716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5828795Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5829034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5829109Z hidden_states = self.encoder( 2025-08-14T21:45:29.5829348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5829421Z layer_outputs = layer_module( 2025-08-14T21:45:29.5829624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5829696Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5829947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5830025Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5830580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5830660Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5830925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:29.5831052Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:29.5831289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:29.5831366Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5831370Z 2025-08-14T21:45:29.5831473Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5831660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5831730Z return mod(**inputs) 2025-08-14T21:45:29.5831977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5832078Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5832332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5832402Z hidden_states = self.encoder( 2025-08-14T21:45:29.5832646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5832722Z layer_outputs = layer_module( 2025-08-14T21:45:29.5832941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5833023Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5833298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5833378Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5833612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5833679Z return func(*args, **kwargs) 2025-08-14T21:45:29.5833924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5834000Z self_outputs = self.self( 2025-08-14T21:45:29.5834225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5834302Z return func(*args, **kwargs) 2025-08-14T21:45:29.5834545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:29.5834622Z query_layer = self.query(hidden_states) 2025-08-14T21:45:29.5834626Z 2025-08-14T21:45:29.5834730Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5834920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5834990Z return mod(**inputs) 2025-08-14T21:45:29.5835242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5835323Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5835577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5835646Z hidden_states = self.encoder( 2025-08-14T21:45:29.5835892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5835968Z layer_outputs = layer_module( 2025-08-14T21:45:29.5836176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5836275Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5836525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5836601Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5836836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5836905Z return func(*args, **kwargs) 2025-08-14T21:45:29.5837162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5837230Z self_outputs = self.self( 2025-08-14T21:45:29.5837462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5837538Z return func(*args, **kwargs) 2025-08-14T21:45:29.5837936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:29.5838063Z key_layer = self.key(current_states) 2025-08-14T21:45:29.5838068Z 2025-08-14T21:45:29.5838180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5838376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5838449Z return mod(**inputs) 2025-08-14T21:45:29.5838708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5838816Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5839083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5839181Z hidden_states = self.encoder( 2025-08-14T21:45:29.5839443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5839524Z layer_outputs = layer_module( 2025-08-14T21:45:29.5839739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5839833Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5840086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5840162Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5840400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5840467Z return func(*args, **kwargs) 2025-08-14T21:45:29.5840724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5840791Z self_outputs = self.self( 2025-08-14T21:45:29.5841027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5841102Z return func(*args, **kwargs) 2025-08-14T21:45:29.5841353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:29.5841430Z value_layer = self.value(current_states) 2025-08-14T21:45:29.5841433Z 2025-08-14T21:45:29.5841519Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5841595Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5841698Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5841890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5841952Z return mod(**inputs) 2025-08-14T21:45:29.5842212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5842330Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5842601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5842683Z hidden_states = self.encoder( 2025-08-14T21:45:29.5842951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5843032Z layer_outputs = layer_module( 2025-08-14T21:45:29.5843259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5843339Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5843617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5843696Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5843944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5844032Z return func(*args, **kwargs) 2025-08-14T21:45:29.5844290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:29.5844420Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:29.5844680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:29.5844761Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5844773Z 2025-08-14T21:45:29.5844892Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5845090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5845182Z return mod(**inputs) 2025-08-14T21:45:29.5845446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5845532Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5845860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5845930Z hidden_states = self.encoder( 2025-08-14T21:45:29.5846194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5846263Z layer_outputs = layer_module( 2025-08-14T21:45:29.5846484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5846568Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5846827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5846908Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5847175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5847247Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5847528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5847641Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5847886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:29.5847972Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5847976Z 2025-08-14T21:45:29.5848071Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5848266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5848351Z return mod(**inputs) 2025-08-14T21:45:29.5848599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5848687Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5848930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5848998Z hidden_states = self.encoder( 2025-08-14T21:45:29.5849250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5849318Z layer_outputs = layer_module( 2025-08-14T21:45:29.5849532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5849604Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5849847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5849951Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5850191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5850271Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5850542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5850654Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5850945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:29.5851052Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:29.5851270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:29.5851348Z return self.act(input) 2025-08-14T21:45:29.5851353Z 2025-08-14T21:45:29.5851448Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5851640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5851704Z return mod(**inputs) 2025-08-14T21:45:29.5851952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5852039Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5852283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5852357Z hidden_states = self.encoder( 2025-08-14T21:45:29.5852601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5852669Z layer_outputs = layer_module( 2025-08-14T21:45:29.5852882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5852957Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5853199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5853285Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5853527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5853610Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5853890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:29.5854012Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:29.5854258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:29.5854372Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5854376Z 2025-08-14T21:45:29.5854478Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5854655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5854716Z return mod(**inputs) 2025-08-14T21:45:29.5854964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5855043Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5855281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5855356Z hidden_states = self.encoder( 2025-08-14T21:45:29.5855595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5855686Z layer_outputs = layer_module( 2025-08-14T21:45:29.5855886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5855957Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5856198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5856272Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5856491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5856587Z return func(*args, **kwargs) 2025-08-14T21:45:29.5856824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5856913Z self_outputs = self.self( 2025-08-14T21:45:29.5857133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5857200Z return func(*args, **kwargs) 2025-08-14T21:45:29.5857442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:29.5857516Z query_layer = self.query(hidden_states) 2025-08-14T21:45:29.5857520Z 2025-08-14T21:45:29.5857617Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5857796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5857857Z return mod(**inputs) 2025-08-14T21:45:29.5858101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5858179Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5858413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5858489Z hidden_states = self.encoder( 2025-08-14T21:45:29.5858719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5858789Z layer_outputs = layer_module( 2025-08-14T21:45:29.5858986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5859058Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5859306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5859381Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5859611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5859678Z return func(*args, **kwargs) 2025-08-14T21:45:29.5859941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5860016Z self_outputs = self.self( 2025-08-14T21:45:29.5860239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5860304Z return func(*args, **kwargs) 2025-08-14T21:45:29.5860555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:29.5860628Z key_layer = self.key(current_states) 2025-08-14T21:45:29.5860632Z 2025-08-14T21:45:29.5860734Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5860929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5860989Z return mod(**inputs) 2025-08-14T21:45:29.5861239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5861335Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5861571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5861643Z hidden_states = self.encoder( 2025-08-14T21:45:29.5861877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5861949Z layer_outputs = layer_module( 2025-08-14T21:45:29.5862164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5862236Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5862497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5862573Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5862802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5862865Z return func(*args, **kwargs) 2025-08-14T21:45:29.5863101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5863175Z self_outputs = self.self( 2025-08-14T21:45:29.5863392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5863456Z return func(*args, **kwargs) 2025-08-14T21:45:29.5863700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:29.5863773Z value_layer = self.value(current_states) 2025-08-14T21:45:29.5863777Z 2025-08-14T21:45:29.5863858Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5863932Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5864027Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5864214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5864274Z return mod(**inputs) 2025-08-14T21:45:29.5864514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5864600Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5864839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5864911Z hidden_states = self.encoder( 2025-08-14T21:45:29.5865147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5865213Z layer_outputs = layer_module( 2025-08-14T21:45:29.5865418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5865507Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5865755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5865833Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5866052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5866122Z return func(*args, **kwargs) 2025-08-14T21:45:29.5866361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:29.5866477Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:29.5866725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:29.5866823Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5866827Z 2025-08-14T21:45:29.5866926Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5867107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5867167Z return mod(**inputs) 2025-08-14T21:45:29.5867414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5867491Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5867753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5867820Z hidden_states = self.encoder( 2025-08-14T21:45:29.5868070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5868147Z layer_outputs = layer_module( 2025-08-14T21:45:29.5868352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5868424Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5868669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5868745Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5868984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5869054Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5869322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5869442Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5869678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:29.5869761Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5869765Z 2025-08-14T21:45:29.5869856Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5870037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5870105Z return mod(**inputs) 2025-08-14T21:45:29.5870345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5870426Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5870668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5870733Z hidden_states = self.encoder( 2025-08-14T21:45:29.5870975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5871059Z layer_outputs = layer_module( 2025-08-14T21:45:29.5871258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5871336Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5871572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5871656Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5871890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5871959Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5872230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5872338Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5872593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:29.5872704Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:29.5872897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:29.5872968Z return self.act(input) 2025-08-14T21:45:29.5872971Z 2025-08-14T21:45:29.5873062Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5873255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5873326Z return mod(**inputs) 2025-08-14T21:45:29.5873567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5873668Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5873905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5873972Z hidden_states = self.encoder( 2025-08-14T21:45:29.5874217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5874283Z layer_outputs = layer_module( 2025-08-14T21:45:29.5874482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5874560Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5874796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5874879Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5875112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5875184Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5875458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:29.5875580Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:29.5875823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:29.5875897Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5875900Z 2025-08-14T21:45:29.5875994Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5876185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5876247Z return mod(**inputs) 2025-08-14T21:45:29.5876492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5876600Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5876844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5876920Z hidden_states = self.encoder( 2025-08-14T21:45:29.5877164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5877231Z layer_outputs = layer_module( 2025-08-14T21:45:29.5877445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5877519Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5877765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5877862Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5878083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5878183Z return func(*args, **kwargs) 2025-08-14T21:45:29.5878423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5878492Z self_outputs = self.self( 2025-08-14T21:45:29.5878721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5878788Z return func(*args, **kwargs) 2025-08-14T21:45:29.5879052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 241, in forward 2025-08-14T21:45:29.5879131Z query_layer = self.query(hidden_states) 2025-08-14T21:45:29.5879135Z 2025-08-14T21:45:29.5879252Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5879452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5879518Z return mod(**inputs) 2025-08-14T21:45:29.5879768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5879857Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5880103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5880177Z hidden_states = self.encoder( 2025-08-14T21:45:29.5880426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5880492Z layer_outputs = layer_module( 2025-08-14T21:45:29.5880706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5880781Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5881038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5881116Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5881345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5881419Z return func(*args, **kwargs) 2025-08-14T21:45:29.5881664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5881733Z self_outputs = self.self( 2025-08-14T21:45:29.5881976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5882043Z return func(*args, **kwargs) 2025-08-14T21:45:29.5882301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 270, in forward 2025-08-14T21:45:29.5882394Z key_layer = self.key(current_states) 2025-08-14T21:45:29.5882399Z 2025-08-14T21:45:29.5882499Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5882696Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5882760Z return mod(**inputs) 2025-08-14T21:45:29.5883010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5883101Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5883351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5883426Z hidden_states = self.encoder( 2025-08-14T21:45:29.5883680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5883752Z layer_outputs = layer_module( 2025-08-14T21:45:29.5883995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5884070Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5884335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5884415Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5884654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5884729Z return func(*args, **kwargs) 2025-08-14T21:45:29.5885002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 401, in forward 2025-08-14T21:45:29.5885075Z self_outputs = self.self( 2025-08-14T21:45:29.5885336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5885412Z return func(*args, **kwargs) 2025-08-14T21:45:29.5885776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 274, in forward 2025-08-14T21:45:29.5885867Z value_layer = self.value(current_states) 2025-08-14T21:45:29.5885871Z 2025-08-14T21:45:29.5885956Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5886048Z cudagraph partition due to non gpu ops 2025-08-14T21:45:29.5886156Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5886366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5886445Z return mod(**inputs) 2025-08-14T21:45:29.5886726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5886833Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5887081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5887151Z hidden_states = self.encoder( 2025-08-14T21:45:29.5887406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5887473Z layer_outputs = layer_module( 2025-08-14T21:45:29.5887687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5887760Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5888006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 474, in forward 2025-08-14T21:45:29.5888089Z self_attention_outputs = self.attention( 2025-08-14T21:45:29.5888318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:29.5888409Z return func(*args, **kwargs) 2025-08-14T21:45:29.5888665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 411, in forward 2025-08-14T21:45:29.5888785Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:45:29.5889040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 348, in forward 2025-08-14T21:45:29.5889116Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5889119Z 2025-08-14T21:45:29.5889212Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5889408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5889470Z return mod(**inputs) 2025-08-14T21:45:29.5889725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5889805Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5890066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5890140Z hidden_states = self.encoder( 2025-08-14T21:45:29.5890383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5890451Z layer_outputs = layer_module( 2025-08-14T21:45:29.5890668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5890740Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5891005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5891086Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5891348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5891434Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5891712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5891835Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5892078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 427, in forward 2025-08-14T21:45:29.5892155Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5892158Z 2025-08-14T21:45:29.5892265Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5892452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5892515Z return mod(**inputs) 2025-08-14T21:45:29.5892773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5892860Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5893122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5893192Z hidden_states = self.encoder( 2025-08-14T21:45:29.5893441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5893518Z layer_outputs = layer_module( 2025-08-14T21:45:29.5893729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5893813Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5894074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5894153Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5894419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5894494Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5894774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 512, in feed_forward_chunk 2025-08-14T21:45:29.5894892Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:45:29.5895129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 428, in forward 2025-08-14T21:45:29.5895241Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:45:29.5895437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:45:29.5895500Z return self.act(input) 2025-08-14T21:45:29.5895504Z 2025-08-14T21:45:29.5895606Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5895806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5895873Z return mod(**inputs) 2025-08-14T21:45:29.5896115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1317, in forward 2025-08-14T21:45:29.5896195Z discriminator_hidden_states = self.electra( 2025-08-14T21:45:29.5896437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 799, in forward 2025-08-14T21:45:29.5896501Z hidden_states = self.encoder( 2025-08-14T21:45:29.5896754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 566, in forward 2025-08-14T21:45:29.5896830Z layer_outputs = layer_module( 2025-08-14T21:45:29.5897051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:29.5897133Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:29.5897376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 504, in forward 2025-08-14T21:45:29.5897450Z layer_output = apply_chunking_to_forward( 2025-08-14T21:45:29.5897691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:45:29.5897760Z return forward_fn(*input_tensors) 2025-08-14T21:45:29.5898026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 513, in feed_forward_chunk 2025-08-14T21:45:29.5898155Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:45:29.5898394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 441, in forward 2025-08-14T21:45:29.5898477Z hidden_states = self.dense(hidden_states) 2025-08-14T21:45:29.5898483Z 2025-08-14T21:45:29.5898575Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5898756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5898824Z return mod(**inputs) 2025-08-14T21:45:29.5899064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1330, in forward 2025-08-14T21:45:29.5899147Z logits = self.qa_outputs(sequence_output) 2025-08-14T21:45:29.5899150Z 2025-08-14T21:45:29.5899240Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5899422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5899489Z return mod(**inputs) 2025-08-14T21:45:29.5899730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1348, in forward 2025-08-14T21:45:29.5899862Z start_loss = loss_fct(start_logits, start_positions) 2025-08-14T21:45:29.5899868Z 2025-08-14T21:45:29.5899962Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:29.5900139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:29.5900206Z return mod(**inputs) 2025-08-14T21:45:29.5900450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/electra/modeling_electra.py", line 1349, in forward 2025-08-14T21:45:29.5900535Z end_loss = loss_fct(end_logits, end_positions) 2025-08-14T21:45:29.5900539Z 2025-08-14T21:45:36.3930847Z Compilation time (from dynamo_timed): 13.380790159 2025-08-14T21:45:36.3934762Z pass 2025-08-14T21:45:36.3941624Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:45:36.3944816Z TIMING: _recursive_pre_grad_passes:0.00657 _recursive_joint_graph_passes:0.42681 _recursive_post_grad_passes:0.08281 async_compile.wait:0.00207 code_gen:6.3198 inductor_compile:7.44545 backend_compile:10.75851 gc:0.00023 entire_frame_compile:13.38079 total_wall_time:13.38079 2025-08-14T21:45:36.3946002Z STATS: call_* op count: 378 | FakeTensorMode.__torch_dispatch__:15006 | FakeTensor.__torch_dispatch__:4704 | ProxyTorchDispatchMode.__torch_dispatch__:5698 2025-08-14T21:45:36.3948368Z Dynamo produced 1 graphs covering 378 ops with 0 graph breaks (0 unique) 2025-08-14T21:45:41.3811771Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:45:41.3812831Z from pkg_resources import resource_filename 2025-08-14T21:45:41.9484578Z 2025-08-14T21:45:43.4610047Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:45:43.4610614Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:45:43.4621944Z cpu eval GPT2ForSequenceClassification 2025-08-14T21:45:44.2523126Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:45:44.6078841Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:45:44.9344973Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:45:51.6058319Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6058670Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6058888Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6059112Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6059313Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6059508Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6059701Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6059893Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6060101Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6060315Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6060561Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6060778Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6061010Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6061384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6061721Z return mod(**inputs) 2025-08-14T21:45:51.6062118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1509, in forward 2025-08-14T21:45:51.6062546Z last_non_pad_token = (token_indices * non_pad_mask).argmax(-1) 2025-08-14T21:45:51.6062714Z 2025-08-14T21:45:51.6062817Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6063168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6063810Z return mod(**inputs) 2025-08-14T21:45:51.6064161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6064546Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6064928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6065299Z outputs = block( 2025-08-14T21:45:51.6065609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6065963Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6066344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6066714Z return func(*args, **kwargs) 2025-08-14T21:45:51.6067078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6067545Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6067943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6068318Z return func(*args, **kwargs) 2025-08-14T21:45:51.6068671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:45:51.6069142Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:45:51.6069635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6070020Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6070195Z 2025-08-14T21:45:51.6070274Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6070519Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6070723Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6070914Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6071148Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6071494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6071801Z return mod(**inputs) 2025-08-14T21:45:51.6072149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6072530Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6072909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6073264Z outputs = block( 2025-08-14T21:45:51.6073579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6073932Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6074290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6074655Z return func(*args, **kwargs) 2025-08-14T21:45:51.6075013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6075437Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6075797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6076144Z return func(*args, **kwargs) 2025-08-14T21:45:51.6076492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6076866Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6077287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:45:51.6077775Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:51.6077952Z 2025-08-14T21:45:51.6078062Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6078400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6078709Z return mod(**inputs) 2025-08-14T21:45:51.6079051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6079427Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6079789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6080145Z outputs = block( 2025-08-14T21:45:51.6080451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6080787Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6081171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6081538Z return func(*args, **kwargs) 2025-08-14T21:45:51.6081902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6082283Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6082661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6083025Z return func(*args, **kwargs) 2025-08-14T21:45:51.6083404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6083808Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6084272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:45:51.6084736Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:45:51.6084900Z 2025-08-14T21:45:51.6085009Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6085560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6085923Z return mod(**inputs) 2025-08-14T21:45:51.6086305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6086736Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6087112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6087470Z outputs = block( 2025-08-14T21:45:51.6087776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6088127Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6088494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6088860Z return func(*args, **kwargs) 2025-08-14T21:45:51.6089206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6089590Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6089963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6090316Z return func(*args, **kwargs) 2025-08-14T21:45:51.6090671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:45:51.6091051Z attn_output = self.c_proj(attn_output) 2025-08-14T21:45:51.6091405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6091818Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6091996Z 2025-08-14T21:45:51.6092097Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6092446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6092762Z return mod(**inputs) 2025-08-14T21:45:51.6093105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6093486Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6093858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6094210Z outputs = block( 2025-08-14T21:45:51.6094520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6094900Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6095268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6095624Z return func(*args, **kwargs) 2025-08-14T21:45:51.6095985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6096385Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6096774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:45:51.6097166Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:45:51.6097515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6097917Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6098088Z 2025-08-14T21:45:51.6098190Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6098540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6098853Z return mod(**inputs) 2025-08-14T21:45:51.6099199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6099576Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6099954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6100317Z outputs = block( 2025-08-14T21:45:51.6100620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6100973Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6101339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6101759Z return func(*args, **kwargs) 2025-08-14T21:45:51.6102109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6102504Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6102900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:45:51.6103268Z hidden_states = self.act(hidden_states) 2025-08-14T21:45:51.6103606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:45:51.6104053Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:45:51.6104282Z 2025-08-14T21:45:51.6104391Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6104736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6105134Z return mod(**inputs) 2025-08-14T21:45:51.6105474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6105841Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6106195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6106541Z outputs = block( 2025-08-14T21:45:51.6106839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6107172Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6107526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6107882Z return func(*args, **kwargs) 2025-08-14T21:45:51.6108226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6108657Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6109047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:45:51.6109430Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:45:51.6109779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6110154Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6110325Z 2025-08-14T21:45:51.6110446Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6110791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6111110Z return mod(**inputs) 2025-08-14T21:45:51.6111460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6111836Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6112214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6112566Z outputs = block( 2025-08-14T21:45:51.6112876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6113227Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6113588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6113958Z return func(*args, **kwargs) 2025-08-14T21:45:51.6114305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6114682Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6115035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6115391Z return func(*args, **kwargs) 2025-08-14T21:45:51.6115731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:45:51.6116197Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:45:51.6116627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6116999Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6117160Z 2025-08-14T21:45:51.6117244Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6117441Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6117642Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6117837Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6118075Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6118418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6118731Z return mod(**inputs) 2025-08-14T21:45:51.6119090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6119453Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6119812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6120160Z outputs = block( 2025-08-14T21:45:51.6120459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6120790Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6121145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6121510Z return func(*args, **kwargs) 2025-08-14T21:45:51.6121846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6122214Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6122571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6122915Z return func(*args, **kwargs) 2025-08-14T21:45:51.6123267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6123649Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6124083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:45:51.6124533Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:51.6124718Z 2025-08-14T21:45:51.6124818Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6125170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6125594Z return mod(**inputs) 2025-08-14T21:45:51.6125965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6126405Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6126830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6127235Z outputs = block( 2025-08-14T21:45:51.6127537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6127897Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6128274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6128641Z return func(*args, **kwargs) 2025-08-14T21:45:51.6129007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6129395Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6129778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6130138Z return func(*args, **kwargs) 2025-08-14T21:45:51.6130501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6130902Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6131334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:45:51.6131831Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:45:51.6132003Z 2025-08-14T21:45:51.6132109Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6132480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6132805Z return mod(**inputs) 2025-08-14T21:45:51.6133172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6133572Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6134213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6134584Z outputs = block( 2025-08-14T21:45:51.6134912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6135281Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6135658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6136050Z return func(*args, **kwargs) 2025-08-14T21:45:51.6136413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6136808Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6137164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6137529Z return func(*args, **kwargs) 2025-08-14T21:45:51.6138142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:45:51.6138520Z attn_output = self.c_proj(attn_output) 2025-08-14T21:45:51.6138883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6139272Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6139433Z 2025-08-14T21:45:51.6139538Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6139864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6140168Z return mod(**inputs) 2025-08-14T21:45:51.6140498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6140862Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6141210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6141553Z outputs = block( 2025-08-14T21:45:51.6141850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6142178Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6142528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6142869Z return func(*args, **kwargs) 2025-08-14T21:45:51.6143207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6143574Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6143949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:45:51.6144308Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:45:51.6144639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6144995Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6145161Z 2025-08-14T21:45:51.6145261Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6145637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6145932Z return mod(**inputs) 2025-08-14T21:45:51.6146266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6146635Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6146991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6147324Z outputs = block( 2025-08-14T21:45:51.6147624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6147954Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6148296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6148643Z return func(*args, **kwargs) 2025-08-14T21:45:51.6149016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6149398Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6149766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:45:51.6150123Z hidden_states = self.act(hidden_states) 2025-08-14T21:45:51.6150445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:45:51.6150892Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:45:51.6151113Z 2025-08-14T21:45:51.6151210Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6151614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6151939Z return mod(**inputs) 2025-08-14T21:45:51.6152297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6152682Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6153062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6153407Z outputs = block( 2025-08-14T21:45:51.6153706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6154047Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6154411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6154763Z return func(*args, **kwargs) 2025-08-14T21:45:51.6155115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6155506Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6155894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:45:51.6156265Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:45:51.6156610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6156986Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6157148Z 2025-08-14T21:45:51.6157254Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6157596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6157912Z return mod(**inputs) 2025-08-14T21:45:51.6158256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6158659Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6159020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6159370Z outputs = block( 2025-08-14T21:45:51.6159674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6160006Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6160358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6160711Z return func(*args, **kwargs) 2025-08-14T21:45:51.6161050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6161423Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6161786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6162164Z return func(*args, **kwargs) 2025-08-14T21:45:51.6162508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:45:51.6162977Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:45:51.6163415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6163791Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6163953Z 2025-08-14T21:45:51.6164052Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6164262Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6164463Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6164650Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6164889Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6165235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6165621Z return mod(**inputs) 2025-08-14T21:45:51.6165962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6166340Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6166709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6167056Z outputs = block( 2025-08-14T21:45:51.6167374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6167727Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6168094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6168453Z return func(*args, **kwargs) 2025-08-14T21:45:51.6168820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6169197Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6169558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6169909Z return func(*args, **kwargs) 2025-08-14T21:45:51.6170255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6170636Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6171052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:45:51.6171507Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:51.6171679Z 2025-08-14T21:45:51.6171784Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6172156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6172501Z return mod(**inputs) 2025-08-14T21:45:51.6172831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6173194Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6173540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6173880Z outputs = block( 2025-08-14T21:45:51.6174175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6174506Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6174846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6175192Z return func(*args, **kwargs) 2025-08-14T21:45:51.6175561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6175920Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6176278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6176622Z return func(*args, **kwargs) 2025-08-14T21:45:51.6176962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6177347Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6177759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:45:51.6178230Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:45:51.6178384Z 2025-08-14T21:45:51.6178487Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6178815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6179115Z return mod(**inputs) 2025-08-14T21:45:51.6179442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6179797Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6180153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6180492Z outputs = block( 2025-08-14T21:45:51.6180786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6181111Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6181458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6181803Z return func(*args, **kwargs) 2025-08-14T21:45:51.6182131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6182493Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6182844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6183181Z return func(*args, **kwargs) 2025-08-14T21:45:51.6183508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:45:51.6183867Z attn_output = self.c_proj(attn_output) 2025-08-14T21:45:51.6184196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6184561Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6184739Z 2025-08-14T21:45:51.6184837Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6185174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6185481Z return mod(**inputs) 2025-08-14T21:45:51.6185811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6186187Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6186546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6186893Z outputs = block( 2025-08-14T21:45:51.6187188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6187529Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6187883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6188246Z return func(*args, **kwargs) 2025-08-14T21:45:51.6188598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6188985Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6189366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:45:51.6189725Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:45:51.6190077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6190455Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6190615Z 2025-08-14T21:45:51.6190721Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6191070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6191380Z return mod(**inputs) 2025-08-14T21:45:51.6191714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6192079Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6192442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6192789Z outputs = block( 2025-08-14T21:45:51.6193092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6193428Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6193784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6194132Z return func(*args, **kwargs) 2025-08-14T21:45:51.6194472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6194859Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6195241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:45:51.6195605Z hidden_states = self.act(hidden_states) 2025-08-14T21:45:51.6195924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:45:51.6196345Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:45:51.6196562Z 2025-08-14T21:45:51.6196678Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6197017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6197323Z return mod(**inputs) 2025-08-14T21:45:51.6197670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6198080Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6198447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6198809Z outputs = block( 2025-08-14T21:45:51.6199121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6199470Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6199828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6200188Z return func(*args, **kwargs) 2025-08-14T21:45:51.6200547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6200936Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6201333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:45:51.6201735Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:45:51.6202084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6202461Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6202634Z 2025-08-14T21:45:51.6202733Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6203097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6203416Z return mod(**inputs) 2025-08-14T21:45:51.6203760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6204173Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6204552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6204914Z outputs = block( 2025-08-14T21:45:51.6205227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6205659Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6206056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6206453Z return func(*args, **kwargs) 2025-08-14T21:45:51.6206839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-08-14T21:45:51.6207266Z hidden_states = residual + feed_forward_hidden_states 2025-08-14T21:45:51.6207421Z 2025-08-14T21:45:51.6207522Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6207880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6208197Z return mod(**inputs) 2025-08-14T21:45:51.6208546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6208924Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6209299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6209664Z outputs = block( 2025-08-14T21:45:51.6209968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6210321Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6210683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6211053Z return func(*args, **kwargs) 2025-08-14T21:45:51.6211401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6211808Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6212175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6212532Z return func(*args, **kwargs) 2025-08-14T21:45:51.6212880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:45:51.6213366Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:45:51.6213818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6214195Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6214368Z 2025-08-14T21:45:51.6214451Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6214662Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6214883Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6215075Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6215298Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6215644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6215951Z return mod(**inputs) 2025-08-14T21:45:51.6216383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6216781Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6217185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6217548Z outputs = block( 2025-08-14T21:45:51.6217924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6218292Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6218654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6219017Z return func(*args, **kwargs) 2025-08-14T21:45:51.6219373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6219757Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6220128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6220487Z return func(*args, **kwargs) 2025-08-14T21:45:51.6220844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6221238Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6221670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:45:51.6222133Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:51.6222304Z 2025-08-14T21:45:51.6222410Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6222745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6223055Z return mod(**inputs) 2025-08-14T21:45:51.6223395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6223766Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6224123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6224474Z outputs = block( 2025-08-14T21:45:51.6224781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6225138Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6225492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6225844Z return func(*args, **kwargs) 2025-08-14T21:45:51.6226191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6226560Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6226923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6227291Z return func(*args, **kwargs) 2025-08-14T21:45:51.6227656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6228051Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6228495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:45:51.6228980Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:45:51.6229132Z 2025-08-14T21:45:51.6229230Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6229572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6229877Z return mod(**inputs) 2025-08-14T21:45:51.6230215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6230595Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6230964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6231329Z outputs = block( 2025-08-14T21:45:51.6231629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6231972Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6232328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6232676Z return func(*args, **kwargs) 2025-08-14T21:45:51.6233018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6233393Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6233754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6234103Z return func(*args, **kwargs) 2025-08-14T21:45:51.6234439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:45:51.6234807Z attn_output = self.c_proj(attn_output) 2025-08-14T21:45:51.6235149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6235514Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6235683Z 2025-08-14T21:45:51.6235782Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6236123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6236424Z return mod(**inputs) 2025-08-14T21:45:51.6236759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6237132Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6237493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6238028Z outputs = block( 2025-08-14T21:45:51.6238339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6238722Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6239076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6239415Z return func(*args, **kwargs) 2025-08-14T21:45:51.6239758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6240140Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6240516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:45:51.6240891Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:45:51.6241256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6241670Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6241895Z 2025-08-14T21:45:51.6242006Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6242387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6242733Z return mod(**inputs) 2025-08-14T21:45:51.6243112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6243522Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6243958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6244356Z outputs = block( 2025-08-14T21:45:51.6244686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6245092Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6245558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6245976Z return func(*args, **kwargs) 2025-08-14T21:45:51.6246367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6246812Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6247260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:45:51.6247667Z hidden_states = self.act(hidden_states) 2025-08-14T21:45:51.6248029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:45:51.6248508Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:45:51.6248756Z 2025-08-14T21:45:51.6248876Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6249254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6249602Z return mod(**inputs) 2025-08-14T21:45:51.6249983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6250408Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6250803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6251193Z outputs = block( 2025-08-14T21:45:51.6251532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6251903Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6252301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6252680Z return func(*args, **kwargs) 2025-08-14T21:45:51.6253025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6253400Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6253778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:45:51.6254151Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:45:51.6254495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6254859Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6255026Z 2025-08-14T21:45:51.6255123Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6255464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6255765Z return mod(**inputs) 2025-08-14T21:45:51.6256134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6256546Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6256908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6257263Z outputs = block( 2025-08-14T21:45:51.6257581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6257942Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6258337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6258700Z return func(*args, **kwargs) 2025-08-14T21:45:51.6259075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6259452Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6259815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6260167Z return func(*args, **kwargs) 2025-08-14T21:45:51.6260513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:45:51.6260975Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:45:51.6261407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6261784Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6261946Z 2025-08-14T21:45:51.6262034Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6262232Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6262432Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6262633Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6262853Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6263184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6263493Z return mod(**inputs) 2025-08-14T21:45:51.6263828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6264200Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6264566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6264915Z outputs = block( 2025-08-14T21:45:51.6265217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6265553Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6265923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6266277Z return func(*args, **kwargs) 2025-08-14T21:45:51.6266606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6266968Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6267321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6267671Z return func(*args, **kwargs) 2025-08-14T21:45:51.6268009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6268389Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6268801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:45:51.6269255Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:51.6269437Z 2025-08-14T21:45:51.6269533Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6269870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6270174Z return mod(**inputs) 2025-08-14T21:45:51.6270502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6270870Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6271252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6271604Z outputs = block( 2025-08-14T21:45:51.6271951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6272296Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6272654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6272997Z return func(*args, **kwargs) 2025-08-14T21:45:51.6273347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6273724Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6274089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6274424Z return func(*args, **kwargs) 2025-08-14T21:45:51.6274765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6275135Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6275538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:45:51.6275964Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:45:51.6276121Z 2025-08-14T21:45:51.6276216Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6276547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6276840Z return mod(**inputs) 2025-08-14T21:45:51.6277230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6277606Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6277971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6278317Z outputs = block( 2025-08-14T21:45:51.6278623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6278990Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6279342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6279702Z return func(*args, **kwargs) 2025-08-14T21:45:51.6280058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6280439Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6280808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6281180Z return func(*args, **kwargs) 2025-08-14T21:45:51.6281550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:45:51.6281939Z attn_output = self.c_proj(attn_output) 2025-08-14T21:45:51.6282283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6282695Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6282862Z 2025-08-14T21:45:51.6282973Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6283319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6283640Z return mod(**inputs) 2025-08-14T21:45:51.6283994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6284386Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6284773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6285148Z outputs = block( 2025-08-14T21:45:51.6285581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6285959Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6286345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6286724Z return func(*args, **kwargs) 2025-08-14T21:45:51.6287099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6287502Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6287909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:45:51.6288305Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:45:51.6288658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6289062Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6289240Z 2025-08-14T21:45:51.6289344Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6289705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6290025Z return mod(**inputs) 2025-08-14T21:45:51.6290382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6290778Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6291168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6291534Z outputs = block( 2025-08-14T21:45:51.6291856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6292217Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6292588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6292984Z return func(*args, **kwargs) 2025-08-14T21:45:51.6293349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6293755Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6294151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:45:51.6294533Z hidden_states = self.act(hidden_states) 2025-08-14T21:45:51.6294879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:45:51.6295322Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:45:51.6295560Z 2025-08-14T21:45:51.6295665Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6296026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6296372Z return mod(**inputs) 2025-08-14T21:45:51.6296698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6297064Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6297425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6297779Z outputs = block( 2025-08-14T21:45:51.6298075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6298434Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6298793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6299147Z return func(*args, **kwargs) 2025-08-14T21:45:51.6299491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6299865Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6300238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:45:51.6300592Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:45:51.6300922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6301282Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6301440Z 2025-08-14T21:45:51.6301543Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6301866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6302166Z return mod(**inputs) 2025-08-14T21:45:51.6302493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6302853Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6303209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6303550Z outputs = block( 2025-08-14T21:45:51.6303840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6304163Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6304509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6304849Z return func(*args, **kwargs) 2025-08-14T21:45:51.6305180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-08-14T21:45:51.6305560Z hidden_states = residual + feed_forward_hidden_states 2025-08-14T21:45:51.6305730Z 2025-08-14T21:45:51.6305827Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6306153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6306444Z return mod(**inputs) 2025-08-14T21:45:51.6306774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6307153Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6307516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6307876Z outputs = block( 2025-08-14T21:45:51.6308188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6308536Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6308892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6309285Z return func(*args, **kwargs) 2025-08-14T21:45:51.6309641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6310005Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6310355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6310699Z return func(*args, **kwargs) 2025-08-14T21:45:51.6311049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:45:51.6311504Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:45:51.6311958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6312328Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6312487Z 2025-08-14T21:45:51.6312570Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6312764Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6312958Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6313149Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6313358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6313685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6313980Z return mod(**inputs) 2025-08-14T21:45:51.6314311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6314663Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6315019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6315361Z outputs = block( 2025-08-14T21:45:51.6315653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6315985Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6316331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6316672Z return func(*args, **kwargs) 2025-08-14T21:45:51.6317000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6317364Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6317719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6318053Z return func(*args, **kwargs) 2025-08-14T21:45:51.6318388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6318779Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6319184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:45:51.6319614Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:51.6319783Z 2025-08-14T21:45:51.6319878Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6320207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6320502Z return mod(**inputs) 2025-08-14T21:45:51.6320823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6321183Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6321538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6321906Z outputs = block( 2025-08-14T21:45:51.6322211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6322554Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6322916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6323270Z return func(*args, **kwargs) 2025-08-14T21:45:51.6323629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6324079Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6324483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6324847Z return func(*args, **kwargs) 2025-08-14T21:45:51.6325221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6325696Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6326143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:45:51.6326615Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:45:51.6326782Z 2025-08-14T21:45:51.6326886Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6327247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6327563Z return mod(**inputs) 2025-08-14T21:45:51.6327919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6328311Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6328689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6329059Z outputs = block( 2025-08-14T21:45:51.6329378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6329733Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6330104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6330469Z return func(*args, **kwargs) 2025-08-14T21:45:51.6330825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6331213Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6331582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6331945Z return func(*args, **kwargs) 2025-08-14T21:45:51.6332303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:45:51.6332700Z attn_output = self.c_proj(attn_output) 2025-08-14T21:45:51.6333050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6333439Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6333603Z 2025-08-14T21:45:51.6333709Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6334053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6334373Z return mod(**inputs) 2025-08-14T21:45:51.6334722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6335101Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6335480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6335859Z outputs = block( 2025-08-14T21:45:51.6336166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6336506Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6336866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6337232Z return func(*args, **kwargs) 2025-08-14T21:45:51.6337596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6338147Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6338534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:45:51.6338943Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:45:51.6339275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6339650Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6339817Z 2025-08-14T21:45:51.6339916Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6340258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6340559Z return mod(**inputs) 2025-08-14T21:45:51.6340897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6341272Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6341627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6341982Z outputs = block( 2025-08-14T21:45:51.6342284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6342630Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6342977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6343326Z return func(*args, **kwargs) 2025-08-14T21:45:51.6343674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6344052Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6344434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:45:51.6344798Z hidden_states = self.act(hidden_states) 2025-08-14T21:45:51.6345124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:45:51.6345543Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:45:51.6345797Z 2025-08-14T21:45:51.6345894Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6346235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6346544Z return mod(**inputs) 2025-08-14T21:45:51.6346906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6347286Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6347666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6348005Z outputs = block( 2025-08-14T21:45:51.6348309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6348648Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6349001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6349371Z return func(*args, **kwargs) 2025-08-14T21:45:51.6349717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6350104Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6350482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:45:51.6350849Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:45:51.6351211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6351590Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6351750Z 2025-08-14T21:45:51.6351864Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6352211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6352519Z return mod(**inputs) 2025-08-14T21:45:51.6352856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6353221Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6353583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6353937Z outputs = block( 2025-08-14T21:45:51.6354239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6354580Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6354934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6355283Z return func(*args, **kwargs) 2025-08-14T21:45:51.6355625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6355999Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6356363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6356715Z return func(*args, **kwargs) 2025-08-14T21:45:51.6357060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:45:51.6357515Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:45:51.6357936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6358297Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6358466Z 2025-08-14T21:45:51.6358573Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6358781Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6358988Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6359174Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6359390Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6359726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6360028Z return mod(**inputs) 2025-08-14T21:45:51.6360372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6360749Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6361114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6361463Z outputs = block( 2025-08-14T21:45:51.6361765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6362132Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6362478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6362832Z return func(*args, **kwargs) 2025-08-14T21:45:51.6363178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6363553Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6363924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6364275Z return func(*args, **kwargs) 2025-08-14T21:45:51.6364633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6365016Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6365499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:45:51.6365973Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:51.6366154Z 2025-08-14T21:45:51.6366268Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6366627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6366944Z return mod(**inputs) 2025-08-14T21:45:51.6367300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6367671Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6368035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6368386Z outputs = block( 2025-08-14T21:45:51.6368688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6369023Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6369370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6369713Z return func(*args, **kwargs) 2025-08-14T21:45:51.6370056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6370415Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6370779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6371113Z return func(*args, **kwargs) 2025-08-14T21:45:51.6371439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6371832Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6372237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:45:51.6372655Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:45:51.6372803Z 2025-08-14T21:45:51.6372898Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6373229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6373530Z return mod(**inputs) 2025-08-14T21:45:51.6373860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6374213Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6374570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6374912Z outputs = block( 2025-08-14T21:45:51.6375217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6375549Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6375894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6376234Z return func(*args, **kwargs) 2025-08-14T21:45:51.6376563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6376926Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6377298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6377636Z return func(*args, **kwargs) 2025-08-14T21:45:51.6377990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:45:51.6378363Z attn_output = self.c_proj(attn_output) 2025-08-14T21:45:51.6378698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6379066Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6379232Z 2025-08-14T21:45:51.6379332Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6379671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6379976Z return mod(**inputs) 2025-08-14T21:45:51.6380307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6380679Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6381042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6381388Z outputs = block( 2025-08-14T21:45:51.6381692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6382035Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6382396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6382743Z return func(*args, **kwargs) 2025-08-14T21:45:51.6383095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6383484Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6383861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:45:51.6384231Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:45:51.6384571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6384970Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6385132Z 2025-08-14T21:45:51.6385231Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6385570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6385879Z return mod(**inputs) 2025-08-14T21:45:51.6386215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6386579Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6386939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6387341Z outputs = block( 2025-08-14T21:45:51.6387634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6387975Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6388359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6388719Z return func(*args, **kwargs) 2025-08-14T21:45:51.6389068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6389467Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6389851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:45:51.6390256Z hidden_states = self.act(hidden_states) 2025-08-14T21:45:51.6390584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:45:51.6391025Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:45:51.6391244Z 2025-08-14T21:45:51.6391361Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6391690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6391993Z return mod(**inputs) 2025-08-14T21:45:51.6392331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6392700Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6393049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6393397Z outputs = block( 2025-08-14T21:45:51.6393696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6394025Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6394377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6394726Z return func(*args, **kwargs) 2025-08-14T21:45:51.6395068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6395435Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6395820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:45:51.6396187Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:45:51.6396515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6396884Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6397045Z 2025-08-14T21:45:51.6397142Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6397476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6397789Z return mod(**inputs) 2025-08-14T21:45:51.6398116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6398475Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6398828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6399161Z outputs = block( 2025-08-14T21:45:51.6399456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6399790Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6400127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6400467Z return func(*args, **kwargs) 2025-08-14T21:45:51.6400802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-08-14T21:45:51.6401208Z hidden_states = residual + feed_forward_hidden_states 2025-08-14T21:45:51.6401356Z 2025-08-14T21:45:51.6401452Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6401800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6402114Z return mod(**inputs) 2025-08-14T21:45:51.6402451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6402840Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6403213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6403560Z outputs = block( 2025-08-14T21:45:51.6403878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6404228Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6404580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6404929Z return func(*args, **kwargs) 2025-08-14T21:45:51.6405269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6405739Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6406151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6406528Z return func(*args, **kwargs) 2025-08-14T21:45:51.6406877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:45:51.6407360Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:45:51.6407790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6408153Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6408320Z 2025-08-14T21:45:51.6408397Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6408602Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6408790Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6408990Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6409209Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6409543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6409837Z return mod(**inputs) 2025-08-14T21:45:51.6410170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6410532Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6410899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6411245Z outputs = block( 2025-08-14T21:45:51.6411540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6411869Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6412209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6412547Z return func(*args, **kwargs) 2025-08-14T21:45:51.6412883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6413240Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6413595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6413933Z return func(*args, **kwargs) 2025-08-14T21:45:51.6414286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6414648Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6415054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:45:51.6415493Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:51.6415663Z 2025-08-14T21:45:51.6415769Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6416117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6416432Z return mod(**inputs) 2025-08-14T21:45:51.6416794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6417154Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6417521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6417869Z outputs = block( 2025-08-14T21:45:51.6418169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6418499Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6418853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6418920Z return func(*args, **kwargs) 2025-08-14T21:45:51.6419159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6419242Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6419472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6419541Z return func(*args, **kwargs) 2025-08-14T21:45:51.6419768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6419863Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6420129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:45:51.6420233Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:45:51.6420244Z 2025-08-14T21:45:51.6420342Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6420531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6420600Z return mod(**inputs) 2025-08-14T21:45:51.6420837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6420935Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6421171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6421233Z outputs = block( 2025-08-14T21:45:51.6421443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6421518Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6421742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6421817Z return func(*args, **kwargs) 2025-08-14T21:45:51.6422048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6422130Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6422363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6422449Z return func(*args, **kwargs) 2025-08-14T21:45:51.6422688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:45:51.6422766Z attn_output = self.c_proj(attn_output) 2025-08-14T21:45:51.6422971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6423094Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6423098Z 2025-08-14T21:45:51.6423211Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6423409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6423472Z return mod(**inputs) 2025-08-14T21:45:51.6423721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6423812Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6424047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6424108Z outputs = block( 2025-08-14T21:45:51.6424323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6424399Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6424631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6424699Z return func(*args, **kwargs) 2025-08-14T21:45:51.6424929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6425033Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6425263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:45:51.6425343Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:45:51.6425557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6425668Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6425671Z 2025-08-14T21:45:51.6425775Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6425963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6426024Z return mod(**inputs) 2025-08-14T21:45:51.6426271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6426350Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6426588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6426668Z outputs = block( 2025-08-14T21:45:51.6426876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6426954Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6427179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6427245Z return func(*args, **kwargs) 2025-08-14T21:45:51.6427482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6427581Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6427816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:45:51.6427893Z hidden_states = self.act(hidden_states) 2025-08-14T21:45:51.6428092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:45:51.6428285Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:45:51.6428288Z 2025-08-14T21:45:51.6428385Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6428579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6428641Z return mod(**inputs) 2025-08-14T21:45:51.6428873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6428972Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6429205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6429280Z outputs = block( 2025-08-14T21:45:51.6429499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6429579Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6429816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6429882Z return func(*args, **kwargs) 2025-08-14T21:45:51.6430118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6430221Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6430460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:45:51.6430543Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:45:51.6430760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6430878Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6430882Z 2025-08-14T21:45:51.6430989Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6431181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6431244Z return mod(**inputs) 2025-08-14T21:45:51.6431497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6431575Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6431815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6431875Z outputs = block( 2025-08-14T21:45:51.6432081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6432165Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6432407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6432473Z return func(*args, **kwargs) 2025-08-14T21:45:51.6432714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6432797Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6433029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6433092Z return func(*args, **kwargs) 2025-08-14T21:45:51.6433327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:45:51.6433509Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:45:51.6433715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6433851Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6433855Z 2025-08-14T21:45:51.6433932Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6434008Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6434089Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6434162Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6434258Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6434456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6434519Z return mod(**inputs) 2025-08-14T21:45:51.6434779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6434866Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6435113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6435183Z outputs = block( 2025-08-14T21:45:51.6435394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6435468Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6435703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6435769Z return func(*args, **kwargs) 2025-08-14T21:45:51.6436008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6436092Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6436318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6436392Z return func(*args, **kwargs) 2025-08-14T21:45:51.6436624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6436718Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6437000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:45:51.6437121Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:51.6437125Z 2025-08-14T21:45:51.6437229Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6437415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6437480Z return mod(**inputs) 2025-08-14T21:45:51.6437838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6437925Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6438162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6438265Z outputs = block( 2025-08-14T21:45:51.6438474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6438557Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6438783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6438849Z return func(*args, **kwargs) 2025-08-14T21:45:51.6439088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6439170Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6439402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6439467Z return func(*args, **kwargs) 2025-08-14T21:45:51.6439699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6439820Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6440094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:45:51.6440200Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:45:51.6440213Z 2025-08-14T21:45:51.6440311Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6440525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6440601Z return mod(**inputs) 2025-08-14T21:45:51.6440834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6440931Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6441172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6441234Z outputs = block( 2025-08-14T21:45:51.6441443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6441517Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6441740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6441811Z return func(*args, **kwargs) 2025-08-14T21:45:51.6442043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6442125Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6442359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6442426Z return func(*args, **kwargs) 2025-08-14T21:45:51.6442666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:45:51.6442743Z attn_output = self.c_proj(attn_output) 2025-08-14T21:45:51.6442945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6443061Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6443065Z 2025-08-14T21:45:51.6443160Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6443357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6443421Z return mod(**inputs) 2025-08-14T21:45:51.6443663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6443750Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6444004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6444068Z outputs = block( 2025-08-14T21:45:51.6444285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6444361Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6444598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6444666Z return func(*args, **kwargs) 2025-08-14T21:45:51.6444907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6445012Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6445256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:45:51.6445337Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:45:51.6445638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6445753Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6445757Z 2025-08-14T21:45:51.6445865Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6446063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6446128Z return mod(**inputs) 2025-08-14T21:45:51.6446400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6446481Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6446745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6446805Z outputs = block( 2025-08-14T21:45:51.6447011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6447092Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6447313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6447377Z return func(*args, **kwargs) 2025-08-14T21:45:51.6447661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6447758Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6447996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:45:51.6448072Z hidden_states = self.act(hidden_states) 2025-08-14T21:45:51.6448271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:45:51.6448450Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:45:51.6448454Z 2025-08-14T21:45:51.6448548Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6448743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6448804Z return mod(**inputs) 2025-08-14T21:45:51.6449045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6449125Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6449353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6449411Z outputs = block( 2025-08-14T21:45:51.6449622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6449719Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6449947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6450010Z return func(*args, **kwargs) 2025-08-14T21:45:51.6450236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6450337Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6450560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:45:51.6450640Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:45:51.6450845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6450954Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6450959Z 2025-08-14T21:45:51.6451060Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6451261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6451322Z return mod(**inputs) 2025-08-14T21:45:51.6451559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6451635Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6451866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6451925Z outputs = block( 2025-08-14T21:45:51.6452142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6452223Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6452454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6452523Z return func(*args, **kwargs) 2025-08-14T21:45:51.6452756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-08-14T21:45:51.6452854Z hidden_states = residual + feed_forward_hidden_states 2025-08-14T21:45:51.6452858Z 2025-08-14T21:45:51.6452959Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6453142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6453204Z return mod(**inputs) 2025-08-14T21:45:51.6453438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6453514Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6453739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6453807Z outputs = block( 2025-08-14T21:45:51.6454008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6454089Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6454307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6454377Z return func(*args, **kwargs) 2025-08-14T21:45:51.6454612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6454692Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6454919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6454984Z return func(*args, **kwargs) 2025-08-14T21:45:51.6455209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:45:51.6455406Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:45:51.6455605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6455713Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6455724Z 2025-08-14T21:45:51.6455800Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6455874Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6455951Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6456022Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6456116Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6456303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6456365Z return mod(**inputs) 2025-08-14T21:45:51.6456592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6456699Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6456926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6456992Z outputs = block( 2025-08-14T21:45:51.6457192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6457261Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6457511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6457580Z return func(*args, **kwargs) 2025-08-14T21:45:51.6457837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6457932Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6458164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6458236Z return func(*args, **kwargs) 2025-08-14T21:45:51.6458471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6458565Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6458853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:45:51.6458977Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:51.6458980Z 2025-08-14T21:45:51.6459083Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6459275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6459339Z return mod(**inputs) 2025-08-14T21:45:51.6459587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6459668Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6459910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6459977Z outputs = block( 2025-08-14T21:45:51.6460191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6460269Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6460506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6460572Z return func(*args, **kwargs) 2025-08-14T21:45:51.6460820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6460922Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6461156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6461222Z return func(*args, **kwargs) 2025-08-14T21:45:51.6461454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6461551Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6461825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:45:51.6461931Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:45:51.6461935Z 2025-08-14T21:45:51.6462040Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6462231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6462303Z return mod(**inputs) 2025-08-14T21:45:51.6462567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6462646Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6462885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6462948Z outputs = block( 2025-08-14T21:45:51.6463156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6463238Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6463483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6463560Z return func(*args, **kwargs) 2025-08-14T21:45:51.6463813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6463901Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6464142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6464207Z return func(*args, **kwargs) 2025-08-14T21:45:51.6464454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:45:51.6464533Z attn_output = self.c_proj(attn_output) 2025-08-14T21:45:51.6464744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6464866Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6464870Z 2025-08-14T21:45:51.6464969Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6465163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6465235Z return mod(**inputs) 2025-08-14T21:45:51.6465480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6465566Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6465805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6465867Z outputs = block( 2025-08-14T21:45:51.6466086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6466163Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6466403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6466472Z return func(*args, **kwargs) 2025-08-14T21:45:51.6466713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6466838Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6467079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:45:51.6467156Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:45:51.6467373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6467484Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6467488Z 2025-08-14T21:45:51.6467593Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6467787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6467849Z return mod(**inputs) 2025-08-14T21:45:51.6468088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6468165Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6468412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6468480Z outputs = block( 2025-08-14T21:45:51.6468685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6468764Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6468989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6469069Z return func(*args, **kwargs) 2025-08-14T21:45:51.6469308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6469402Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6469663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:45:51.6469741Z hidden_states = self.act(hidden_states) 2025-08-14T21:45:51.6469934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:45:51.6470105Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:45:51.6470108Z 2025-08-14T21:45:51.6470201Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6470427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6470507Z return mod(**inputs) 2025-08-14T21:45:51.6470736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6470819Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6471040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6471103Z outputs = block( 2025-08-14T21:45:51.6471320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6471398Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6471641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6471711Z return func(*args, **kwargs) 2025-08-14T21:45:51.6471971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6472086Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6472349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:45:51.6472441Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:45:51.6472705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6472828Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6472832Z 2025-08-14T21:45:51.6472948Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6473155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6473224Z return mod(**inputs) 2025-08-14T21:45:51.6473490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6473573Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6473812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6473883Z outputs = block( 2025-08-14T21:45:51.6474105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6474203Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6474433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6474501Z return func(*args, **kwargs) 2025-08-14T21:45:51.6474746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6474831Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6475084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6475153Z return func(*args, **kwargs) 2025-08-14T21:45:51.6475436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:45:51.6475642Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:45:51.6475875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6475996Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6476007Z 2025-08-14T21:45:51.6476094Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6476177Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6476264Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6476343Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6476451Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6476668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6476737Z return mod(**inputs) 2025-08-14T21:45:51.6477000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6477095Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6477325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6477393Z outputs = block( 2025-08-14T21:45:51.6477626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6477696Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6477918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6477982Z return func(*args, **kwargs) 2025-08-14T21:45:51.6478207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6478295Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6478515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6478614Z return func(*args, **kwargs) 2025-08-14T21:45:51.6478839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6478929Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6479202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:45:51.6479319Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:51.6479322Z 2025-08-14T21:45:51.6479426Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6479611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6479671Z return mod(**inputs) 2025-08-14T21:45:51.6479905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6480001Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6480232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6480298Z outputs = block( 2025-08-14T21:45:51.6480502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6480582Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6480811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6480894Z return func(*args, **kwargs) 2025-08-14T21:45:51.6481140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6481239Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6481478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6481548Z return func(*args, **kwargs) 2025-08-14T21:45:51.6481781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6481882Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6482159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:45:51.6482261Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:45:51.6482265Z 2025-08-14T21:45:51.6482367Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6482553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6482625Z return mod(**inputs) 2025-08-14T21:45:51.6482858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6482937Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6483170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6483230Z outputs = block( 2025-08-14T21:45:51.6483434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6483514Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6483744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6483816Z return func(*args, **kwargs) 2025-08-14T21:45:51.6484051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6484136Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6485133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6485204Z return func(*args, **kwargs) 2025-08-14T21:45:51.6485535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:45:51.6485628Z attn_output = self.c_proj(attn_output) 2025-08-14T21:45:51.6485857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6485988Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6485996Z 2025-08-14T21:45:51.6486104Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6486319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6486397Z return mod(**inputs) 2025-08-14T21:45:51.6486671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6486791Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6487028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6487092Z outputs = block( 2025-08-14T21:45:51.6487321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6487394Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6487636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6487704Z return func(*args, **kwargs) 2025-08-14T21:45:51.6487949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6488057Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6488288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:45:51.6488362Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:45:51.6488572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6488681Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6488685Z 2025-08-14T21:45:51.6488787Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6488974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6489036Z return mod(**inputs) 2025-08-14T21:45:51.6489276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6489354Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6489585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6489654Z outputs = block( 2025-08-14T21:45:51.6489861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6489939Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6490166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6490233Z return func(*args, **kwargs) 2025-08-14T21:45:51.6490470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6490563Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6490800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:45:51.6490892Z hidden_states = self.act(hidden_states) 2025-08-14T21:45:51.6491089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:45:51.6491260Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:45:51.6491264Z 2025-08-14T21:45:51.6491358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6491547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6491615Z return mod(**inputs) 2025-08-14T21:45:51.6491849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6491932Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6492164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6492225Z outputs = block( 2025-08-14T21:45:51.6492453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6492525Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6492755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6492821Z return func(*args, **kwargs) 2025-08-14T21:45:51.6493048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6493150Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6493391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:45:51.6493474Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:45:51.6493700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6493812Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6493815Z 2025-08-14T21:45:51.6493917Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6494105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6494168Z return mod(**inputs) 2025-08-14T21:45:51.6494410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6494487Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6494716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6494784Z outputs = block( 2025-08-14T21:45:51.6494993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6495075Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6495296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6495361Z return func(*args, **kwargs) 2025-08-14T21:45:51.6495597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 442, in forward 2025-08-14T21:45:51.6495697Z hidden_states = residual + feed_forward_hidden_states 2025-08-14T21:45:51.6495700Z 2025-08-14T21:45:51.6495804Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6495991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6496053Z return mod(**inputs) 2025-08-14T21:45:51.6496293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6496370Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6496617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6496685Z outputs = block( 2025-08-14T21:45:51.6496889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6496968Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6497191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6497256Z return func(*args, **kwargs) 2025-08-14T21:45:51.6497493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6497576Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6497798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6497871Z return func(*args, **kwargs) 2025-08-14T21:45:51.6498147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 294, in forward 2025-08-14T21:45:51.6498326Z query_states, key_states, value_states = self.c_attn(hidden_states).split(self.split_size, dim=2) 2025-08-14T21:45:51.6498531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6498641Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6498644Z 2025-08-14T21:45:51.6498728Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6498819Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6498902Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6498973Z cudagraph partition due to non gpu ops 2025-08-14T21:45:51.6499090Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6499286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6499350Z return mod(**inputs) 2025-08-14T21:45:51.6499585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6499670Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6499901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6499970Z outputs = block( 2025-08-14T21:45:51.6500173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6500247Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6500477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6500543Z return func(*args, **kwargs) 2025-08-14T21:45:51.6500772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6500864Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6501094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6501165Z return func(*args, **kwargs) 2025-08-14T21:45:51.6501385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6501472Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6501744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:45:51.6501860Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:45:51.6501863Z 2025-08-14T21:45:51.6501965Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6502161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6502223Z return mod(**inputs) 2025-08-14T21:45:51.6502458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6502533Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6502754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6502821Z outputs = block( 2025-08-14T21:45:51.6503020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6503096Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6503313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6503377Z return func(*args, **kwargs) 2025-08-14T21:45:51.6503615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6503714Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6503946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6504016Z return func(*args, **kwargs) 2025-08-14T21:45:51.6504240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 336, in forward 2025-08-14T21:45:51.6504332Z attn_output, attn_weights = attention_interface( 2025-08-14T21:45:51.6504610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:45:51.6504715Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:45:51.6504733Z 2025-08-14T21:45:51.6504838Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6505027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6505096Z return mod(**inputs) 2025-08-14T21:45:51.6505337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6505411Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6505640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6505699Z outputs = block( 2025-08-14T21:45:51.6505900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6505979Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6506198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6506270Z return func(*args, **kwargs) 2025-08-14T21:45:51.6506493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 404, in forward 2025-08-14T21:45:51.6506571Z attn_output, self_attn_weights = self.attn( 2025-08-14T21:45:51.6506795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6506857Z return func(*args, **kwargs) 2025-08-14T21:45:51.6507088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 349, in forward 2025-08-14T21:45:51.6507165Z attn_output = self.c_proj(attn_output) 2025-08-14T21:45:51.6507362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6507475Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6507480Z 2025-08-14T21:45:51.6507574Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6507772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6507840Z return mod(**inputs) 2025-08-14T21:45:51.6508067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6508154Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6508385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6508445Z outputs = block( 2025-08-14T21:45:51.6508669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6508741Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6508959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6509033Z return func(*args, **kwargs) 2025-08-14T21:45:51.6509272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6509373Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6509598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 365, in forward 2025-08-14T21:45:51.6509671Z hidden_states = self.c_fc(hidden_states) 2025-08-14T21:45:51.6509876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6509997Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6510001Z 2025-08-14T21:45:51.6510102Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6510298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6510360Z return mod(**inputs) 2025-08-14T21:45:51.6510602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6510677Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6510900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6510966Z outputs = block( 2025-08-14T21:45:51.6511175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6511254Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6511480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6511546Z return func(*args, **kwargs) 2025-08-14T21:45:51.6511794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6511889Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6512113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 366, in forward 2025-08-14T21:45:51.6512198Z hidden_states = self.act(hidden_states) 2025-08-14T21:45:51.6512394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:45:51.6512569Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:45:51.6512572Z 2025-08-14T21:45:51.6512671Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6512866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6512934Z return mod(**inputs) 2025-08-14T21:45:51.6513162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1480, in forward 2025-08-14T21:45:51.6513260Z transformer_outputs = self.transformer( 2025-08-14T21:45:51.6513486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 917, in forward 2025-08-14T21:45:51.6513547Z outputs = block( 2025-08-14T21:45:51.6513758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:45:51.6513830Z return super().__call__(*args, **kwargs) 2025-08-14T21:45:51.6514055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:45:51.6514128Z return func(*args, **kwargs) 2025-08-14T21:45:51.6514358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 440, in forward 2025-08-14T21:45:51.6514461Z feed_forward_hidden_states = self.mlp(hidden_states) 2025-08-14T21:45:51.6514688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 367, in forward 2025-08-14T21:45:51.6514789Z hidden_states = self.c_proj(hidden_states) 2025-08-14T21:45:51.6514997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 116, in forward 2025-08-14T21:45:51.6515106Z x = torch.addmm(self.bias, x.view(-1, x.size(-1)), self.weight) 2025-08-14T21:45:51.6515109Z 2025-08-14T21:45:51.6515223Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6515404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6515480Z return mod(**inputs) 2025-08-14T21:45:51.6515719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1494, in forward 2025-08-14T21:45:51.6515789Z logits = self.score(hidden_states) 2025-08-14T21:45:51.6515809Z 2025-08-14T21:45:51.6515903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6516092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6516153Z return mod(**inputs) 2025-08-14T21:45:51.6516386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1537, in forward 2025-08-14T21:45:51.6516517Z loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1)) 2025-08-14T21:45:51.6516521Z 2025-08-14T21:45:51.6516613Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:45:51.6516797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:45:51.6516859Z return mod(**inputs) 2025-08-14T21:45:51.6517091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1537, in forward 2025-08-14T21:45:51.6517219Z loss = loss_fct(pooled_logits.view(-1, self.num_labels), labels.view(-1)) 2025-08-14T21:45:51.6517224Z 2025-08-14T21:46:02.1927487Z Compilation time (from dynamo_timed): 15.882111102 2025-08-14T21:46:02.1927973Z pass 2025-08-14T21:46:02.1928434Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:46:02.1929789Z TIMING: _recursive_pre_grad_passes:0.01299 _recursive_joint_graph_passes:0.57039 _recursive_post_grad_passes:0.08015 async_compile.wait:0.67849 code_gen:7.76421 inductor_compile:8.87874 backend_compile:11.96183 gc:0.00072 entire_frame_compile:15.88211 total_wall_time:15.88211 2025-08-14T21:46:02.1930792Z STATS: call_* op count: 1138 | FakeTensorMode.__torch_dispatch__:12461 | FakeTensor.__torch_dispatch__:4654 | ProxyTorchDispatchMode.__torch_dispatch__:4144 2025-08-14T21:46:02.1931271Z Dynamo produced 2 graphs covering 1138 ops with 0 graph breaks (0 unique) 2025-08-14T21:46:07.0498931Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:46:07.0500132Z from pkg_resources import resource_filename 2025-08-14T21:46:07.6081409Z 2025-08-14T21:46:08.7440846Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:46:08.7441130Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:46:08.7454767Z cpu eval GoogleFnet 2025-08-14T21:46:09.1372079Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:46:09.2857480Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:46:09.4371949Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:46:14.7549945Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7555434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7557543Z return mod(**inputs) 2025-08-14T21:46:14.7562370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7562916Z outputs = self.fnet( 2025-08-14T21:46:14.7563333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7563767Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7564196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7564820Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7565255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7565862Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7566314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7566783Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7567241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7567682Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7568112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7568553Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7568735Z 2025-08-14T21:46:14.7568858Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7569253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7569612Z return mod(**inputs) 2025-08-14T21:46:14.7569998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7570417Z outputs = self.fnet( 2025-08-14T21:46:14.7570795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7571194Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7571601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7572034Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7572436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7572837Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7573257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7573710Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7574165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7574533Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7574903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7575299Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7575451Z 2025-08-14T21:46:14.7575554Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7575911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7576234Z return mod(**inputs) 2025-08-14T21:46:14.7576578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7576935Z outputs = self.fnet( 2025-08-14T21:46:14.7577283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7577708Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7578063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7578453Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7578818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7579177Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7579575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7579980Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7580415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7580787Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7581162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7581561Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7581712Z 2025-08-14T21:46:14.7581820Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7582166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7582485Z return mod(**inputs) 2025-08-14T21:46:14.7582842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7583221Z outputs = self.fnet( 2025-08-14T21:46:14.7583576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7583958Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7584336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7584731Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7585095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7585459Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7585840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7586273Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7586661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7587036Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7587404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7587818Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7587975Z 2025-08-14T21:46:14.7588079Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7588436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7588760Z return mod(**inputs) 2025-08-14T21:46:14.7589106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7589470Z outputs = self.fnet( 2025-08-14T21:46:14.7589815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7590196Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7590570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7590964Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7591338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7591699Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7592075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7592473Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7592870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7593294Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7593695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7594147Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7594313Z 2025-08-14T21:46:14.7594425Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7594815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7595159Z return mod(**inputs) 2025-08-14T21:46:14.7595519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7595893Z outputs = self.fnet( 2025-08-14T21:46:14.7596242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7596623Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7596988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7597377Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7597743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7598097Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7598480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7598887Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7599287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7599670Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7600051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7600458Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7600611Z 2025-08-14T21:46:14.7600723Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7601077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7601420Z return mod(**inputs) 2025-08-14T21:46:14.7601774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7602141Z outputs = self.fnet( 2025-08-14T21:46:14.7602493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7602872Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7603248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7603633Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7604008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7604418Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7604828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7605267Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7605773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7606200Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7606597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7607029Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7607206Z 2025-08-14T21:46:14.7607332Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7607697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7608018Z return mod(**inputs) 2025-08-14T21:46:14.7608423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7608830Z outputs = self.fnet( 2025-08-14T21:46:14.7609203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7609599Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7609994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7610409Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7610785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7611169Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7611578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7612005Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7612421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7612835Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7613241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7613677Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7613844Z 2025-08-14T21:46:14.7613961Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7614309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7614902Z return mod(**inputs) 2025-08-14T21:46:14.7615244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7615612Z outputs = self.fnet( 2025-08-14T21:46:14.7615946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7616325Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7616673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7617044Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7617386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7617725Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7618088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7618468Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7618848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7619214Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7619574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7619980Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7620125Z 2025-08-14T21:46:14.7620230Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7620561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7620871Z return mod(**inputs) 2025-08-14T21:46:14.7621203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7621565Z outputs = self.fnet( 2025-08-14T21:46:14.7621903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7622287Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7622648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7623034Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7623380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7623728Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7624092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7624482Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7624877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7625267Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7625634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7626036Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7626196Z 2025-08-14T21:46:14.7626300Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7626660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7626965Z return mod(**inputs) 2025-08-14T21:46:14.7627306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7627668Z outputs = self.fnet( 2025-08-14T21:46:14.7628005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7628373Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7628739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7629119Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7629487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7629838Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7630207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7630587Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7630968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7631337Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7631703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7632086Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7632242Z 2025-08-14T21:46:14.7632349Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7632705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7633054Z return mod(**inputs) 2025-08-14T21:46:14.7633388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7633753Z outputs = self.fnet( 2025-08-14T21:46:14.7634095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7634457Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7635786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7636181Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7636554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7636898Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7637276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7637848Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7638239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7638619Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7638991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7639389Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7639539Z 2025-08-14T21:46:14.7639641Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7640000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7640319Z return mod(**inputs) 2025-08-14T21:46:14.7640666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7641035Z outputs = self.fnet( 2025-08-14T21:46:14.7641390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 512, in forward 2025-08-14T21:46:14.7641779Z embedding_output = self.embeddings( 2025-08-14T21:46:14.7642156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 142, in forward 2025-08-14T21:46:14.7642538Z embeddings = self.projection(embeddings) 2025-08-14T21:46:14.7642681Z 2025-08-14T21:46:14.7642763Z cudagraph partition due to non gpu ops 2025-08-14T21:46:14.7642996Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7643334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7643673Z return mod(**inputs) 2025-08-14T21:46:14.7644072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7644443Z outputs = self.fnet( 2025-08-14T21:46:14.7644797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7645176Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7645646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7646040Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7646398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7646751Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7647127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7647518Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7647944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7648366Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7648730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7649129Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7649289Z 2025-08-14T21:46:14.7649390Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7649759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7650070Z return mod(**inputs) 2025-08-14T21:46:14.7650437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7650809Z outputs = self.fnet( 2025-08-14T21:46:14.7651135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7651497Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7651850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7652222Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7652561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7652904Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7653273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7653660Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7654041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7654421Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7654795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7655198Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7655370Z 2025-08-14T21:46:14.7655471Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7655821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7656142Z return mod(**inputs) 2025-08-14T21:46:14.7656481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7656856Z outputs = self.fnet( 2025-08-14T21:46:14.7657202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7657582Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7657953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7658319Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7658667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7659008Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7659378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7659776Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7660151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7660512Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7660876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7661280Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7661427Z 2025-08-14T21:46:14.7661525Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7661866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7662174Z return mod(**inputs) 2025-08-14T21:46:14.7662506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7662858Z outputs = self.fnet( 2025-08-14T21:46:14.7663227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7663596Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7663996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7664387Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7664750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7665114Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7665474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7665866Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7666253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7666629Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7666990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7667388Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7667540Z 2025-08-14T21:46:14.7667653Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7667994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7668320Z return mod(**inputs) 2025-08-14T21:46:14.7668670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7669034Z outputs = self.fnet( 2025-08-14T21:46:14.7669365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7669743Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7670106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7670481Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7670837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7671205Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7671574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7671952Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7672361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7672754Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7673152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.7673607Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.7674020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:46:14.7674402Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.7674553Z 2025-08-14T21:46:14.7674654Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7675000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7675315Z return mod(**inputs) 2025-08-14T21:46:14.7675653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7676003Z outputs = self.fnet( 2025-08-14T21:46:14.7676341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7676724Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7677098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7677484Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7677820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7678155Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7678496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7678858Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7679228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7679593Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7679960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.7680373Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.7680763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:46:14.7681141Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:14.7681494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:46:14.7681912Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:46:14.7682126Z 2025-08-14T21:46:14.7695739Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7696151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7696486Z return mod(**inputs) 2025-08-14T21:46:14.7696889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7697277Z outputs = self.fnet( 2025-08-14T21:46:14.7697632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7698087Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7698465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7698858Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7699213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7699574Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7699956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7700334Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7700731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7701122Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7701528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:46:14.7702009Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:46:14.7702435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:46:14.7702818Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.7702958Z 2025-08-14T21:46:14.7703052Z cudagraph partition due to non gpu ops 2025-08-14T21:46:14.7703284Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7703671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7704000Z return mod(**inputs) 2025-08-14T21:46:14.7704370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7704748Z outputs = self.fnet( 2025-08-14T21:46:14.7705094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7705467Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7705826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7706211Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7706564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7706934Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7707299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7707687Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7708062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7708432Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7708797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7709176Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7709334Z 2025-08-14T21:46:14.7709437Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7709790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7710106Z return mod(**inputs) 2025-08-14T21:46:14.7710447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7710799Z outputs = self.fnet( 2025-08-14T21:46:14.7711133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7711520Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7711869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7712245Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7712595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7712938Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7713292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7713673Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7714047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7714407Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7714768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7715171Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7715318Z 2025-08-14T21:46:14.7715415Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7715756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7716069Z return mod(**inputs) 2025-08-14T21:46:14.7716400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7716745Z outputs = self.fnet( 2025-08-14T21:46:14.7717094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7717461Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7717836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7718230Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7718577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7718920Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7719275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7719664Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7720043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7720413Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7720772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7721156Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7721305Z 2025-08-14T21:46:14.7721412Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7721748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7722057Z return mod(**inputs) 2025-08-14T21:46:14.7722392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7722742Z outputs = self.fnet( 2025-08-14T21:46:14.7723066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7723425Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7723775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7724138Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7724485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7724870Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7725248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7725735Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7726148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7726578Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7727003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7727446Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7727627Z 2025-08-14T21:46:14.7727732Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7728089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7728421Z return mod(**inputs) 2025-08-14T21:46:14.7728772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7729139Z outputs = self.fnet( 2025-08-14T21:46:14.7729485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7729848Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7730212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7730616Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7730965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7731341Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7731725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7732119Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7732513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7732917Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7733319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.7733760Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.7734162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:46:14.7734541Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.7734676Z 2025-08-14T21:46:14.7734787Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7735137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7735461Z return mod(**inputs) 2025-08-14T21:46:14.7735809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7736177Z outputs = self.fnet( 2025-08-14T21:46:14.7736515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7736885Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7737253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7738020Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7738387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7738745Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7739166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7739540Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7739927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7740309Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7740708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.7741137Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.7741546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:46:14.7741949Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:14.7742311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:46:14.7742791Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:46:14.7743030Z 2025-08-14T21:46:14.7743136Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7743491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7743798Z return mod(**inputs) 2025-08-14T21:46:14.7744146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7744520Z outputs = self.fnet( 2025-08-14T21:46:14.7744877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7745234Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7745608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7745988Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7746325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7746672Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7747034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7747404Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7747777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7748152Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7748538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:46:14.7748974Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:46:14.7749379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:46:14.7749749Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.7749878Z 2025-08-14T21:46:14.7749962Z cudagraph partition due to non gpu ops 2025-08-14T21:46:14.7750182Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7750522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7750833Z return mod(**inputs) 2025-08-14T21:46:14.7751165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7751512Z outputs = self.fnet( 2025-08-14T21:46:14.7751848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7752208Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7752634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7753012Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7753358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7753703Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7754058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7754443Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7754839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7755231Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7755596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7756009Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7756160Z 2025-08-14T21:46:14.7756267Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7756609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7756912Z return mod(**inputs) 2025-08-14T21:46:14.7757246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7757599Z outputs = self.fnet( 2025-08-14T21:46:14.7757937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7758302Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7758681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7759048Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7759392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7759728Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7760087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7760459Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7760833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7761198Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7761556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7761935Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7762085Z 2025-08-14T21:46:14.7762185Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7762519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7762828Z return mod(**inputs) 2025-08-14T21:46:14.7763150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7763496Z outputs = self.fnet( 2025-08-14T21:46:14.7763829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7764179Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7764529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7764914Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7765266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7765704Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7766119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7766544Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7766968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7767377Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7767806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7768185Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7768327Z 2025-08-14T21:46:14.7768421Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7768754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7769081Z return mod(**inputs) 2025-08-14T21:46:14.7769400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7769745Z outputs = self.fnet( 2025-08-14T21:46:14.7770066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7770412Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7770747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7771125Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7771464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7771789Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7772165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7772545Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7772912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7773259Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7773617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7774011Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7774161Z 2025-08-14T21:46:14.7774269Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7774614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7774927Z return mod(**inputs) 2025-08-14T21:46:14.7775268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7775633Z outputs = self.fnet( 2025-08-14T21:46:14.7775954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7776311Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7776660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7777021Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7777363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7777705Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7778064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7778429Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7778824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7779199Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7779581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.7780008Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.7780405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:46:14.7780778Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.7780910Z 2025-08-14T21:46:14.7781009Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7781348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7781655Z return mod(**inputs) 2025-08-14T21:46:14.7781990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7782356Z outputs = self.fnet( 2025-08-14T21:46:14.7782692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7783050Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7783392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7783760Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7784119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7784466Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7784835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7785202Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7785583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7785946Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7786337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.7786758Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.7787148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:46:14.7787534Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:14.7787892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:46:14.7788315Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:46:14.7788535Z 2025-08-14T21:46:14.7788643Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7788978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7789288Z return mod(**inputs) 2025-08-14T21:46:14.7789630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7789985Z outputs = self.fnet( 2025-08-14T21:46:14.7790336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7790697Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7791051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7791413Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7791761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7792124Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7792488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7792852Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7793232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7793603Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7793981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:46:14.7794416Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:46:14.7794821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:46:14.7795191Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.7795340Z 2025-08-14T21:46:14.7795417Z cudagraph partition due to non gpu ops 2025-08-14T21:46:14.7795648Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7795986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7796289Z return mod(**inputs) 2025-08-14T21:46:14.7796628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7796979Z outputs = self.fnet( 2025-08-14T21:46:14.7797343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7797699Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7798069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7798443Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7798792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7799123Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7799480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7799862Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7800232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7800601Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7800966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7801359Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7801503Z 2025-08-14T21:46:14.7801600Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7801930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7802230Z return mod(**inputs) 2025-08-14T21:46:14.7802554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7802906Z outputs = self.fnet( 2025-08-14T21:46:14.7803234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7803616Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7803970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7804354Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7804700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7805059Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7805416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7805878Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7806315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7806731Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7807189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7807589Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7807749Z 2025-08-14T21:46:14.7807858Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7808199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7808510Z return mod(**inputs) 2025-08-14T21:46:14.7808868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7809221Z outputs = self.fnet( 2025-08-14T21:46:14.7809557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7809919Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7810272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7810671Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7811014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7811365Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7811729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7812112Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7812495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7812863Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7813223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7813615Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7813773Z 2025-08-14T21:46:14.7813876Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7814227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7814545Z return mod(**inputs) 2025-08-14T21:46:14.7814889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7815257Z outputs = self.fnet( 2025-08-14T21:46:14.7815597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7815965Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7816317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7816687Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7817026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7817367Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7817731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7818130Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7818516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7818898Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7819255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7819630Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7819781Z 2025-08-14T21:46:14.7819879Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7820216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7820523Z return mod(**inputs) 2025-08-14T21:46:14.7820845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7821195Z outputs = self.fnet( 2025-08-14T21:46:14.7821527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7821896Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7822246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7822619Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7822964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7823295Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7823668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7824039Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7824437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7824805Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7825191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.7825619Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.7826006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:46:14.7826373Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.7826513Z 2025-08-14T21:46:14.7826611Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7826951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7827257Z return mod(**inputs) 2025-08-14T21:46:14.7827587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7827940Z outputs = self.fnet( 2025-08-14T21:46:14.7828270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7828628Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7828987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7829361Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7829701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7830043Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7830416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7830803Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7831190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7831593Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7832005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.7832435Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.7832837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:46:14.7833232Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:14.7833590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:46:14.7834027Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:46:14.7834254Z 2025-08-14T21:46:14.7834364Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7834714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7835055Z return mod(**inputs) 2025-08-14T21:46:14.7835406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7835767Z outputs = self.fnet( 2025-08-14T21:46:14.7836160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7836533Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7836895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7837291Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7837824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7838227Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7838590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7838976Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7839361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7839748Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7840134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:46:14.7840579Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:46:14.7840991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:46:14.7841369Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.7841505Z 2025-08-14T21:46:14.7841589Z cudagraph partition due to non gpu ops 2025-08-14T21:46:14.7841831Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7842192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7842508Z return mod(**inputs) 2025-08-14T21:46:14.7842873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7843269Z outputs = self.fnet( 2025-08-14T21:46:14.7843636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7844036Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7844430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7844841Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7845220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7845695Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7846113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7846543Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7846952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7847343Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7847734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7848148Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7848307Z 2025-08-14T21:46:14.7848413Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7848777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7849106Z return mod(**inputs) 2025-08-14T21:46:14.7849489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7849865Z outputs = self.fnet( 2025-08-14T21:46:14.7850216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7850610Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7850961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7851341Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7851716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7852068Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7852461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7852867Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7853266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7853651Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7854023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7854416Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7854563Z 2025-08-14T21:46:14.7854674Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7855014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7855327Z return mod(**inputs) 2025-08-14T21:46:14.7855665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7856022Z outputs = self.fnet( 2025-08-14T21:46:14.7856361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7856731Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7857089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7857463Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7857810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7858157Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7858516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7858913Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7859294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7859678Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7860041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7860446Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7860604Z 2025-08-14T21:46:14.7860707Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7861066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7861395Z return mod(**inputs) 2025-08-14T21:46:14.7861763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7862155Z outputs = self.fnet( 2025-08-14T21:46:14.7862525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7862966Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7863363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7863837Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7864228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7864618Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7865053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7865464Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7865862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7866244Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7866607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7866983Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7867147Z 2025-08-14T21:46:14.7867248Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7867605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7867948Z return mod(**inputs) 2025-08-14T21:46:14.7868310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7868709Z outputs = self.fnet( 2025-08-14T21:46:14.7869070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7869441Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7869814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7870207Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7870591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7870967Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7871345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7871736Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7872134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7872518Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7872904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.7873331Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.7873743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:46:14.7874113Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.7874247Z 2025-08-14T21:46:14.7874347Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7874687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7874988Z return mod(**inputs) 2025-08-14T21:46:14.7875321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7875674Z outputs = self.fnet( 2025-08-14T21:46:14.7876000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7876365Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7876719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7877137Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7877476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7877817Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7878176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7878543Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7878926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7879308Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7879717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.7880156Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.7880568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:46:14.7880978Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:14.7881355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:46:14.7881788Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:46:14.7882022Z 2025-08-14T21:46:14.7882128Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7882487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7882806Z return mod(**inputs) 2025-08-14T21:46:14.7883154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7883532Z outputs = self.fnet( 2025-08-14T21:46:14.7883891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7884270Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7884647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7885051Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7885416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7885847Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7886230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7886654Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7887115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7887536Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7887963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:46:14.7888412Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:46:14.7888825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:46:14.7889211Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.7889355Z 2025-08-14T21:46:14.7889438Z cudagraph partition due to non gpu ops 2025-08-14T21:46:14.7889674Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7890017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7890332Z return mod(**inputs) 2025-08-14T21:46:14.7890692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7891042Z outputs = self.fnet( 2025-08-14T21:46:14.7891383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7891750Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7892108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7892481Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7892849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7893203Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7893588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7893991Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7894374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7894749Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7895114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7895507Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7895657Z 2025-08-14T21:46:14.7895764Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7896113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7896423Z return mod(**inputs) 2025-08-14T21:46:14.7896772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7897138Z outputs = self.fnet( 2025-08-14T21:46:14.7897471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7897846Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7898212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7898595Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7898941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7899293Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7899670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7900057Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7900451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7900847Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7901216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7901609Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7901763Z 2025-08-14T21:46:14.7901860Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7902199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7902503Z return mod(**inputs) 2025-08-14T21:46:14.7902830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7903181Z outputs = self.fnet( 2025-08-14T21:46:14.7903520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7903907Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7904273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7904658Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7905014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7905363Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7905742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7906162Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7906547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7906940Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7907313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7907707Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7907855Z 2025-08-14T21:46:14.7907954Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7908299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7908610Z return mod(**inputs) 2025-08-14T21:46:14.7908946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7909300Z outputs = self.fnet( 2025-08-14T21:46:14.7909634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7909998Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7910351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7910732Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7911085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7911441Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7911804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7912201Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7912615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7913022Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7913414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7913835Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7914032Z 2025-08-14T21:46:14.7914154Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7914496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7914819Z return mod(**inputs) 2025-08-14T21:46:14.7915150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7915503Z outputs = self.fnet( 2025-08-14T21:46:14.7915828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7916194Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7916554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7916930Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7917291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7917679Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7918088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7918545Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7918952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7919353Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7919789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.7920235Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.7920674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:46:14.7921071Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.7921207Z 2025-08-14T21:46:14.7921309Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7921664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7921989Z return mod(**inputs) 2025-08-14T21:46:14.7922339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7922704Z outputs = self.fnet( 2025-08-14T21:46:14.7923056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7923438Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7923808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7924200Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7924560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7924919Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7925307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7925816Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7926252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7926702Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7927130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.7927596Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.7928037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:46:14.7928497Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:14.7928896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:46:14.7929377Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:46:14.7929622Z 2025-08-14T21:46:14.7929737Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7930105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7930449Z return mod(**inputs) 2025-08-14T21:46:14.7930845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7931258Z outputs = self.fnet( 2025-08-14T21:46:14.7931638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7932069Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7932469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7932875Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7933261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7933638Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7934064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7934486Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7934924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7935359Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7935797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:46:14.7936300Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:46:14.7936775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:46:14.7937199Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.7937344Z 2025-08-14T21:46:14.7937433Z cudagraph partition due to non gpu ops 2025-08-14T21:46:14.7937849Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7939378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7939805Z return mod(**inputs) 2025-08-14T21:46:14.7940254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7940698Z outputs = self.fnet( 2025-08-14T21:46:14.7941062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7941442Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7941825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7942230Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7942603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7942959Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7943319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7943704Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7944081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7944675Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7945042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7945431Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7945581Z 2025-08-14T21:46:14.7945685Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7946031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7946343Z return mod(**inputs) 2025-08-14T21:46:14.7946677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7947023Z outputs = self.fnet( 2025-08-14T21:46:14.7947392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7947799Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7948154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7948556Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7948904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7949247Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7949608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7950034Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7950420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7950822Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7951183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7951571Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7951716Z 2025-08-14T21:46:14.7951927Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7952280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7952593Z return mod(**inputs) 2025-08-14T21:46:14.7952930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7953331Z outputs = self.fnet( 2025-08-14T21:46:14.7953661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7954021Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7954377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7954745Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7954960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7955036Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7955272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7955363Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7955593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7955678Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7955908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7956008Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7956036Z 2025-08-14T21:46:14.7956136Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7956325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7956392Z return mod(**inputs) 2025-08-14T21:46:14.7956621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7956685Z outputs = self.fnet( 2025-08-14T21:46:14.7956922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7956993Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7957230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7957313Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7957520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7957625Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7957857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7957947Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7958183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7958258Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7958509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7958604Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7958608Z 2025-08-14T21:46:14.7958739Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7958940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7959004Z return mod(**inputs) 2025-08-14T21:46:14.7959242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7959304Z outputs = self.fnet( 2025-08-14T21:46:14.7959531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7959605Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7959838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7959916Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7960134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7960219Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7960451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7960532Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7960774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7960853Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7961110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.7961225Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.7961453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:46:14.7961530Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.7961533Z 2025-08-14T21:46:14.7961635Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7961836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7961897Z return mod(**inputs) 2025-08-14T21:46:14.7962126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7962186Z outputs = self.fnet( 2025-08-14T21:46:14.7962411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7962477Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7962707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7962793Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7962999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7963083Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7963327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7963403Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7963647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7963719Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7963971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.7964104Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.7964334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:46:14.7964459Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:14.7964659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:46:14.7964835Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:46:14.7964839Z 2025-08-14T21:46:14.7964946Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7965134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7965209Z return mod(**inputs) 2025-08-14T21:46:14.7965440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7965573Z outputs = self.fnet( 2025-08-14T21:46:14.7965821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7965893Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7966151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7966250Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7966480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7966577Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7966813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7966893Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7967152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7967230Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7967503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:46:14.7967651Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:46:14.7967883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:46:14.7967979Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.7967983Z 2025-08-14T21:46:14.7968059Z cudagraph partition due to non gpu ops 2025-08-14T21:46:14.7968153Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7968344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7968407Z return mod(**inputs) 2025-08-14T21:46:14.7968641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7968702Z outputs = self.fnet( 2025-08-14T21:46:14.7968927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7969018Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7969259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7969339Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7969564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7969638Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7969874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7969982Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7970208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7970306Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7970528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7970625Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7970635Z 2025-08-14T21:46:14.7970728Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7970906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7970973Z return mod(**inputs) 2025-08-14T21:46:14.7971193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7971252Z outputs = self.fnet( 2025-08-14T21:46:14.7971479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7971547Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7971776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7971855Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7972053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7972134Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7972356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7972444Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7972676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7972749Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7972979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7973076Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7973095Z 2025-08-14T21:46:14.7973193Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7973385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7973444Z return mod(**inputs) 2025-08-14T21:46:14.7973676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7973740Z outputs = self.fnet( 2025-08-14T21:46:14.7973967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7974046Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7974278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7974355Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7974569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7974663Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7974896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7974985Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7975211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7975294Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7975543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7975645Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7975649Z 2025-08-14T21:46:14.7975761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7975947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7976025Z return mod(**inputs) 2025-08-14T21:46:14.7976248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7976308Z outputs = self.fnet( 2025-08-14T21:46:14.7976536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7976603Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7976838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7976918Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7977134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7977213Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7977436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7977524Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7977753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7977825Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7978055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7978146Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7978149Z 2025-08-14T21:46:14.7978242Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7978433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7978493Z return mod(**inputs) 2025-08-14T21:46:14.7978724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7978804Z outputs = self.fnet( 2025-08-14T21:46:14.7979024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7979098Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7979319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7979396Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7979610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7979685Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7979921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7979999Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7980260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7980339Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7980599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.7980714Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.7980944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:46:14.7981057Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.7981061Z 2025-08-14T21:46:14.7981168Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7981371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7981436Z return mod(**inputs) 2025-08-14T21:46:14.7981677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7981741Z outputs = self.fnet( 2025-08-14T21:46:14.7981976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7982045Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7982276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7982363Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7982570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7982645Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7982884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7982962Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7983216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7983287Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7983547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.7983661Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.7983894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:46:14.7984006Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:14.7984205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:46:14.7984379Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:46:14.7984405Z 2025-08-14T21:46:14.7984511Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7984704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7984777Z return mod(**inputs) 2025-08-14T21:46:14.7985009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7985076Z outputs = self.fnet( 2025-08-14T21:46:14.7985315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7985390Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7985620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7985710Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7985923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7986021Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7986250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7986328Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.7986575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.7986645Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.7986917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:46:14.7987048Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:46:14.7987294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:46:14.7987384Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.7987388Z 2025-08-14T21:46:14.7987467Z cudagraph partition due to non gpu ops 2025-08-14T21:46:14.7987563Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7987758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7987820Z return mod(**inputs) 2025-08-14T21:46:14.7988055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7988116Z outputs = self.fnet( 2025-08-14T21:46:14.7988345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7988420Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7988652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7988733Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7988946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7989017Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7989253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7989344Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7989576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7989657Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7989885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7989981Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7990008Z 2025-08-14T21:46:14.7990107Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7990294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7990361Z return mod(**inputs) 2025-08-14T21:46:14.7990589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7990650Z outputs = self.fnet( 2025-08-14T21:46:14.7990886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7990956Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7991190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7991269Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7991478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7991583Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7991817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7991909Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7992151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7992227Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7992480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7992608Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7992614Z 2025-08-14T21:46:14.7992747Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7992949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7993018Z return mod(**inputs) 2025-08-14T21:46:14.7993265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7993332Z outputs = self.fnet( 2025-08-14T21:46:14.7993573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7993653Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7993906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7993990Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7994210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7994289Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7994533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7994629Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7994863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7994949Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7995185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7995280Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7995292Z 2025-08-14T21:46:14.7995392Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7995581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7995656Z return mod(**inputs) 2025-08-14T21:46:14.7995895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7995975Z outputs = self.fnet( 2025-08-14T21:46:14.7996211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7996279Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7996512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7996589Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7996795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7996890Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7997164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.7997257Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.7997496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.7997590Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.7997830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.7997924Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.7997927Z 2025-08-14T21:46:14.7998022Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.7998236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.7998299Z return mod(**inputs) 2025-08-14T21:46:14.7998537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.7998663Z outputs = self.fnet( 2025-08-14T21:46:14.7998893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.7998970Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.7999201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.7999280Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.7999492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.7999566Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.7999806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.7999884Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.8000129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.8000211Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.8000474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.8000589Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.8000819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:46:14.8000896Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.8000900Z 2025-08-14T21:46:14.8001003Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8001191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8001254Z return mod(**inputs) 2025-08-14T21:46:14.8001494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.8001557Z outputs = self.fnet( 2025-08-14T21:46:14.8001812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.8001886Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.8002136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.8002223Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.8002431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.8002505Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.8002764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.8002843Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.8003106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.8003211Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.8003477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.8003593Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.8003834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:46:14.8003945Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:14.8004172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:46:14.8004350Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:46:14.8004353Z 2025-08-14T21:46:14.8004477Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8004677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8004751Z return mod(**inputs) 2025-08-14T21:46:14.8004994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.8005059Z outputs = self.fnet( 2025-08-14T21:46:14.8005308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.8005381Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.8005885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.8005989Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.8006217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.8006309Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.8006575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.8006664Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.8006986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.8007061Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.8007335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:46:14.8007472Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:46:14.8007715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:46:14.8007804Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.8007810Z 2025-08-14T21:46:14.8007893Z cudagraph partition due to non gpu ops 2025-08-14T21:46:14.8008021Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8008228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8008295Z return mod(**inputs) 2025-08-14T21:46:14.8008551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.8008620Z outputs = self.fnet( 2025-08-14T21:46:14.8008867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.8008949Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.8009200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.8009288Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.8009524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.8009620Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.8009872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.8009969Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.8010212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.8010301Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.8010567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.8010669Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.8010681Z 2025-08-14T21:46:14.8010782Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8010997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8011074Z return mod(**inputs) 2025-08-14T21:46:14.8011321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.8011386Z outputs = self.fnet( 2025-08-14T21:46:14.8011634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.8011705Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.8011955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.8012038Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.8012253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.8012339Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.8012579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.8012678Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.8012927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.8013006Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.8013256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.8013356Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.8013359Z 2025-08-14T21:46:14.8013460Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8013664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8013729Z return mod(**inputs) 2025-08-14T21:46:14.8013977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.8014092Z outputs = self.fnet( 2025-08-14T21:46:14.8014337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.8014416Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.8014659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.8014740Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.8014965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.8015042Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.8015291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.8015385Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.8015627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.8015734Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.8015976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.8016075Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.8016086Z 2025-08-14T21:46:14.8016186Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8016384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8016472Z return mod(**inputs) 2025-08-14T21:46:14.8016719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.8016800Z outputs = self.fnet( 2025-08-14T21:46:14.8017050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.8017123Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.8017372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.8017454Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.8017672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.8017754Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.8017996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.8018090Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.8018337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.8018414Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.8018662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.8018760Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.8018764Z 2025-08-14T21:46:14.8018862Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8019062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8019127Z return mod(**inputs) 2025-08-14T21:46:14.8019376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.8019440Z outputs = self.fnet( 2025-08-14T21:46:14.8019681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.8019761Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.8020031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.8020113Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.8020329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.8020402Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.8020642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.8020721Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.8020969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.8021049Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.8021315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.8021452Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.8021687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:46:14.8021764Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.8021768Z 2025-08-14T21:46:14.8021873Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8022065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8022129Z return mod(**inputs) 2025-08-14T21:46:14.8022388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.8022454Z outputs = self.fnet( 2025-08-14T21:46:14.8022718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.8022791Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.8023031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.8023117Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.8023326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.8023401Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.8023685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.8023765Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.8024029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.8024104Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.8024377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.8024501Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.8024744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:46:14.8024859Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:14.8025066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:46:14.8025241Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:46:14.8025246Z 2025-08-14T21:46:14.8025356Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8025551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8025624Z return mod(**inputs) 2025-08-14T21:46:14.8025864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.8025959Z outputs = self.fnet( 2025-08-14T21:46:14.8026227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.8026301Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.8026540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.8026628Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.8026843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.8026939Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.8027180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.8027264Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.8027543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.8027617Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.8027888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:46:14.8028022Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:46:14.8028263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:46:14.8028378Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.8028382Z 2025-08-14T21:46:14.8028465Z cudagraph partition due to non gpu ops 2025-08-14T21:46:14.8028566Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8028788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8028858Z return mod(**inputs) 2025-08-14T21:46:14.8029109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.8029174Z outputs = self.fnet( 2025-08-14T21:46:14.8029414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.8029494Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.8029736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.8029821Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.8030042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.8030119Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.8030365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.8030463Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.8030701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.8030789Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.8031027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.8031127Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.8031138Z 2025-08-14T21:46:14.8031240Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8031434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8031508Z return mod(**inputs) 2025-08-14T21:46:14.8031746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.8031833Z outputs = self.fnet( 2025-08-14T21:46:14.8032089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.8032162Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.8032413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.8032495Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.8032716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.8032801Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.8033061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.8033164Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.8033433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.8033543Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.8033793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.8033891Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.8033894Z 2025-08-14T21:46:14.8033993Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8034196Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8034278Z return mod(**inputs) 2025-08-14T21:46:14.8034530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.8034595Z outputs = self.fnet( 2025-08-14T21:46:14.8034852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.8034933Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.8035180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.8035263Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.8035488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.8035565Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.8035815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.8035911Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.8036153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.8036239Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.8036482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.8036587Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.8036591Z 2025-08-14T21:46:14.8036691Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8036885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8036957Z return mod(**inputs) 2025-08-14T21:46:14.8037202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.8037268Z outputs = self.fnet( 2025-08-14T21:46:14.8037518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.8037593Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.8038274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.8038424Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.8038655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.8038746Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.8039004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.8039106Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.8039375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.8039460Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.8039726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.8039833Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.8039865Z 2025-08-14T21:46:14.8039974Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8040191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8040260Z return mod(**inputs) 2025-08-14T21:46:14.8040520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.8040589Z outputs = self.fnet( 2025-08-14T21:46:14.8040885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.8040974Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.8041258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.8041350Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.8041589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.8041673Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.8041936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.8042023Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.8042295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.8042383Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.8042676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.8042802Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.8043060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:46:14.8043143Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.8043147Z 2025-08-14T21:46:14.8043257Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8043460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8043529Z return mod(**inputs) 2025-08-14T21:46:14.8043797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.8043866Z outputs = self.fnet( 2025-08-14T21:46:14.8044129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.8044206Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.8044462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.8044578Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.8044812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.8044892Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.8045154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.8045237Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.8045566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.8045655Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.8045931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.8046051Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.8046293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:46:14.8046431Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:14.8046640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:46:14.8046815Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:46:14.8046819Z 2025-08-14T21:46:14.8046927Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8047137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8047212Z return mod(**inputs) 2025-08-14T21:46:14.8047459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.8047554Z outputs = self.fnet( 2025-08-14T21:46:14.8047808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.8047885Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.8048129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.8048220Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.8048440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.8048523Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.8048766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.8048847Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.8049111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.8049188Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.8049461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:46:14.8049593Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:46:14.8049835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:46:14.8049919Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.8049923Z 2025-08-14T21:46:14.8050003Z cudagraph partition due to non gpu ops 2025-08-14T21:46:14.8050106Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8050314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8050375Z return mod(**inputs) 2025-08-14T21:46:14.8050613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.8050692Z outputs = self.fnet( 2025-08-14T21:46:14.8050920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.8050998Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.8051228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.8051308Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.8051523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.8051599Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.8051837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.8051932Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.8054983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.8055110Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.8055360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.8055461Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.8055465Z 2025-08-14T21:46:14.8055578Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8055784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8055849Z return mod(**inputs) 2025-08-14T21:46:14.8056096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.8056159Z outputs = self.fnet( 2025-08-14T21:46:14.8056419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.8056527Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.8056758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.8056848Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.8057054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.8057129Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.8057364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.8057456Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.8057689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.8057764Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.8057993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.8058098Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.8058101Z 2025-08-14T21:46:14.8058197Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8058382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8058452Z return mod(**inputs) 2025-08-14T21:46:14.8058679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.8058747Z outputs = self.fnet( 2025-08-14T21:46:14.8058973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.8059043Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.8059276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.8059381Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.8059587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.8059668Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.8059899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.8059997Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.8060225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.8060299Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.8060538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.8060633Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.8060661Z 2025-08-14T21:46:14.8060827Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8061014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8061077Z return mod(**inputs) 2025-08-14T21:46:14.8061313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.8061375Z outputs = self.fnet( 2025-08-14T21:46:14.8061605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.8061680Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.8061907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.8062014Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.8062225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.8062301Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.8062542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 249, in forward 2025-08-14T21:46:14.8062633Z self_fourier_outputs = self.fourier(hidden_states) 2025-08-14T21:46:14.8062875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 202, in forward 2025-08-14T21:46:14.8062950Z self_outputs = self.self(hidden_states) 2025-08-14T21:46:14.8063184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 181, in forward 2025-08-14T21:46:14.8063285Z outputs = self.fourier_transform(hidden_states).real 2025-08-14T21:46:14.8063288Z 2025-08-14T21:46:14.8063386Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8063588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8063660Z return mod(**inputs) 2025-08-14T21:46:14.8063908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.8063980Z outputs = self.fnet( 2025-08-14T21:46:14.8064228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.8064299Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.8064551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.8064634Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.8064855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.8064941Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.8065207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.8065298Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.8065559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.8065634Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.8065930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.8066041Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.8066290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 219, in forward 2025-08-14T21:46:14.8066371Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.8066374Z 2025-08-14T21:46:14.8066472Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8066726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8066790Z return mod(**inputs) 2025-08-14T21:46:14.8067031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.8067101Z outputs = self.fnet( 2025-08-14T21:46:14.8067339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.8067418Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.8067659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.8067741Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.8067982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.8068065Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.8068320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.8068401Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.8068659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.8068743Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.8069020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 261, in feed_forward_chunk 2025-08-14T21:46:14.8069132Z intermediate_output = self.intermediate(fourier_output) 2025-08-14T21:46:14.8069380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 220, in forward 2025-08-14T21:46:14.8069489Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:14.8069706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 47, in forward 2025-08-14T21:46:14.8069884Z return 0.5 * input * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (input + 0.044715 * torch.pow(input, 3.0)))) 2025-08-14T21:46:14.8069887Z 2025-08-14T21:46:14.8069988Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8070192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8070257Z return mod(**inputs) 2025-08-14T21:46:14.8070508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 671, in forward 2025-08-14T21:46:14.8070574Z outputs = self.fnet( 2025-08-14T21:46:14.8070815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 518, in forward 2025-08-14T21:46:14.8070896Z encoder_outputs = self.encoder( 2025-08-14T21:46:14.8071157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 280, in forward 2025-08-14T21:46:14.8071243Z layer_outputs = layer_module(hidden_states) 2025-08-14T21:46:14.8071467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:14.8071543Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:14.8071791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 252, in forward 2025-08-14T21:46:14.8071873Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:14.8072125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:14.8072210Z return forward_fn(*input_tensors) 2025-08-14T21:46:14.8072484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 262, in feed_forward_chunk 2025-08-14T21:46:14.8072667Z layer_output = self.output(intermediate_output, fourier_output) 2025-08-14T21:46:14.8072913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 233, in forward 2025-08-14T21:46:14.8072994Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.8072997Z 2025-08-14T21:46:14.8073108Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8073303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8073370Z return mod(**inputs) 2025-08-14T21:46:14.8073619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 681, in forward 2025-08-14T21:46:14.8073717Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:46:14.8073997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 359, in forward 2025-08-14T21:46:14.8074119Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:46:14.8074383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 340, in forward 2025-08-14T21:46:14.8074489Z hidden_states = self.transform(hidden_states) 2025-08-14T21:46:14.8074747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 321, in forward 2025-08-14T21:46:14.8074835Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:14.8074838Z 2025-08-14T21:46:14.8074939Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8075136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8075208Z return mod(**inputs) 2025-08-14T21:46:14.8075453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 681, in forward 2025-08-14T21:46:14.8075544Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:46:14.8075800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 359, in forward 2025-08-14T21:46:14.8075910Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:46:14.8076161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 341, in forward 2025-08-14T21:46:14.8076248Z hidden_states = self.decoder(hidden_states) 2025-08-14T21:46:14.8076252Z 2025-08-14T21:46:14.8076353Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:14.8076561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:14.8076627Z return mod(**inputs) 2025-08-14T21:46:14.8076877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/fnet/modeling_fnet.py", line 686, in forward 2025-08-14T21:46:14.8077066Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:46:14.8077089Z 2025-08-14T21:46:22.4957812Z Compilation time (from dynamo_timed): 11.853013562 2025-08-14T21:46:22.5006234Z pass 2025-08-14T21:46:22.5010471Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:46:22.5015413Z TIMING: _recursive_pre_grad_passes:0.00545 _recursive_joint_graph_passes:0.20402 _recursive_post_grad_passes:0.07831 async_compile.wait:0.74352 code_gen:7.31114 inductor_compile:8.45623 backend_compile:10.23369 gc:0.00018 entire_frame_compile:11.85301 total_wall_time:11.85301 2025-08-14T21:46:22.5016600Z STATS: call_* op count: 232 | FakeTensorMode.__torch_dispatch__:7521 | FakeTensor.__torch_dispatch__:3660 | ProxyTorchDispatchMode.__torch_dispatch__:2859 2025-08-14T21:46:22.5017061Z Dynamo produced 1 graphs covering 232 ops with 0 graph breaks (0 unique) 2025-08-14T21:46:27.3468287Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:46:27.3469504Z from pkg_resources import resource_filename 2025-08-14T21:46:27.9279228Z 2025-08-14T21:46:29.2667502Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:46:29.2672160Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:46:29.2679789Z cpu eval LayoutLMForMaskedLM 2025-08-14T21:46:29.8499723Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:46:30.0825200Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:46:30.3066836Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:46:38.4444446Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4450167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4451730Z return mod(**inputs) 2025-08-14T21:46:38.4452292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4452838Z return func(*args, **kwargs) 2025-08-14T21:46:38.4456929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4457443Z return func(*args, **kwargs) 2025-08-14T21:46:38.4457828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4458220Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4458689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4459137Z outputs = self.layoutlm( 2025-08-14T21:46:38.4459546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4459945Z return func(*args, **kwargs) 2025-08-14T21:46:38.4460328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4460713Z return func(*args, **kwargs) 2025-08-14T21:46:38.4461081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4461455Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4461890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4462328Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4462726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4463450Z return func(*args, **kwargs) 2025-08-14T21:46:38.4463823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4464229Z return func(*args, **kwargs) 2025-08-14T21:46:38.4464602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4465000Z return func(*args, **kwargs) 2025-08-14T21:46:38.4465199Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4465581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4465986Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4466396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4466833Z layer_outputs = layer_module( 2025-08-14T21:46:38.4467297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4467765Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4468165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4468575Z return func(*args, **kwargs) 2025-08-14T21:46:38.4468960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4469354Z return func(*args, **kwargs) 2025-08-14T21:46:38.4469721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4470104Z return func(*args, **kwargs) 2025-08-14T21:46:38.4470599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.4471030Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.4471438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4471823Z return func(*args, **kwargs) 2025-08-14T21:46:38.4472196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4472580Z return func(*args, **kwargs) 2025-08-14T21:46:38.4472966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4473397Z return func(*args, **kwargs) 2025-08-14T21:46:38.4473830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.4474255Z self_outputs = self.self( 2025-08-14T21:46:38.4474639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4475034Z return func(*args, **kwargs) 2025-08-14T21:46:38.4475415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4475833Z return func(*args, **kwargs) 2025-08-14T21:46:38.4476228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4476618Z return func(*args, **kwargs) 2025-08-14T21:46:38.4477021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:46:38.4477525Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.4477740Z 2025-08-14T21:46:38.4477865Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4478253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4478658Z return mod(**inputs) 2025-08-14T21:46:38.4479038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4479431Z return func(*args, **kwargs) 2025-08-14T21:46:38.4479804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4480189Z return func(*args, **kwargs) 2025-08-14T21:46:38.4480537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4480916Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4481334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4481757Z outputs = self.layoutlm( 2025-08-14T21:46:38.4482143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4482559Z return func(*args, **kwargs) 2025-08-14T21:46:38.4482978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4483385Z return func(*args, **kwargs) 2025-08-14T21:46:38.4483752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4484141Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4484595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4485049Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4485658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4489228Z return func(*args, **kwargs) 2025-08-14T21:46:38.4489647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4490072Z return func(*args, **kwargs) 2025-08-14T21:46:38.4490446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4490842Z return func(*args, **kwargs) 2025-08-14T21:46:38.4491056Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4491436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4491799Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4492207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4492612Z layer_outputs = layer_module( 2025-08-14T21:46:38.4492959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4493326Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4493712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4494094Z return func(*args, **kwargs) 2025-08-14T21:46:38.4494469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4494861Z return func(*args, **kwargs) 2025-08-14T21:46:38.4495238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4495611Z return func(*args, **kwargs) 2025-08-14T21:46:38.4495994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.4496416Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.4496796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4497185Z return func(*args, **kwargs) 2025-08-14T21:46:38.4497538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4497896Z return func(*args, **kwargs) 2025-08-14T21:46:38.4498246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4498600Z return func(*args, **kwargs) 2025-08-14T21:46:38.4498974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.4499375Z self_outputs = self.self( 2025-08-14T21:46:38.4499726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4500807Z return func(*args, **kwargs) 2025-08-14T21:46:38.4501184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4501623Z return func(*args, **kwargs) 2025-08-14T21:46:38.4501974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4502355Z return func(*args, **kwargs) 2025-08-14T21:46:38.4502760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:46:38.4503241Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.4503458Z 2025-08-14T21:46:38.4503568Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4503944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4504272Z return mod(**inputs) 2025-08-14T21:46:38.4504632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4505005Z return func(*args, **kwargs) 2025-08-14T21:46:38.4505362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4505728Z return func(*args, **kwargs) 2025-08-14T21:46:38.4506054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4506403Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4506799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4507205Z outputs = self.layoutlm( 2025-08-14T21:46:38.4507554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4507907Z return func(*args, **kwargs) 2025-08-14T21:46:38.4508245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4508600Z return func(*args, **kwargs) 2025-08-14T21:46:38.4508921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4509261Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4509675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4510070Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4510437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4510819Z return func(*args, **kwargs) 2025-08-14T21:46:38.4511161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4511524Z return func(*args, **kwargs) 2025-08-14T21:46:38.4511906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4512271Z return func(*args, **kwargs) 2025-08-14T21:46:38.4512482Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4512852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4513235Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4513650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4514053Z layer_outputs = layer_module( 2025-08-14T21:46:38.4514400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4514760Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4515129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4515519Z return func(*args, **kwargs) 2025-08-14T21:46:38.4515881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4516232Z return func(*args, **kwargs) 2025-08-14T21:46:38.4516572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4516922Z return func(*args, **kwargs) 2025-08-14T21:46:38.4517293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.4517682Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.4518046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4518416Z return func(*args, **kwargs) 2025-08-14T21:46:38.4518759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4519124Z return func(*args, **kwargs) 2025-08-14T21:46:38.4519472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4519835Z return func(*args, **kwargs) 2025-08-14T21:46:38.4520204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.4520599Z self_outputs = self.self( 2025-08-14T21:46:38.4520952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4521301Z return func(*args, **kwargs) 2025-08-14T21:46:38.4521653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4522018Z return func(*args, **kwargs) 2025-08-14T21:46:38.4522382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4522740Z return func(*args, **kwargs) 2025-08-14T21:46:38.4523122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:46:38.4523601Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.4523805Z 2025-08-14T21:46:38.4523899Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.4524109Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.4524350Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4524714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4525036Z return mod(**inputs) 2025-08-14T21:46:38.4525394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4525926Z return func(*args, **kwargs) 2025-08-14T21:46:38.4526291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4526700Z return func(*args, **kwargs) 2025-08-14T21:46:38.4527066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4527463Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4527898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4528303Z outputs = self.layoutlm( 2025-08-14T21:46:38.4528659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4529027Z return func(*args, **kwargs) 2025-08-14T21:46:38.4529375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4530581Z return func(*args, **kwargs) 2025-08-14T21:46:38.4530934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4531315Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4531738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4532176Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4532560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4532920Z return func(*args, **kwargs) 2025-08-14T21:46:38.4533270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4533664Z return func(*args, **kwargs) 2025-08-14T21:46:38.4534012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4534375Z return func(*args, **kwargs) 2025-08-14T21:46:38.4534567Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4534913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4535256Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4535654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4536052Z layer_outputs = layer_module( 2025-08-14T21:46:38.4536389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4536752Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4537126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4537493Z return func(*args, **kwargs) 2025-08-14T21:46:38.4538150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4538539Z return func(*args, **kwargs) 2025-08-14T21:46:38.4538901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4539264Z return func(*args, **kwargs) 2025-08-14T21:46:38.4539653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.4540072Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.4540456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4540822Z return func(*args, **kwargs) 2025-08-14T21:46:38.4541173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4541608Z return func(*args, **kwargs) 2025-08-14T21:46:38.4541951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4542301Z return func(*args, **kwargs) 2025-08-14T21:46:38.4542681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:46:38.4543139Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:38.4543587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:46:38.4544001Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.4544150Z 2025-08-14T21:46:38.4544266Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4544617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4544989Z return mod(**inputs) 2025-08-14T21:46:38.4545329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4545685Z return func(*args, **kwargs) 2025-08-14T21:46:38.4546020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4546378Z return func(*args, **kwargs) 2025-08-14T21:46:38.4546707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4547057Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4547456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4547873Z outputs = self.layoutlm( 2025-08-14T21:46:38.4548234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4548599Z return func(*args, **kwargs) 2025-08-14T21:46:38.4548938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4549298Z return func(*args, **kwargs) 2025-08-14T21:46:38.4549621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4549956Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4550352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4550751Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4551114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4551460Z return func(*args, **kwargs) 2025-08-14T21:46:38.4551802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4552161Z return func(*args, **kwargs) 2025-08-14T21:46:38.4552500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4552858Z return func(*args, **kwargs) 2025-08-14T21:46:38.4553049Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4553388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4553725Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4554117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4554514Z layer_outputs = layer_module( 2025-08-14T21:46:38.4554855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4555227Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4555592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4555955Z return func(*args, **kwargs) 2025-08-14T21:46:38.4556293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4556649Z return func(*args, **kwargs) 2025-08-14T21:46:38.4556991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4557432Z return func(*args, **kwargs) 2025-08-14T21:46:38.4557807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.4558253Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.4558701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.4559091Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.4559499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.4559956Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.4560385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:46:38.4560778Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.4560919Z 2025-08-14T21:46:38.4561022Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4561376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4561707Z return mod(**inputs) 2025-08-14T21:46:38.4562057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4562420Z return func(*args, **kwargs) 2025-08-14T21:46:38.4562766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4563114Z return func(*args, **kwargs) 2025-08-14T21:46:38.4563438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4563783Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4564170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4564559Z outputs = self.layoutlm( 2025-08-14T21:46:38.4564904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4565260Z return func(*args, **kwargs) 2025-08-14T21:46:38.4565690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4566101Z return func(*args, **kwargs) 2025-08-14T21:46:38.4566467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4566830Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4567279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4567663Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4568012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4568358Z return func(*args, **kwargs) 2025-08-14T21:46:38.4568697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4569071Z return func(*args, **kwargs) 2025-08-14T21:46:38.4569411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4569749Z return func(*args, **kwargs) 2025-08-14T21:46:38.4569933Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4570310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4570639Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4571023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4571406Z layer_outputs = layer_module( 2025-08-14T21:46:38.4571731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4572108Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4572486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4572862Z return func(*args, **kwargs) 2025-08-14T21:46:38.4573194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4573545Z return func(*args, **kwargs) 2025-08-14T21:46:38.4573882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4574233Z return func(*args, **kwargs) 2025-08-14T21:46:38.4574592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.4574987Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.4575386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.4575758Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.4576161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.4576612Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.4577030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:46:38.4577442Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:38.4577808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:38.4578119Z return self.act(input) 2025-08-14T21:46:38.4578224Z 2025-08-14T21:46:38.4578335Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4578692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4579015Z return mod(**inputs) 2025-08-14T21:46:38.4579369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4579732Z return func(*args, **kwargs) 2025-08-14T21:46:38.4580092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4580460Z return func(*args, **kwargs) 2025-08-14T21:46:38.4580775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4581106Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4581488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4581874Z outputs = self.layoutlm( 2025-08-14T21:46:38.4582213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4582591Z return func(*args, **kwargs) 2025-08-14T21:46:38.4582943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4583297Z return func(*args, **kwargs) 2025-08-14T21:46:38.4583612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4583963Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4584341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4584721Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4585063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4585412Z return func(*args, **kwargs) 2025-08-14T21:46:38.4585745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4586102Z return func(*args, **kwargs) 2025-08-14T21:46:38.4586468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4586820Z return func(*args, **kwargs) 2025-08-14T21:46:38.4587003Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4587326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4587658Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4588037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4588409Z layer_outputs = layer_module( 2025-08-14T21:46:38.4588749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4589093Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4589451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4589791Z return func(*args, **kwargs) 2025-08-14T21:46:38.4590126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4590474Z return func(*args, **kwargs) 2025-08-14T21:46:38.4590804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4591155Z return func(*args, **kwargs) 2025-08-14T21:46:38.4591520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.4591911Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.4592297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.4592678Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.4593093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:46:38.4593579Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:38.4594029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:46:38.4594437Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.4594577Z 2025-08-14T21:46:38.4594688Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4595042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4595369Z return mod(**inputs) 2025-08-14T21:46:38.4595717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4596110Z return func(*args, **kwargs) 2025-08-14T21:46:38.4596460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4596827Z return func(*args, **kwargs) 2025-08-14T21:46:38.4597158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4597502Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4597906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4598302Z outputs = self.layoutlm( 2025-08-14T21:46:38.4598656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4599091Z return func(*args, **kwargs) 2025-08-14T21:46:38.4599445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4599857Z return func(*args, **kwargs) 2025-08-14T21:46:38.4600184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4600534Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4600928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4601412Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4601770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4602133Z return func(*args, **kwargs) 2025-08-14T21:46:38.4602486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4602865Z return func(*args, **kwargs) 2025-08-14T21:46:38.4603214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4603580Z return func(*args, **kwargs) 2025-08-14T21:46:38.4603776Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4604118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4604469Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4604871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4605273Z layer_outputs = layer_module( 2025-08-14T21:46:38.4605693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4606070Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4606468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4606914Z return func(*args, **kwargs) 2025-08-14T21:46:38.4607274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4607638Z return func(*args, **kwargs) 2025-08-14T21:46:38.4607995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4608352Z return func(*args, **kwargs) 2025-08-14T21:46:38.4608734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.4609144Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.4609513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4609877Z return func(*args, **kwargs) 2025-08-14T21:46:38.4610227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4610613Z return func(*args, **kwargs) 2025-08-14T21:46:38.4610955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4611319Z return func(*args, **kwargs) 2025-08-14T21:46:38.4611688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.4612075Z self_outputs = self.self( 2025-08-14T21:46:38.4612436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4612801Z return func(*args, **kwargs) 2025-08-14T21:46:38.4613151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4613510Z return func(*args, **kwargs) 2025-08-14T21:46:38.4613883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4614258Z return func(*args, **kwargs) 2025-08-14T21:46:38.4614632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:46:38.4615089Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.4615292Z 2025-08-14T21:46:38.4615403Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4615784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4616121Z return mod(**inputs) 2025-08-14T21:46:38.4616469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4616855Z return func(*args, **kwargs) 2025-08-14T21:46:38.4617209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4617573Z return func(*args, **kwargs) 2025-08-14T21:46:38.4617892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4618229Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4618608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4618991Z outputs = self.layoutlm( 2025-08-14T21:46:38.4619333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4619686Z return func(*args, **kwargs) 2025-08-14T21:46:38.4620021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4620376Z return func(*args, **kwargs) 2025-08-14T21:46:38.4620698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4621033Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4621417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4621804Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4622159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4622506Z return func(*args, **kwargs) 2025-08-14T21:46:38.4622857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4623219Z return func(*args, **kwargs) 2025-08-14T21:46:38.4623568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4623927Z return func(*args, **kwargs) 2025-08-14T21:46:38.4624139Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4624482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4624805Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4625176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4625558Z layer_outputs = layer_module( 2025-08-14T21:46:38.4625894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4626236Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4626616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4626977Z return func(*args, **kwargs) 2025-08-14T21:46:38.4627330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4627728Z return func(*args, **kwargs) 2025-08-14T21:46:38.4628068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4628419Z return func(*args, **kwargs) 2025-08-14T21:46:38.4628783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.4629176Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.4629538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4629882Z return func(*args, **kwargs) 2025-08-14T21:46:38.4630237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4630592Z return func(*args, **kwargs) 2025-08-14T21:46:38.4630939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4631286Z return func(*args, **kwargs) 2025-08-14T21:46:38.4631655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.4632046Z self_outputs = self.self( 2025-08-14T21:46:38.4632387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4632744Z return func(*args, **kwargs) 2025-08-14T21:46:38.4633091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4633454Z return func(*args, **kwargs) 2025-08-14T21:46:38.4633787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4634142Z return func(*args, **kwargs) 2025-08-14T21:46:38.4634515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:46:38.4634960Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.4635145Z 2025-08-14T21:46:38.4635247Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4635595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4635912Z return mod(**inputs) 2025-08-14T21:46:38.4636239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4636600Z return func(*args, **kwargs) 2025-08-14T21:46:38.4636942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4637297Z return func(*args, **kwargs) 2025-08-14T21:46:38.4637763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4638125Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4638514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4638893Z outputs = self.layoutlm( 2025-08-14T21:46:38.4639242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4639601Z return func(*args, **kwargs) 2025-08-14T21:46:38.4639946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4640293Z return func(*args, **kwargs) 2025-08-14T21:46:38.4640624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4640968Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4641480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4641893Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4642269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4642642Z return func(*args, **kwargs) 2025-08-14T21:46:38.4642993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4643367Z return func(*args, **kwargs) 2025-08-14T21:46:38.4643727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4644098Z return func(*args, **kwargs) 2025-08-14T21:46:38.4644312Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4644664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4645020Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4645409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4645869Z layer_outputs = layer_module( 2025-08-14T21:46:38.4646223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4646582Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4646950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4647314Z return func(*args, **kwargs) 2025-08-14T21:46:38.4647670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4648032Z return func(*args, **kwargs) 2025-08-14T21:46:38.4648395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4648765Z return func(*args, **kwargs) 2025-08-14T21:46:38.4649151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.4649557Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.4649935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4650297Z return func(*args, **kwargs) 2025-08-14T21:46:38.4650644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4651010Z return func(*args, **kwargs) 2025-08-14T21:46:38.4651366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4651765Z return func(*args, **kwargs) 2025-08-14T21:46:38.4652141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.4652536Z self_outputs = self.self( 2025-08-14T21:46:38.4652891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4653257Z return func(*args, **kwargs) 2025-08-14T21:46:38.4653598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4653958Z return func(*args, **kwargs) 2025-08-14T21:46:38.4654303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4654658Z return func(*args, **kwargs) 2025-08-14T21:46:38.4655043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:46:38.4655562Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.4655758Z 2025-08-14T21:46:38.4655844Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.4656048Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.4656281Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4656628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4656939Z return mod(**inputs) 2025-08-14T21:46:38.4657277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4657637Z return func(*args, **kwargs) 2025-08-14T21:46:38.4657982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4658351Z return func(*args, **kwargs) 2025-08-14T21:46:38.4658680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4659032Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4659409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4659796Z outputs = self.layoutlm( 2025-08-14T21:46:38.4660140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4660490Z return func(*args, **kwargs) 2025-08-14T21:46:38.4660821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4661173Z return func(*args, **kwargs) 2025-08-14T21:46:38.4661495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4661827Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4662214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4662603Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4662962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4663306Z return func(*args, **kwargs) 2025-08-14T21:46:38.4663648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4663997Z return func(*args, **kwargs) 2025-08-14T21:46:38.4664327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4664674Z return func(*args, **kwargs) 2025-08-14T21:46:38.4664863Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4665200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4665556Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4665944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4666329Z layer_outputs = layer_module( 2025-08-14T21:46:38.4666656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4667003Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4667366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4667713Z return func(*args, **kwargs) 2025-08-14T21:46:38.4668046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4668402Z return func(*args, **kwargs) 2025-08-14T21:46:38.4668758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4669130Z return func(*args, **kwargs) 2025-08-14T21:46:38.4669493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.4669893Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.4670251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4670589Z return func(*args, **kwargs) 2025-08-14T21:46:38.4670925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4671270Z return func(*args, **kwargs) 2025-08-14T21:46:38.4671623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4671962Z return func(*args, **kwargs) 2025-08-14T21:46:38.4672342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:46:38.4672791Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:38.4673233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:46:38.4673642Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.4673782Z 2025-08-14T21:46:38.4673893Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4674229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4674530Z return mod(**inputs) 2025-08-14T21:46:38.4674857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4675202Z return func(*args, **kwargs) 2025-08-14T21:46:38.4675538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4675876Z return func(*args, **kwargs) 2025-08-14T21:46:38.4676188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4676524Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4676902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4677297Z outputs = self.layoutlm( 2025-08-14T21:46:38.4677633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4677976Z return func(*args, **kwargs) 2025-08-14T21:46:38.4678299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4678674Z return func(*args, **kwargs) 2025-08-14T21:46:38.4678993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4679320Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4679695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4680080Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4680428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4680771Z return func(*args, **kwargs) 2025-08-14T21:46:38.4681111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4681466Z return func(*args, **kwargs) 2025-08-14T21:46:38.4681800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4682187Z return func(*args, **kwargs) 2025-08-14T21:46:38.4682396Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4682761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4683122Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4683555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4683987Z layer_outputs = layer_module( 2025-08-14T21:46:38.4684343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4684737Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4685154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4685633Z return func(*args, **kwargs) 2025-08-14T21:46:38.4686018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4686428Z return func(*args, **kwargs) 2025-08-14T21:46:38.4686862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4687266Z return func(*args, **kwargs) 2025-08-14T21:46:38.4687665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.4688110Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.4688543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.4688975Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.4689424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.4689940Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.4690422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:46:38.4690856Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.4691009Z 2025-08-14T21:46:38.4691119Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4691512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4691870Z return mod(**inputs) 2025-08-14T21:46:38.4692229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4692624Z return func(*args, **kwargs) 2025-08-14T21:46:38.4693002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4693414Z return func(*args, **kwargs) 2025-08-14T21:46:38.4693749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4694105Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4694520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4694918Z outputs = self.layoutlm( 2025-08-14T21:46:38.4695255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4695603Z return func(*args, **kwargs) 2025-08-14T21:46:38.4695932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4696286Z return func(*args, **kwargs) 2025-08-14T21:46:38.4696605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4696990Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4697375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4697762Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4698117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4698462Z return func(*args, **kwargs) 2025-08-14T21:46:38.4698802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4699153Z return func(*args, **kwargs) 2025-08-14T21:46:38.4699494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4699860Z return func(*args, **kwargs) 2025-08-14T21:46:38.4700055Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4700395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4700719Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4701095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4701474Z layer_outputs = layer_module( 2025-08-14T21:46:38.4701798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4702135Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4702500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4702866Z return func(*args, **kwargs) 2025-08-14T21:46:38.4703231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4703624Z return func(*args, **kwargs) 2025-08-14T21:46:38.4703981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4704352Z return func(*args, **kwargs) 2025-08-14T21:46:38.4704713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.4705110Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.4705509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.4705877Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.4706271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.4706727Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.4707168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:46:38.4707576Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:38.4707935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:38.4708257Z return self.act(input) 2025-08-14T21:46:38.4708366Z 2025-08-14T21:46:38.4708477Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4708821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4709140Z return mod(**inputs) 2025-08-14T21:46:38.4709479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4709840Z return func(*args, **kwargs) 2025-08-14T21:46:38.4710180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4710561Z return func(*args, **kwargs) 2025-08-14T21:46:38.4710901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4711235Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4711621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4712014Z outputs = self.layoutlm( 2025-08-14T21:46:38.4712371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4712742Z return func(*args, **kwargs) 2025-08-14T21:46:38.4713122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4713501Z return func(*args, **kwargs) 2025-08-14T21:46:38.4713838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4714187Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4714576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4714972Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4715322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4715679Z return func(*args, **kwargs) 2025-08-14T21:46:38.4716022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4716373Z return func(*args, **kwargs) 2025-08-14T21:46:38.4716721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4717077Z return func(*args, **kwargs) 2025-08-14T21:46:38.4717265Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4717600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4717944Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4718345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4718737Z layer_outputs = layer_module( 2025-08-14T21:46:38.4719083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4719472Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4719879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4720284Z return func(*args, **kwargs) 2025-08-14T21:46:38.4720673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4721090Z return func(*args, **kwargs) 2025-08-14T21:46:38.4721468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4721866Z return func(*args, **kwargs) 2025-08-14T21:46:38.4722294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.4722737Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.4723163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.4723590Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.4724051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:46:38.4724584Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:38.4725112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:46:38.4725654Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.4725811Z 2025-08-14T21:46:38.4725937Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4726340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4726693Z return mod(**inputs) 2025-08-14T21:46:38.4727075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4727486Z return func(*args, **kwargs) 2025-08-14T21:46:38.4727862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4728284Z return func(*args, **kwargs) 2025-08-14T21:46:38.4728654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4729043Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4729476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4729909Z outputs = self.layoutlm( 2025-08-14T21:46:38.4730299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4730689Z return func(*args, **kwargs) 2025-08-14T21:46:38.4731074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4731469Z return func(*args, **kwargs) 2025-08-14T21:46:38.4731833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4732208Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4732648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4733092Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4733484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4733892Z return func(*args, **kwargs) 2025-08-14T21:46:38.4734279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4734686Z return func(*args, **kwargs) 2025-08-14T21:46:38.4735063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4735456Z return func(*args, **kwargs) 2025-08-14T21:46:38.4735665Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4736041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4736480Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4736916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4737351Z layer_outputs = layer_module( 2025-08-14T21:46:38.4737877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4738290Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4738704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4739108Z return func(*args, **kwargs) 2025-08-14T21:46:38.4739497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4739914Z return func(*args, **kwargs) 2025-08-14T21:46:38.4740304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4740772Z return func(*args, **kwargs) 2025-08-14T21:46:38.4741156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.4741571Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.4741949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4742310Z return func(*args, **kwargs) 2025-08-14T21:46:38.4742682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4743081Z return func(*args, **kwargs) 2025-08-14T21:46:38.4743483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4743890Z return func(*args, **kwargs) 2025-08-14T21:46:38.4744303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.4744717Z self_outputs = self.self( 2025-08-14T21:46:38.4745071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4745442Z return func(*args, **kwargs) 2025-08-14T21:46:38.4745794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4746148Z return func(*args, **kwargs) 2025-08-14T21:46:38.4746502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4746864Z return func(*args, **kwargs) 2025-08-14T21:46:38.4747249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:46:38.4747725Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.4747933Z 2025-08-14T21:46:38.4748038Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4748398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4748721Z return mod(**inputs) 2025-08-14T21:46:38.4749059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4749420Z return func(*args, **kwargs) 2025-08-14T21:46:38.4749773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4750127Z return func(*args, **kwargs) 2025-08-14T21:46:38.4750460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4750813Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4751242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4751634Z outputs = self.layoutlm( 2025-08-14T21:46:38.4751988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4752374Z return func(*args, **kwargs) 2025-08-14T21:46:38.4752738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4753123Z return func(*args, **kwargs) 2025-08-14T21:46:38.4753478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4753829Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4754221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4754646Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4755029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4755393Z return func(*args, **kwargs) 2025-08-14T21:46:38.4755735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4756083Z return func(*args, **kwargs) 2025-08-14T21:46:38.4756420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4756765Z return func(*args, **kwargs) 2025-08-14T21:46:38.4756951Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4757286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4757631Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4758019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4758405Z layer_outputs = layer_module( 2025-08-14T21:46:38.4758736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4759078Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4759440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4759796Z return func(*args, **kwargs) 2025-08-14T21:46:38.4760134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4760488Z return func(*args, **kwargs) 2025-08-14T21:46:38.4760830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4761181Z return func(*args, **kwargs) 2025-08-14T21:46:38.4761548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.4761948Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.4762317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4762680Z return func(*args, **kwargs) 2025-08-14T21:46:38.4763036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4763423Z return func(*args, **kwargs) 2025-08-14T21:46:38.4763793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4764178Z return func(*args, **kwargs) 2025-08-14T21:46:38.4764584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.4765029Z self_outputs = self.self( 2025-08-14T21:46:38.4765412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4765866Z return func(*args, **kwargs) 2025-08-14T21:46:38.4766252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4766649Z return func(*args, **kwargs) 2025-08-14T21:46:38.4767027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4767475Z return func(*args, **kwargs) 2025-08-14T21:46:38.4767849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:46:38.4768309Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.4768501Z 2025-08-14T21:46:38.4768604Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4768997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4769317Z return mod(**inputs) 2025-08-14T21:46:38.4769655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4770002Z return func(*args, **kwargs) 2025-08-14T21:46:38.4770349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4770700Z return func(*args, **kwargs) 2025-08-14T21:46:38.4771014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4771355Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4771756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4772148Z outputs = self.layoutlm( 2025-08-14T21:46:38.4772496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4772862Z return func(*args, **kwargs) 2025-08-14T21:46:38.4773212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4773568Z return func(*args, **kwargs) 2025-08-14T21:46:38.4773897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4774253Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4774641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4775030Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4775396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4775769Z return func(*args, **kwargs) 2025-08-14T21:46:38.4776115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4776481Z return func(*args, **kwargs) 2025-08-14T21:46:38.4776832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4777206Z return func(*args, **kwargs) 2025-08-14T21:46:38.4777387Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4777734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4778078Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4778459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4778866Z layer_outputs = layer_module( 2025-08-14T21:46:38.4779205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4779552Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4779914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4780269Z return func(*args, **kwargs) 2025-08-14T21:46:38.4780610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4780965Z return func(*args, **kwargs) 2025-08-14T21:46:38.4781303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4781660Z return func(*args, **kwargs) 2025-08-14T21:46:38.4782033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.4782457Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.4782852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4783216Z return func(*args, **kwargs) 2025-08-14T21:46:38.4783568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4783937Z return func(*args, **kwargs) 2025-08-14T21:46:38.4784276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4784632Z return func(*args, **kwargs) 2025-08-14T21:46:38.4784995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.4785392Z self_outputs = self.self( 2025-08-14T21:46:38.4785739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4786096Z return func(*args, **kwargs) 2025-08-14T21:46:38.4786431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4786784Z return func(*args, **kwargs) 2025-08-14T21:46:38.4787124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4787467Z return func(*args, **kwargs) 2025-08-14T21:46:38.4787838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:46:38.4788295Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.4788489Z 2025-08-14T21:46:38.4788575Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.4788776Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.4789005Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4789352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4789657Z return mod(**inputs) 2025-08-14T21:46:38.4790003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4790367Z return func(*args, **kwargs) 2025-08-14T21:46:38.4790718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4791072Z return func(*args, **kwargs) 2025-08-14T21:46:38.4791398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4791744Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4792133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4792551Z outputs = self.layoutlm( 2025-08-14T21:46:38.4792905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4793266Z return func(*args, **kwargs) 2025-08-14T21:46:38.4793610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4793971Z return func(*args, **kwargs) 2025-08-14T21:46:38.4794304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4794652Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4795041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4795444Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4795862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4796269Z return func(*args, **kwargs) 2025-08-14T21:46:38.4796623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4796984Z return func(*args, **kwargs) 2025-08-14T21:46:38.4797332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4797685Z return func(*args, **kwargs) 2025-08-14T21:46:38.4797876Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4798220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4798561Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4798975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4799388Z layer_outputs = layer_module( 2025-08-14T21:46:38.4799743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4800098Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4800479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4800853Z return func(*args, **kwargs) 2025-08-14T21:46:38.4801205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4801580Z return func(*args, **kwargs) 2025-08-14T21:46:38.4801942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4802316Z return func(*args, **kwargs) 2025-08-14T21:46:38.4802714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.4803161Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.4803571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4803960Z return func(*args, **kwargs) 2025-08-14T21:46:38.4804332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4804728Z return func(*args, **kwargs) 2025-08-14T21:46:38.4805105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4805569Z return func(*args, **kwargs) 2025-08-14T21:46:38.4805998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:46:38.4806490Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:38.4807000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:46:38.4807407Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.4807552Z 2025-08-14T21:46:38.4807657Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4808023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4808343Z return mod(**inputs) 2025-08-14T21:46:38.4808686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4809054Z return func(*args, **kwargs) 2025-08-14T21:46:38.4809405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4809763Z return func(*args, **kwargs) 2025-08-14T21:46:38.4810103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4810462Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4810856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4811242Z outputs = self.layoutlm( 2025-08-14T21:46:38.4811586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4811940Z return func(*args, **kwargs) 2025-08-14T21:46:38.4812274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4812631Z return func(*args, **kwargs) 2025-08-14T21:46:38.4812962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4813328Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4813718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4814122Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4814494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4814841Z return func(*args, **kwargs) 2025-08-14T21:46:38.4815185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4815536Z return func(*args, **kwargs) 2025-08-14T21:46:38.4815876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4816222Z return func(*args, **kwargs) 2025-08-14T21:46:38.4816410Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4816748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4817082Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4817472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4817863Z layer_outputs = layer_module( 2025-08-14T21:46:38.4818199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4818537Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4818898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4819254Z return func(*args, **kwargs) 2025-08-14T21:46:38.4819588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4819945Z return func(*args, **kwargs) 2025-08-14T21:46:38.4820287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4820662Z return func(*args, **kwargs) 2025-08-14T21:46:38.4821033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.4821435Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.4821827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.4822203Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.4822618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.4823082Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.4823517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:46:38.4823905Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.4824068Z 2025-08-14T21:46:38.4824186Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4824539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4824854Z return mod(**inputs) 2025-08-14T21:46:38.4825181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4825538Z return func(*args, **kwargs) 2025-08-14T21:46:38.4825889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4826224Z return func(*args, **kwargs) 2025-08-14T21:46:38.4826538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4826880Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4827257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4827634Z outputs = self.layoutlm( 2025-08-14T21:46:38.4827977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4828335Z return func(*args, **kwargs) 2025-08-14T21:46:38.4828671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4829095Z return func(*args, **kwargs) 2025-08-14T21:46:38.4829413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4829750Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4830125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4830518Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4830871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4831221Z return func(*args, **kwargs) 2025-08-14T21:46:38.4831551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4831912Z return func(*args, **kwargs) 2025-08-14T21:46:38.4832265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4832658Z return func(*args, **kwargs) 2025-08-14T21:46:38.4832864Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4833231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4833601Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4834024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4834512Z layer_outputs = layer_module( 2025-08-14T21:46:38.4834866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4835223Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4835601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4835968Z return func(*args, **kwargs) 2025-08-14T21:46:38.4836325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4836684Z return func(*args, **kwargs) 2025-08-14T21:46:38.4837043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4837414Z return func(*args, **kwargs) 2025-08-14T21:46:38.4837918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.4838089Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.4838354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.4838439Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.4838745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.4838874Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.4839161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:46:38.4839279Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:38.4839532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:38.4839612Z return self.act(input) 2025-08-14T21:46:38.4839618Z 2025-08-14T21:46:38.4839730Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4839949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4840019Z return mod(**inputs) 2025-08-14T21:46:38.4840269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4840349Z return func(*args, **kwargs) 2025-08-14T21:46:38.4840602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4840680Z return func(*args, **kwargs) 2025-08-14T21:46:38.4840917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4840999Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4841300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4841379Z outputs = self.layoutlm( 2025-08-14T21:46:38.4841634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4841713Z return func(*args, **kwargs) 2025-08-14T21:46:38.4841970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4842050Z return func(*args, **kwargs) 2025-08-14T21:46:38.4842280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4842359Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4842659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4842776Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4843042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4843115Z return func(*args, **kwargs) 2025-08-14T21:46:38.4843369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4843450Z return func(*args, **kwargs) 2025-08-14T21:46:38.4843704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4843776Z return func(*args, **kwargs) 2025-08-14T21:46:38.4843869Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4844103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4844190Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4844476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4844601Z layer_outputs = layer_module( 2025-08-14T21:46:38.4844852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4844938Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4845194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4845274Z return func(*args, **kwargs) 2025-08-14T21:46:38.4845597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4845688Z return func(*args, **kwargs) 2025-08-14T21:46:38.4845970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4846045Z return func(*args, **kwargs) 2025-08-14T21:46:38.4846349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.4846445Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.4846726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.4846821Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.4847150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:46:38.4847302Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:38.4847594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:46:38.4847688Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.4847693Z 2025-08-14T21:46:38.4847818Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4848040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4848124Z return mod(**inputs) 2025-08-14T21:46:38.4848389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4848464Z return func(*args, **kwargs) 2025-08-14T21:46:38.4848731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4848803Z return func(*args, **kwargs) 2025-08-14T21:46:38.4849038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4849129Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4849422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4849530Z outputs = self.layoutlm( 2025-08-14T21:46:38.4849808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4849881Z return func(*args, **kwargs) 2025-08-14T21:46:38.4850145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4850216Z return func(*args, **kwargs) 2025-08-14T21:46:38.4850448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4850536Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4850821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4850908Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4851171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4851282Z return func(*args, **kwargs) 2025-08-14T21:46:38.4851566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4851638Z return func(*args, **kwargs) 2025-08-14T21:46:38.4851898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4851971Z return func(*args, **kwargs) 2025-08-14T21:46:38.4852055Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4852292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4852371Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4852658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4852739Z layer_outputs = layer_module( 2025-08-14T21:46:38.4852954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4853037Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4853269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4853336Z return func(*args, **kwargs) 2025-08-14T21:46:38.4853571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4853636Z return func(*args, **kwargs) 2025-08-14T21:46:38.4853866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4853938Z return func(*args, **kwargs) 2025-08-14T21:46:38.4854205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.4854296Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.4854533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4854598Z return func(*args, **kwargs) 2025-08-14T21:46:38.4854839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4854903Z return func(*args, **kwargs) 2025-08-14T21:46:38.4855136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4855208Z return func(*args, **kwargs) 2025-08-14T21:46:38.4855466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.4855542Z self_outputs = self.self( 2025-08-14T21:46:38.4855778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4855862Z return func(*args, **kwargs) 2025-08-14T21:46:38.4856101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4856166Z return func(*args, **kwargs) 2025-08-14T21:46:38.4856408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4856473Z return func(*args, **kwargs) 2025-08-14T21:46:38.4856733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:46:38.4856881Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.4856886Z 2025-08-14T21:46:38.4856988Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4857183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4857273Z return mod(**inputs) 2025-08-14T21:46:38.4857519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4857592Z return func(*args, **kwargs) 2025-08-14T21:46:38.4857822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4857888Z return func(*args, **kwargs) 2025-08-14T21:46:38.4858103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4858175Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4858432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4858508Z outputs = self.layoutlm( 2025-08-14T21:46:38.4858755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4858835Z return func(*args, **kwargs) 2025-08-14T21:46:38.4859065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4859129Z return func(*args, **kwargs) 2025-08-14T21:46:38.4859345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4859419Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4859691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4859764Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4859998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4860070Z return func(*args, **kwargs) 2025-08-14T21:46:38.4860307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4860375Z return func(*args, **kwargs) 2025-08-14T21:46:38.4860616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4860680Z return func(*args, **kwargs) 2025-08-14T21:46:38.4860767Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4860979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4861051Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4861324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4861395Z layer_outputs = layer_module( 2025-08-14T21:46:38.4861614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4861720Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4861950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4862024Z return func(*args, **kwargs) 2025-08-14T21:46:38.4862251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4862316Z return func(*args, **kwargs) 2025-08-14T21:46:38.4862556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4862623Z return func(*args, **kwargs) 2025-08-14T21:46:38.4862884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.4862977Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.4863208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4863317Z return func(*args, **kwargs) 2025-08-14T21:46:38.4863549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4863616Z return func(*args, **kwargs) 2025-08-14T21:46:38.4863856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4863922Z return func(*args, **kwargs) 2025-08-14T21:46:38.4864194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.4864265Z self_outputs = self.self( 2025-08-14T21:46:38.4864502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4864591Z return func(*args, **kwargs) 2025-08-14T21:46:38.4864824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4864891Z return func(*args, **kwargs) 2025-08-14T21:46:38.4865124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4865187Z return func(*args, **kwargs) 2025-08-14T21:46:38.4865455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:46:38.4865589Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.4865593Z 2025-08-14T21:46:38.4865694Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4865895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4865961Z return mod(**inputs) 2025-08-14T21:46:38.4866188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4866264Z return func(*args, **kwargs) 2025-08-14T21:46:38.4866493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4866564Z return func(*args, **kwargs) 2025-08-14T21:46:38.4866769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4866841Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4867106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4867174Z outputs = self.layoutlm( 2025-08-14T21:46:38.4867409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4867476Z return func(*args, **kwargs) 2025-08-14T21:46:38.4867740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4867824Z return func(*args, **kwargs) 2025-08-14T21:46:38.4868026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4868097Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4868359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4868432Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4868661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4868724Z return func(*args, **kwargs) 2025-08-14T21:46:38.4868950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4869022Z return func(*args, **kwargs) 2025-08-14T21:46:38.4869285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4869353Z return func(*args, **kwargs) 2025-08-14T21:46:38.4869437Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4869645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4869723Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4869983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4870052Z layer_outputs = layer_module( 2025-08-14T21:46:38.4870274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4870379Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4870602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4870674Z return func(*args, **kwargs) 2025-08-14T21:46:38.4870893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4870964Z return func(*args, **kwargs) 2025-08-14T21:46:38.4871193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4871257Z return func(*args, **kwargs) 2025-08-14T21:46:38.4871515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.4871592Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.4871821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4871895Z return func(*args, **kwargs) 2025-08-14T21:46:38.4872133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4872213Z return func(*args, **kwargs) 2025-08-14T21:46:38.4872469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4872538Z return func(*args, **kwargs) 2025-08-14T21:46:38.4872831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.4872906Z self_outputs = self.self( 2025-08-14T21:46:38.4873169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4873238Z return func(*args, **kwargs) 2025-08-14T21:46:38.4873493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4873588Z return func(*args, **kwargs) 2025-08-14T21:46:38.4873842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4873907Z return func(*args, **kwargs) 2025-08-14T21:46:38.4874178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:46:38.4874319Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.4874324Z 2025-08-14T21:46:38.4874410Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.4874485Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.4874586Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4874787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4874854Z return mod(**inputs) 2025-08-14T21:46:38.4875088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4875197Z return func(*args, **kwargs) 2025-08-14T21:46:38.4875424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4875498Z return func(*args, **kwargs) 2025-08-14T21:46:38.4875705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4875777Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4876039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4876108Z outputs = self.layoutlm( 2025-08-14T21:46:38.4876341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4876432Z return func(*args, **kwargs) 2025-08-14T21:46:38.4876667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4876741Z return func(*args, **kwargs) 2025-08-14T21:46:38.4876956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4877029Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4877304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4877389Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4877628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4877692Z return func(*args, **kwargs) 2025-08-14T21:46:38.4877924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4877997Z return func(*args, **kwargs) 2025-08-14T21:46:38.4878224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4878290Z return func(*args, **kwargs) 2025-08-14T21:46:38.4878370Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4878575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4878651Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4878907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4878975Z layer_outputs = layer_module( 2025-08-14T21:46:38.4879193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4879270Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4879499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4879594Z return func(*args, **kwargs) 2025-08-14T21:46:38.4879825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4879896Z return func(*args, **kwargs) 2025-08-14T21:46:38.4880128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4880193Z return func(*args, **kwargs) 2025-08-14T21:46:38.4880453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.4880532Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.4880758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4880833Z return func(*args, **kwargs) 2025-08-14T21:46:38.4881081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4881172Z return func(*args, **kwargs) 2025-08-14T21:46:38.4881408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4881475Z return func(*args, **kwargs) 2025-08-14T21:46:38.4881750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:46:38.4881880Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:38.4882156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:46:38.4882246Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.4882250Z 2025-08-14T21:46:38.4882374Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4882596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4882669Z return mod(**inputs) 2025-08-14T21:46:38.4882919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4882996Z return func(*args, **kwargs) 2025-08-14T21:46:38.4883244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4883320Z return func(*args, **kwargs) 2025-08-14T21:46:38.4883545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4883621Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4883908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4883981Z outputs = self.layoutlm( 2025-08-14T21:46:38.4884232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4884312Z return func(*args, **kwargs) 2025-08-14T21:46:38.4884557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4884634Z return func(*args, **kwargs) 2025-08-14T21:46:38.4884857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4884933Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4885218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4885300Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4885644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4885749Z return func(*args, **kwargs) 2025-08-14T21:46:38.4886006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4886089Z return func(*args, **kwargs) 2025-08-14T21:46:38.4886343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4886415Z return func(*args, **kwargs) 2025-08-14T21:46:38.4886510Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4886755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4886840Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4887122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4887201Z layer_outputs = layer_module( 2025-08-14T21:46:38.4887437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4887557Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4887809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4887889Z return func(*args, **kwargs) 2025-08-14T21:46:38.4888138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4888217Z return func(*args, **kwargs) 2025-08-14T21:46:38.4888468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4888537Z return func(*args, **kwargs) 2025-08-14T21:46:38.4888876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.4888968Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.4889252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.4889341Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.4889661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.4889796Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.4890080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:46:38.4890168Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.4890172Z 2025-08-14T21:46:38.4890289Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4890503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4890581Z return mod(**inputs) 2025-08-14T21:46:38.4890842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4890914Z return func(*args, **kwargs) 2025-08-14T21:46:38.4891176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4891249Z return func(*args, **kwargs) 2025-08-14T21:46:38.4891479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4891563Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4891845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4891927Z outputs = self.layoutlm( 2025-08-14T21:46:38.4892180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4892276Z return func(*args, **kwargs) 2025-08-14T21:46:38.4892540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4892619Z return func(*args, **kwargs) 2025-08-14T21:46:38.4892844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4892916Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4893183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4893264Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4893503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4893584Z return func(*args, **kwargs) 2025-08-14T21:46:38.4893830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4893914Z return func(*args, **kwargs) 2025-08-14T21:46:38.4894176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4894247Z return func(*args, **kwargs) 2025-08-14T21:46:38.4894326Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4894547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4894619Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4894885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4894966Z layer_outputs = layer_module( 2025-08-14T21:46:38.4895199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4895283Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4895522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4895587Z return func(*args, **kwargs) 2025-08-14T21:46:38.4895825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4895893Z return func(*args, **kwargs) 2025-08-14T21:46:38.4896122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4896195Z return func(*args, **kwargs) 2025-08-14T21:46:38.4896456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.4896544Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.4896801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.4896880Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.4897186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.4897306Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.4897577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:46:38.4897690Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:38.4897901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:38.4897979Z return self.act(input) 2025-08-14T21:46:38.4897983Z 2025-08-14T21:46:38.4898086Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4898284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4898379Z return mod(**inputs) 2025-08-14T21:46:38.4898616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4898690Z return func(*args, **kwargs) 2025-08-14T21:46:38.4898921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4898988Z return func(*args, **kwargs) 2025-08-14T21:46:38.4899206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4899280Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4899539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4899615Z outputs = self.layoutlm( 2025-08-14T21:46:38.4899846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4899943Z return func(*args, **kwargs) 2025-08-14T21:46:38.4900193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4900260Z return func(*args, **kwargs) 2025-08-14T21:46:38.4900483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4900557Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4900825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4900898Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4920567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4920904Z return func(*args, **kwargs) 2025-08-14T21:46:38.4921222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4921309Z return func(*args, **kwargs) 2025-08-14T21:46:38.4921564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4921634Z return func(*args, **kwargs) 2025-08-14T21:46:38.4921724Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4921945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4922025Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4922310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4922389Z layer_outputs = layer_module( 2025-08-14T21:46:38.4922614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4922711Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4922958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4923036Z return func(*args, **kwargs) 2025-08-14T21:46:38.4923279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4923351Z return func(*args, **kwargs) 2025-08-14T21:46:38.4923611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4923683Z return func(*args, **kwargs) 2025-08-14T21:46:38.4923975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.4924064Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.4924330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.4924450Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.4924757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:46:38.4924894Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:38.4925186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:46:38.4925277Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.4925284Z 2025-08-14T21:46:38.4925409Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4925726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4925804Z return mod(**inputs) 2025-08-14T21:46:38.4926071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4926226Z return func(*args, **kwargs) 2025-08-14T21:46:38.4926491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4926564Z return func(*args, **kwargs) 2025-08-14T21:46:38.4926800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4926894Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4927198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4927281Z outputs = self.layoutlm( 2025-08-14T21:46:38.4927544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4927636Z return func(*args, **kwargs) 2025-08-14T21:46:38.4927907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4927988Z return func(*args, **kwargs) 2025-08-14T21:46:38.4928226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4928326Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4928627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4928708Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4928968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4929038Z return func(*args, **kwargs) 2025-08-14T21:46:38.4929298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4929369Z return func(*args, **kwargs) 2025-08-14T21:46:38.4929624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4929703Z return func(*args, **kwargs) 2025-08-14T21:46:38.4929786Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4930016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4930099Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4930402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4930485Z layer_outputs = layer_module( 2025-08-14T21:46:38.4930720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4930808Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4931065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4931170Z return func(*args, **kwargs) 2025-08-14T21:46:38.4931418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4931496Z return func(*args, **kwargs) 2025-08-14T21:46:38.4931744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4931815Z return func(*args, **kwargs) 2025-08-14T21:46:38.4932116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.4932206Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.4932452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4932532Z return func(*args, **kwargs) 2025-08-14T21:46:38.4932797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4932893Z return func(*args, **kwargs) 2025-08-14T21:46:38.4933141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4933211Z return func(*args, **kwargs) 2025-08-14T21:46:38.4933501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.4933578Z self_outputs = self.self( 2025-08-14T21:46:38.4933826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4933902Z return func(*args, **kwargs) 2025-08-14T21:46:38.4934167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4934244Z return func(*args, **kwargs) 2025-08-14T21:46:38.4934498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4934567Z return func(*args, **kwargs) 2025-08-14T21:46:38.4934855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:46:38.4935012Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.4935017Z 2025-08-14T21:46:38.4935138Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4935348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4935420Z return mod(**inputs) 2025-08-14T21:46:38.4935672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4935743Z return func(*args, **kwargs) 2025-08-14T21:46:38.4935993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4936072Z return func(*args, **kwargs) 2025-08-14T21:46:38.4936292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4936375Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4936649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4936725Z outputs = self.layoutlm( 2025-08-14T21:46:38.4936980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4937048Z return func(*args, **kwargs) 2025-08-14T21:46:38.4937294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4937371Z return func(*args, **kwargs) 2025-08-14T21:46:38.4937731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4937826Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4938109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4938189Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4938449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4938522Z return func(*args, **kwargs) 2025-08-14T21:46:38.4938779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4938849Z return func(*args, **kwargs) 2025-08-14T21:46:38.4939100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4939179Z return func(*args, **kwargs) 2025-08-14T21:46:38.4939346Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4939575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4939661Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4939942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4940025Z layer_outputs = layer_module( 2025-08-14T21:46:38.4940257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4940340Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4940594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4940691Z return func(*args, **kwargs) 2025-08-14T21:46:38.4940940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4941020Z return func(*args, **kwargs) 2025-08-14T21:46:38.4941267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4941347Z return func(*args, **kwargs) 2025-08-14T21:46:38.4941626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.4941713Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.4941963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4942033Z return func(*args, **kwargs) 2025-08-14T21:46:38.4942279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4942359Z return func(*args, **kwargs) 2025-08-14T21:46:38.4942611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4942689Z return func(*args, **kwargs) 2025-08-14T21:46:38.4942966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.4943043Z self_outputs = self.self( 2025-08-14T21:46:38.4943298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4943368Z return func(*args, **kwargs) 2025-08-14T21:46:38.4943613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4943690Z return func(*args, **kwargs) 2025-08-14T21:46:38.4943937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4944048Z return func(*args, **kwargs) 2025-08-14T21:46:38.4944332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:46:38.4944480Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.4944485Z 2025-08-14T21:46:38.4944604Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4944819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4944896Z return mod(**inputs) 2025-08-14T21:46:38.4945146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4945216Z return func(*args, **kwargs) 2025-08-14T21:46:38.4945471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4945544Z return func(*args, **kwargs) 2025-08-14T21:46:38.4945805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4945893Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4946176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4946258Z outputs = self.layoutlm( 2025-08-14T21:46:38.4946505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4946576Z return func(*args, **kwargs) 2025-08-14T21:46:38.4946832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4946902Z return func(*args, **kwargs) 2025-08-14T21:46:38.4947144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4947233Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4947519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4947606Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4947856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4947927Z return func(*args, **kwargs) 2025-08-14T21:46:38.4948182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4948251Z return func(*args, **kwargs) 2025-08-14T21:46:38.4948504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4948575Z return func(*args, **kwargs) 2025-08-14T21:46:38.4948657Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4948892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4948969Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4949247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4949331Z layer_outputs = layer_module( 2025-08-14T21:46:38.4949560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4949648Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4949897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4949979Z return func(*args, **kwargs) 2025-08-14T21:46:38.4950214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4950305Z return func(*args, **kwargs) 2025-08-14T21:46:38.4950539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4950610Z return func(*args, **kwargs) 2025-08-14T21:46:38.4950866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.4950950Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.4951176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4951241Z return func(*args, **kwargs) 2025-08-14T21:46:38.4951479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4951544Z return func(*args, **kwargs) 2025-08-14T21:46:38.4951773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4951864Z return func(*args, **kwargs) 2025-08-14T21:46:38.4952145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.4952226Z self_outputs = self.self( 2025-08-14T21:46:38.4952459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4952525Z return func(*args, **kwargs) 2025-08-14T21:46:38.4952766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4952831Z return func(*args, **kwargs) 2025-08-14T21:46:38.4953071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4953159Z return func(*args, **kwargs) 2025-08-14T21:46:38.4953458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:46:38.4953621Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.4953626Z 2025-08-14T21:46:38.4953721Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.4953801Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.4953910Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4954108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4954181Z return mod(**inputs) 2025-08-14T21:46:38.4954413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4954480Z return func(*args, **kwargs) 2025-08-14T21:46:38.4954719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4954786Z return func(*args, **kwargs) 2025-08-14T21:46:38.4955012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4955092Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4955352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4955428Z outputs = self.layoutlm( 2025-08-14T21:46:38.4955658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4955722Z return func(*args, **kwargs) 2025-08-14T21:46:38.4955954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4956018Z return func(*args, **kwargs) 2025-08-14T21:46:38.4956225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4956322Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4956587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4956668Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4956896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4956961Z return func(*args, **kwargs) 2025-08-14T21:46:38.4957196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4957261Z return func(*args, **kwargs) 2025-08-14T21:46:38.4957497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4957563Z return func(*args, **kwargs) 2025-08-14T21:46:38.4957639Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4957856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4957957Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4958217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4958295Z layer_outputs = layer_module( 2025-08-14T21:46:38.4958507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4958589Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4958818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4958882Z return func(*args, **kwargs) 2025-08-14T21:46:38.4959179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4959248Z return func(*args, **kwargs) 2025-08-14T21:46:38.4959488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4959562Z return func(*args, **kwargs) 2025-08-14T21:46:38.4959828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.4959917Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.4960155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4960222Z return func(*args, **kwargs) 2025-08-14T21:46:38.4960466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4960532Z return func(*args, **kwargs) 2025-08-14T21:46:38.4960769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4960844Z return func(*args, **kwargs) 2025-08-14T21:46:38.4961112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:46:38.4961248Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:38.4961514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:46:38.4961597Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.4961601Z 2025-08-14T21:46:38.4961712Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4961912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4961984Z return mod(**inputs) 2025-08-14T21:46:38.4962265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4962716Z return func(*args, **kwargs) 2025-08-14T21:46:38.4963003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4963076Z return func(*args, **kwargs) 2025-08-14T21:46:38.4963307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4963397Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4963697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4963779Z outputs = self.layoutlm( 2025-08-14T21:46:38.4964041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4964110Z return func(*args, **kwargs) 2025-08-14T21:46:38.4964375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4964468Z return func(*args, **kwargs) 2025-08-14T21:46:38.4964711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4964801Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4965095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4965186Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4965453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4965617Z return func(*args, **kwargs) 2025-08-14T21:46:38.4965891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4965988Z return func(*args, **kwargs) 2025-08-14T21:46:38.4966258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4966335Z return func(*args, **kwargs) 2025-08-14T21:46:38.4966418Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4966670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4966743Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4967008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4967089Z layer_outputs = layer_module( 2025-08-14T21:46:38.4967320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4967413Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4967670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4967743Z return func(*args, **kwargs) 2025-08-14T21:46:38.4968007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4968081Z return func(*args, **kwargs) 2025-08-14T21:46:38.4968336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4968418Z return func(*args, **kwargs) 2025-08-14T21:46:38.4968723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.4968823Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.4969105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.4969189Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.4969525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.4969683Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.4969976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:46:38.4970067Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.4970072Z 2025-08-14T21:46:38.4970185Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4970409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4970481Z return mod(**inputs) 2025-08-14T21:46:38.4970737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4970818Z return func(*args, **kwargs) 2025-08-14T21:46:38.4971075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4971176Z return func(*args, **kwargs) 2025-08-14T21:46:38.4971433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4971515Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4971814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4971890Z outputs = self.layoutlm( 2025-08-14T21:46:38.4972147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4972227Z return func(*args, **kwargs) 2025-08-14T21:46:38.4972484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4972586Z return func(*args, **kwargs) 2025-08-14T21:46:38.4972821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4972906Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4973201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4973282Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4973539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4973618Z return func(*args, **kwargs) 2025-08-14T21:46:38.4973873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4973951Z return func(*args, **kwargs) 2025-08-14T21:46:38.4974207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4974279Z return func(*args, **kwargs) 2025-08-14T21:46:38.4974371Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4974608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4974694Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4974982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4975059Z layer_outputs = layer_module( 2025-08-14T21:46:38.4975305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4975389Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4975646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4975725Z return func(*args, **kwargs) 2025-08-14T21:46:38.4975996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4976084Z return func(*args, **kwargs) 2025-08-14T21:46:38.4976327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4976392Z return func(*args, **kwargs) 2025-08-14T21:46:38.4976669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.4976757Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.4977038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.4977125Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.4977451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.4977587Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.4977902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:46:38.4978022Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:38.4978253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:38.4978327Z return self.act(input) 2025-08-14T21:46:38.4978332Z 2025-08-14T21:46:38.4978456Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4978658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4978724Z return mod(**inputs) 2025-08-14T21:46:38.4978966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4979050Z return func(*args, **kwargs) 2025-08-14T21:46:38.4979285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4979365Z return func(*args, **kwargs) 2025-08-14T21:46:38.4979579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4979660Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4979923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4979992Z outputs = self.layoutlm( 2025-08-14T21:46:38.4980234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4980301Z return func(*args, **kwargs) 2025-08-14T21:46:38.4980536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4980611Z return func(*args, **kwargs) 2025-08-14T21:46:38.4980827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4980910Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4981176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4981249Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4981490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4981557Z return func(*args, **kwargs) 2025-08-14T21:46:38.4981794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4981859Z return func(*args, **kwargs) 2025-08-14T21:46:38.4982092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4982187Z return func(*args, **kwargs) 2025-08-14T21:46:38.4982268Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4982498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4982584Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4982870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4982950Z layer_outputs = layer_module( 2025-08-14T21:46:38.4983185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4983267Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4983528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4983599Z return func(*args, **kwargs) 2025-08-14T21:46:38.4983850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4983966Z return func(*args, **kwargs) 2025-08-14T21:46:38.4984213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4984287Z return func(*args, **kwargs) 2025-08-14T21:46:38.4984549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.4984633Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.4984896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.4984972Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.4985283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:46:38.4985425Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:38.4985691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:46:38.4985781Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.4985785Z 2025-08-14T21:46:38.4985887Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4986085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4986157Z return mod(**inputs) 2025-08-14T21:46:38.4986389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4986463Z return func(*args, **kwargs) 2025-08-14T21:46:38.4986697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4986763Z return func(*args, **kwargs) 2025-08-14T21:46:38.4986988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4987062Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4987326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4987403Z outputs = self.layoutlm( 2025-08-14T21:46:38.4987637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4987713Z return func(*args, **kwargs) 2025-08-14T21:46:38.4987944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4988010Z return func(*args, **kwargs) 2025-08-14T21:46:38.4988231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4988323Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4988593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4988673Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4988910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4988985Z return func(*args, **kwargs) 2025-08-14T21:46:38.4989217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4989283Z return func(*args, **kwargs) 2025-08-14T21:46:38.4989525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4989592Z return func(*args, **kwargs) 2025-08-14T21:46:38.4989675Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4989889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4989996Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4990270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4990340Z layer_outputs = layer_module( 2025-08-14T21:46:38.4990560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4990646Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4990882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4990956Z return func(*args, **kwargs) 2025-08-14T21:46:38.4991206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4991275Z return func(*args, **kwargs) 2025-08-14T21:46:38.4991525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4991592Z return func(*args, **kwargs) 2025-08-14T21:46:38.4991858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.4991953Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.4992203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4992285Z return func(*args, **kwargs) 2025-08-14T21:46:38.4992532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4992602Z return func(*args, **kwargs) 2025-08-14T21:46:38.4992858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4992929Z return func(*args, **kwargs) 2025-08-14T21:46:38.4993220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.4993290Z self_outputs = self.self( 2025-08-14T21:46:38.4993521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4993594Z return func(*args, **kwargs) 2025-08-14T21:46:38.4993821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4993886Z return func(*args, **kwargs) 2025-08-14T21:46:38.4994122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4994186Z return func(*args, **kwargs) 2025-08-14T21:46:38.4994453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:46:38.4994619Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.4994623Z 2025-08-14T21:46:38.4994722Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.4994920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.4994985Z return mod(**inputs) 2025-08-14T21:46:38.4995220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4995285Z return func(*args, **kwargs) 2025-08-14T21:46:38.4995510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4995582Z return func(*args, **kwargs) 2025-08-14T21:46:38.4995790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4995864Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4996165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.4996233Z outputs = self.layoutlm( 2025-08-14T21:46:38.4996467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4996532Z return func(*args, **kwargs) 2025-08-14T21:46:38.4996760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4996832Z return func(*args, **kwargs) 2025-08-14T21:46:38.4997038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4997110Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4997389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.4997466Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.4997704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4997769Z return func(*args, **kwargs) 2025-08-14T21:46:38.4997994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4998067Z return func(*args, **kwargs) 2025-08-14T21:46:38.4998293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4998360Z return func(*args, **kwargs) 2025-08-14T21:46:38.4998442Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.4998649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.4998726Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.4998989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.4999059Z layer_outputs = layer_module( 2025-08-14T21:46:38.4999279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.4999356Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.4999591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4999656Z return func(*args, **kwargs) 2025-08-14T21:46:38.4999886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.4999960Z return func(*args, **kwargs) 2025-08-14T21:46:38.5000194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5000278Z return func(*args, **kwargs) 2025-08-14T21:46:38.5000553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5000636Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5000880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5000946Z return func(*args, **kwargs) 2025-08-14T21:46:38.5001181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5001256Z return func(*args, **kwargs) 2025-08-14T21:46:38.5001493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5001557Z return func(*args, **kwargs) 2025-08-14T21:46:38.5001832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.5001963Z self_outputs = self.self( 2025-08-14T21:46:38.5002205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5002272Z return func(*args, **kwargs) 2025-08-14T21:46:38.5002503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5002576Z return func(*args, **kwargs) 2025-08-14T21:46:38.5002812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5002880Z return func(*args, **kwargs) 2025-08-14T21:46:38.5003149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:46:38.5003303Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.5003309Z 2025-08-14T21:46:38.5003421Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5003622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5003688Z return mod(**inputs) 2025-08-14T21:46:38.5003933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5004001Z return func(*args, **kwargs) 2025-08-14T21:46:38.5004244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5004310Z return func(*args, **kwargs) 2025-08-14T21:46:38.5004527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5004613Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5004898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5004977Z outputs = self.layoutlm( 2025-08-14T21:46:38.5005238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5005307Z return func(*args, **kwargs) 2025-08-14T21:46:38.5005785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5005866Z return func(*args, **kwargs) 2025-08-14T21:46:38.5006102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5006190Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5006491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5006572Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5006859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5006932Z return func(*args, **kwargs) 2025-08-14T21:46:38.5007187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5007265Z return func(*args, **kwargs) 2025-08-14T21:46:38.5007499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5007575Z return func(*args, **kwargs) 2025-08-14T21:46:38.5007654Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5007875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5007955Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5008209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5008306Z layer_outputs = layer_module( 2025-08-14T21:46:38.5008553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5008638Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5008891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5008961Z return func(*args, **kwargs) 2025-08-14T21:46:38.5009213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5009284Z return func(*args, **kwargs) 2025-08-14T21:46:38.5009533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5009611Z return func(*args, **kwargs) 2025-08-14T21:46:38.5009952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5010045Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5010298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5010366Z return func(*args, **kwargs) 2025-08-14T21:46:38.5010620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5010690Z return func(*args, **kwargs) 2025-08-14T21:46:38.5010934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5011012Z return func(*args, **kwargs) 2025-08-14T21:46:38.5011310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.5011386Z self_outputs = self.self( 2025-08-14T21:46:38.5011647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5011719Z return func(*args, **kwargs) 2025-08-14T21:46:38.5011974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5012044Z return func(*args, **kwargs) 2025-08-14T21:46:38.5012289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5012367Z return func(*args, **kwargs) 2025-08-14T21:46:38.5012646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:46:38.5012807Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.5012812Z 2025-08-14T21:46:38.5012899Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.5013002Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.5013124Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5013336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5013408Z return mod(**inputs) 2025-08-14T21:46:38.5013667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5013739Z return func(*args, **kwargs) 2025-08-14T21:46:38.5013992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5014062Z return func(*args, **kwargs) 2025-08-14T21:46:38.5014284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5014370Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5014670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5014778Z outputs = self.layoutlm( 2025-08-14T21:46:38.5015036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5015106Z return func(*args, **kwargs) 2025-08-14T21:46:38.5015356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5015427Z return func(*args, **kwargs) 2025-08-14T21:46:38.5015648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5015734Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5016033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5016126Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5016382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5016451Z return func(*args, **kwargs) 2025-08-14T21:46:38.5016688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5016754Z return func(*args, **kwargs) 2025-08-14T21:46:38.5016984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5017059Z return func(*args, **kwargs) 2025-08-14T21:46:38.5017133Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5017343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5017425Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5017690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5017772Z layer_outputs = layer_module( 2025-08-14T21:46:38.5017990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5018067Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5018307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5018374Z return func(*args, **kwargs) 2025-08-14T21:46:38.5018612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5018679Z return func(*args, **kwargs) 2025-08-14T21:46:38.5018911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5018985Z return func(*args, **kwargs) 2025-08-14T21:46:38.5019245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5019348Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5019593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5019661Z return func(*args, **kwargs) 2025-08-14T21:46:38.5019903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5019970Z return func(*args, **kwargs) 2025-08-14T21:46:38.5020202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5020277Z return func(*args, **kwargs) 2025-08-14T21:46:38.5020546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:46:38.5020676Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:38.5020987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:46:38.5021073Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.5021080Z 2025-08-14T21:46:38.5021188Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5021386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5021452Z return mod(**inputs) 2025-08-14T21:46:38.5021693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5021762Z return func(*args, **kwargs) 2025-08-14T21:46:38.5022000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5022082Z return func(*args, **kwargs) 2025-08-14T21:46:38.5022301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5022388Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5022667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5022740Z outputs = self.layoutlm( 2025-08-14T21:46:38.5023003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5023068Z return func(*args, **kwargs) 2025-08-14T21:46:38.5023308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5023374Z return func(*args, **kwargs) 2025-08-14T21:46:38.5023584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5023665Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5023930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5024004Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5024252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5024318Z return func(*args, **kwargs) 2025-08-14T21:46:38.5024566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5024632Z return func(*args, **kwargs) 2025-08-14T21:46:38.5024870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5024943Z return func(*args, **kwargs) 2025-08-14T21:46:38.5025018Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5025236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5025335Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5025600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5025677Z layer_outputs = layer_module( 2025-08-14T21:46:38.5025896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5025975Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5026221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5026288Z return func(*args, **kwargs) 2025-08-14T21:46:38.5026529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5026597Z return func(*args, **kwargs) 2025-08-14T21:46:38.5026831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5026936Z return func(*args, **kwargs) 2025-08-14T21:46:38.5027203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.5027288Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.5027552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.5027630Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.5027938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.5028063Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.5028342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:46:38.5028436Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.5028441Z 2025-08-14T21:46:38.5028543Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5028750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5028816Z return mod(**inputs) 2025-08-14T21:46:38.5029052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5029127Z return func(*args, **kwargs) 2025-08-14T21:46:38.5029360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5029426Z return func(*args, **kwargs) 2025-08-14T21:46:38.5029649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5029723Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5029997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5030068Z outputs = self.layoutlm( 2025-08-14T21:46:38.5030303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5030376Z return func(*args, **kwargs) 2025-08-14T21:46:38.5030611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5030677Z return func(*args, **kwargs) 2025-08-14T21:46:38.5030896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5030971Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5031241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5031336Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5031578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5031656Z return func(*args, **kwargs) 2025-08-14T21:46:38.5031902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5031972Z return func(*args, **kwargs) 2025-08-14T21:46:38.5032227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5032296Z return func(*args, **kwargs) 2025-08-14T21:46:38.5032382Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5032606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5032683Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5032969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5033079Z layer_outputs = layer_module( 2025-08-14T21:46:38.5033309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5033398Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5033649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5033723Z return func(*args, **kwargs) 2025-08-14T21:46:38.5033959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5034026Z return func(*args, **kwargs) 2025-08-14T21:46:38.5034296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5034367Z return func(*args, **kwargs) 2025-08-14T21:46:38.5034661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.5034752Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.5035028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.5035116Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.5035436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.5035562Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.5035853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:46:38.5035975Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:38.5036206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:38.5036284Z return self.act(input) 2025-08-14T21:46:38.5036287Z 2025-08-14T21:46:38.5036399Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5036618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5036689Z return mod(**inputs) 2025-08-14T21:46:38.5036949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5037021Z return func(*args, **kwargs) 2025-08-14T21:46:38.5037271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5037346Z return func(*args, **kwargs) 2025-08-14T21:46:38.5037576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5037857Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5038157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5038232Z outputs = self.layoutlm( 2025-08-14T21:46:38.5038485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5038554Z return func(*args, **kwargs) 2025-08-14T21:46:38.5038800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5038877Z return func(*args, **kwargs) 2025-08-14T21:46:38.5039101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5039178Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5039486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5039617Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5039894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5039965Z return func(*args, **kwargs) 2025-08-14T21:46:38.5040226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5040304Z return func(*args, **kwargs) 2025-08-14T21:46:38.5040564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5040634Z return func(*args, **kwargs) 2025-08-14T21:46:38.5040721Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5040948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5041059Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5041346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5041424Z layer_outputs = layer_module( 2025-08-14T21:46:38.5041666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5041749Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5042017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5042089Z return func(*args, **kwargs) 2025-08-14T21:46:38.5042349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5042426Z return func(*args, **kwargs) 2025-08-14T21:46:38.5042688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5042776Z return func(*args, **kwargs) 2025-08-14T21:46:38.5043067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.5043160Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.5043434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.5043514Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.5043835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:46:38.5043976Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:38.5044267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:46:38.5044362Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.5044406Z 2025-08-14T21:46:38.5044518Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5044739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5044809Z return mod(**inputs) 2025-08-14T21:46:38.5045069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5045148Z return func(*args, **kwargs) 2025-08-14T21:46:38.5045405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5045536Z return func(*args, **kwargs) 2025-08-14T21:46:38.5045770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5045849Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5046140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5046241Z outputs = self.layoutlm( 2025-08-14T21:46:38.5046512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5046591Z return func(*args, **kwargs) 2025-08-14T21:46:38.5046848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5046922Z return func(*args, **kwargs) 2025-08-14T21:46:38.5047135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5047208Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5047479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5047569Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5047807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5047887Z return func(*args, **kwargs) 2025-08-14T21:46:38.5048120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5048196Z return func(*args, **kwargs) 2025-08-14T21:46:38.5048430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5048496Z return func(*args, **kwargs) 2025-08-14T21:46:38.5048579Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5048794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5048868Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5049140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5049211Z layer_outputs = layer_module( 2025-08-14T21:46:38.5049436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5049515Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5049747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5049820Z return func(*args, **kwargs) 2025-08-14T21:46:38.5050052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5050124Z return func(*args, **kwargs) 2025-08-14T21:46:38.5050357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5050423Z return func(*args, **kwargs) 2025-08-14T21:46:38.5050693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5050799Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5051032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5051106Z return func(*args, **kwargs) 2025-08-14T21:46:38.5051337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5051410Z return func(*args, **kwargs) 2025-08-14T21:46:38.5051643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5051708Z return func(*args, **kwargs) 2025-08-14T21:46:38.5051977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.5052048Z self_outputs = self.self( 2025-08-14T21:46:38.5052281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5052393Z return func(*args, **kwargs) 2025-08-14T21:46:38.5052628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5052701Z return func(*args, **kwargs) 2025-08-14T21:46:38.5052934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5053000Z return func(*args, **kwargs) 2025-08-14T21:46:38.5053269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:46:38.5053413Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.5053418Z 2025-08-14T21:46:38.5053540Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5053740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5053811Z return mod(**inputs) 2025-08-14T21:46:38.5054049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5054117Z return func(*args, **kwargs) 2025-08-14T21:46:38.5054350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5054425Z return func(*args, **kwargs) 2025-08-14T21:46:38.5054638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5054719Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5054981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5055052Z outputs = self.layoutlm( 2025-08-14T21:46:38.5055295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5055367Z return func(*args, **kwargs) 2025-08-14T21:46:38.5055599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5055674Z return func(*args, **kwargs) 2025-08-14T21:46:38.5055883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5055966Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5056229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5056304Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5056545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5056613Z return func(*args, **kwargs) 2025-08-14T21:46:38.5056879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5056945Z return func(*args, **kwargs) 2025-08-14T21:46:38.5057178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5057250Z return func(*args, **kwargs) 2025-08-14T21:46:38.5057326Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5057536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5057617Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5057879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5057955Z layer_outputs = layer_module( 2025-08-14T21:46:38.5058173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5058285Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5058527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5058593Z return func(*args, **kwargs) 2025-08-14T21:46:38.5058822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5058895Z return func(*args, **kwargs) 2025-08-14T21:46:38.5059127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5059198Z return func(*args, **kwargs) 2025-08-14T21:46:38.5059458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5059556Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5059805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5059872Z return func(*args, **kwargs) 2025-08-14T21:46:38.5060104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5060178Z return func(*args, **kwargs) 2025-08-14T21:46:38.5060410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5060483Z return func(*args, **kwargs) 2025-08-14T21:46:38.5060744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.5060814Z self_outputs = self.self( 2025-08-14T21:46:38.5061057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5061123Z return func(*args, **kwargs) 2025-08-14T21:46:38.5061370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5061437Z return func(*args, **kwargs) 2025-08-14T21:46:38.5061673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5061746Z return func(*args, **kwargs) 2025-08-14T21:46:38.5062009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:46:38.5062147Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.5062151Z 2025-08-14T21:46:38.5062261Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5062460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5062534Z return mod(**inputs) 2025-08-14T21:46:38.5062787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5062855Z return func(*args, **kwargs) 2025-08-14T21:46:38.5063096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5063161Z return func(*args, **kwargs) 2025-08-14T21:46:38.5063370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5063452Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5063716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5063790Z outputs = self.layoutlm( 2025-08-14T21:46:38.5064024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5064090Z return func(*args, **kwargs) 2025-08-14T21:46:38.5064359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5064426Z return func(*args, **kwargs) 2025-08-14T21:46:38.5064638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5064718Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5064981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5065063Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5065295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5065362Z return func(*args, **kwargs) 2025-08-14T21:46:38.5065624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5065693Z return func(*args, **kwargs) 2025-08-14T21:46:38.5065934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5066001Z return func(*args, **kwargs) 2025-08-14T21:46:38.5066077Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5066296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5066367Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5066630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5066707Z layer_outputs = layer_module( 2025-08-14T21:46:38.5066922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5067008Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5067252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5067317Z return func(*args, **kwargs) 2025-08-14T21:46:38.5067547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5067613Z return func(*args, **kwargs) 2025-08-14T21:46:38.5067843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5067917Z return func(*args, **kwargs) 2025-08-14T21:46:38.5068182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5068271Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5068505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5068592Z return func(*args, **kwargs) 2025-08-14T21:46:38.5068839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5068904Z return func(*args, **kwargs) 2025-08-14T21:46:38.5069140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5069213Z return func(*args, **kwargs) 2025-08-14T21:46:38.5069479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.5069554Z self_outputs = self.self( 2025-08-14T21:46:38.5069791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5069857Z return func(*args, **kwargs) 2025-08-14T21:46:38.5070104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5070189Z return func(*args, **kwargs) 2025-08-14T21:46:38.5070472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5070539Z return func(*args, **kwargs) 2025-08-14T21:46:38.5070806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:46:38.5070957Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.5070961Z 2025-08-14T21:46:38.5071043Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.5071124Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.5071239Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5071449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5071539Z return mod(**inputs) 2025-08-14T21:46:38.5071796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5071869Z return func(*args, **kwargs) 2025-08-14T21:46:38.5072124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5072196Z return func(*args, **kwargs) 2025-08-14T21:46:38.5072421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5072506Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5072786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5072875Z outputs = self.layoutlm( 2025-08-14T21:46:38.5073111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5073178Z return func(*args, **kwargs) 2025-08-14T21:46:38.5073425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5073494Z return func(*args, **kwargs) 2025-08-14T21:46:38.5073709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5073792Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5074057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5074137Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5074381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5074445Z return func(*args, **kwargs) 2025-08-14T21:46:38.5074684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5074767Z return func(*args, **kwargs) 2025-08-14T21:46:38.5075005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5075071Z return func(*args, **kwargs) 2025-08-14T21:46:38.5075145Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5075363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5075433Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5075689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5075765Z layer_outputs = layer_module( 2025-08-14T21:46:38.5075977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5076060Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5076309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5076390Z return func(*args, **kwargs) 2025-08-14T21:46:38.5076628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5076692Z return func(*args, **kwargs) 2025-08-14T21:46:38.5076921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5076993Z return func(*args, **kwargs) 2025-08-14T21:46:38.5077250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5077335Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5077580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5077646Z return func(*args, **kwargs) 2025-08-14T21:46:38.5077890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5077955Z return func(*args, **kwargs) 2025-08-14T21:46:38.5078185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5078258Z return func(*args, **kwargs) 2025-08-14T21:46:38.5078517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:46:38.5078649Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:38.5078905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:46:38.5078986Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.5078990Z 2025-08-14T21:46:38.5079097Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5079296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5079367Z return mod(**inputs) 2025-08-14T21:46:38.5079596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5079661Z return func(*args, **kwargs) 2025-08-14T21:46:38.5079900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5079966Z return func(*args, **kwargs) 2025-08-14T21:46:38.5080178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5080256Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5080513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5080603Z outputs = self.layoutlm( 2025-08-14T21:46:38.5080833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5080897Z return func(*args, **kwargs) 2025-08-14T21:46:38.5081129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5081193Z return func(*args, **kwargs) 2025-08-14T21:46:38.5081398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5081475Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5081730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5081808Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5082043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5082131Z return func(*args, **kwargs) 2025-08-14T21:46:38.5082392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5082461Z return func(*args, **kwargs) 2025-08-14T21:46:38.5082703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5082771Z return func(*args, **kwargs) 2025-08-14T21:46:38.5082847Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5083076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5083153Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5083461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5083547Z layer_outputs = layer_module( 2025-08-14T21:46:38.5083782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5083873Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5084133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5084203Z return func(*args, **kwargs) 2025-08-14T21:46:38.5084465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5084536Z return func(*args, **kwargs) 2025-08-14T21:46:38.5084792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5084869Z return func(*args, **kwargs) 2025-08-14T21:46:38.5085163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.5085264Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.5085624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.5085713Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.5086035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.5086164Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.5086449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:46:38.5086538Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.5086542Z 2025-08-14T21:46:38.5086651Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5086870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5086965Z return mod(**inputs) 2025-08-14T21:46:38.5087221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5087304Z return func(*args, **kwargs) 2025-08-14T21:46:38.5087563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5087644Z return func(*args, **kwargs) 2025-08-14T21:46:38.5087882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5087959Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5088258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5088332Z outputs = self.layoutlm( 2025-08-14T21:46:38.5088593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5088696Z return func(*args, **kwargs) 2025-08-14T21:46:38.5088981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5089060Z return func(*args, **kwargs) 2025-08-14T21:46:38.5089295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5089371Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5089657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5089733Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5089988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5090077Z return func(*args, **kwargs) 2025-08-14T21:46:38.5090345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5090427Z return func(*args, **kwargs) 2025-08-14T21:46:38.5090676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5090746Z return func(*args, **kwargs) 2025-08-14T21:46:38.5090834Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5091066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5091150Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5091448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5091523Z layer_outputs = layer_module( 2025-08-14T21:46:38.5091759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5091842Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5092093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5092175Z return func(*args, **kwargs) 2025-08-14T21:46:38.5092421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5092497Z return func(*args, **kwargs) 2025-08-14T21:46:38.5092745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5092815Z return func(*args, **kwargs) 2025-08-14T21:46:38.5093100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.5093189Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.5093462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.5093571Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.5093890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.5094023Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.5094307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:46:38.5094418Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:38.5094635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:38.5094705Z return self.act(input) 2025-08-14T21:46:38.5094709Z 2025-08-14T21:46:38.5094819Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5095020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5095103Z return mod(**inputs) 2025-08-14T21:46:38.5095361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5095440Z return func(*args, **kwargs) 2025-08-14T21:46:38.5095666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5095737Z return func(*args, **kwargs) 2025-08-14T21:46:38.5095940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5096022Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5096280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5096365Z outputs = self.layoutlm( 2025-08-14T21:46:38.5096609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5096680Z return func(*args, **kwargs) 2025-08-14T21:46:38.5096915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5096987Z return func(*args, **kwargs) 2025-08-14T21:46:38.5097201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5097281Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5097546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5097618Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5097862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5097929Z return func(*args, **kwargs) 2025-08-14T21:46:38.5098173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5098240Z return func(*args, **kwargs) 2025-08-14T21:46:38.5098472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5098546Z return func(*args, **kwargs) 2025-08-14T21:46:38.5098620Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5098833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5098914Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5099179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5099257Z layer_outputs = layer_module( 2025-08-14T21:46:38.5099474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5099598Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5099840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5099909Z return func(*args, **kwargs) 2025-08-14T21:46:38.5100143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5100223Z return func(*args, **kwargs) 2025-08-14T21:46:38.5100471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5100547Z return func(*args, **kwargs) 2025-08-14T21:46:38.5100826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.5100916Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.5101200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.5101318Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.5101636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:46:38.5101777Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:38.5102043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:46:38.5102137Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.5102140Z 2025-08-14T21:46:38.5102250Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5102472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5102565Z return mod(**inputs) 2025-08-14T21:46:38.5102815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5102897Z return func(*args, **kwargs) 2025-08-14T21:46:38.5103145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5103214Z return func(*args, **kwargs) 2025-08-14T21:46:38.5103446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5103523Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5103801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5103881Z outputs = self.layoutlm( 2025-08-14T21:46:38.5104132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5104210Z return func(*args, **kwargs) 2025-08-14T21:46:38.5104460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5104531Z return func(*args, **kwargs) 2025-08-14T21:46:38.5104765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5104841Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5105128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5105206Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5105455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5105532Z return func(*args, **kwargs) 2025-08-14T21:46:38.5105781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5105872Z return func(*args, **kwargs) 2025-08-14T21:46:38.5106133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5106204Z return func(*args, **kwargs) 2025-08-14T21:46:38.5106290Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5106522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5106595Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5106869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5106939Z layer_outputs = layer_module( 2025-08-14T21:46:38.5107156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5107245Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5107479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5107591Z return func(*args, **kwargs) 2025-08-14T21:46:38.5107824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5107891Z return func(*args, **kwargs) 2025-08-14T21:46:38.5108128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5108194Z return func(*args, **kwargs) 2025-08-14T21:46:38.5108457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5108546Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5108792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5108867Z return func(*args, **kwargs) 2025-08-14T21:46:38.5109105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5109175Z return func(*args, **kwargs) 2025-08-14T21:46:38.5109414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5109481Z return func(*args, **kwargs) 2025-08-14T21:46:38.5109747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.5109818Z self_outputs = self.self( 2025-08-14T21:46:38.5110052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5110129Z return func(*args, **kwargs) 2025-08-14T21:46:38.5110376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5110448Z return func(*args, **kwargs) 2025-08-14T21:46:38.5110707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5110776Z return func(*args, **kwargs) 2025-08-14T21:46:38.5111060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:46:38.5111213Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.5111217Z 2025-08-14T21:46:38.5111324Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5111539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5111608Z return mod(**inputs) 2025-08-14T21:46:38.5111868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5111944Z return func(*args, **kwargs) 2025-08-14T21:46:38.5112197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5112271Z return func(*args, **kwargs) 2025-08-14T21:46:38.5112484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5112558Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5112826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5112893Z outputs = self.layoutlm( 2025-08-14T21:46:38.5113138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5113208Z return func(*args, **kwargs) 2025-08-14T21:46:38.5113456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5113534Z return func(*args, **kwargs) 2025-08-14T21:46:38.5113800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5113880Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5114164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5114242Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5114495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5114565Z return func(*args, **kwargs) 2025-08-14T21:46:38.5114811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5114887Z return func(*args, **kwargs) 2025-08-14T21:46:38.5115158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5115229Z return func(*args, **kwargs) 2025-08-14T21:46:38.5115316Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5115533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5115614Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5115884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5115954Z layer_outputs = layer_module( 2025-08-14T21:46:38.5116181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5116257Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5116497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5116571Z return func(*args, **kwargs) 2025-08-14T21:46:38.5116811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5116885Z return func(*args, **kwargs) 2025-08-14T21:46:38.5117118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5117184Z return func(*args, **kwargs) 2025-08-14T21:46:38.5117454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5117536Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5117769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5117841Z return func(*args, **kwargs) 2025-08-14T21:46:38.5118075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5118169Z return func(*args, **kwargs) 2025-08-14T21:46:38.5118407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5118475Z return func(*args, **kwargs) 2025-08-14T21:46:38.5118745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.5118816Z self_outputs = self.self( 2025-08-14T21:46:38.5119072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5119141Z return func(*args, **kwargs) 2025-08-14T21:46:38.5119386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5119462Z return func(*args, **kwargs) 2025-08-14T21:46:38.5119712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5119821Z return func(*args, **kwargs) 2025-08-14T21:46:38.5120110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:46:38.5120260Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.5120264Z 2025-08-14T21:46:38.5120380Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5120591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5120660Z return mod(**inputs) 2025-08-14T21:46:38.5120916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5120987Z return func(*args, **kwargs) 2025-08-14T21:46:38.5121248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5121331Z return func(*args, **kwargs) 2025-08-14T21:46:38.5121559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5121646Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5121924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5121995Z outputs = self.layoutlm( 2025-08-14T21:46:38.5122245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5122314Z return func(*args, **kwargs) 2025-08-14T21:46:38.5122567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5122636Z return func(*args, **kwargs) 2025-08-14T21:46:38.5122860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5122950Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5123238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5123314Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5123577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5123646Z return func(*args, **kwargs) 2025-08-14T21:46:38.5123904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5123974Z return func(*args, **kwargs) 2025-08-14T21:46:38.5124233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5124312Z return func(*args, **kwargs) 2025-08-14T21:46:38.5124392Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5124636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5124722Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5125012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5125095Z layer_outputs = layer_module( 2025-08-14T21:46:38.5125327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5125409Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5125752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5125830Z return func(*args, **kwargs) 2025-08-14T21:46:38.5126098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5126203Z return func(*args, **kwargs) 2025-08-14T21:46:38.5126490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5126572Z return func(*args, **kwargs) 2025-08-14T21:46:38.5126877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5126968Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5127254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5127328Z return func(*args, **kwargs) 2025-08-14T21:46:38.5127598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5127688Z return func(*args, **kwargs) 2025-08-14T21:46:38.5127963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5128049Z return func(*args, **kwargs) 2025-08-14T21:46:38.5128337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.5128412Z self_outputs = self.self( 2025-08-14T21:46:38.5128686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5128758Z return func(*args, **kwargs) 2025-08-14T21:46:38.5129029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5129101Z return func(*args, **kwargs) 2025-08-14T21:46:38.5129365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5129446Z return func(*args, **kwargs) 2025-08-14T21:46:38.5129755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:46:38.5129915Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.5129926Z 2025-08-14T21:46:38.5130013Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.5130097Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.5130215Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5130443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5130515Z return mod(**inputs) 2025-08-14T21:46:38.5130784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5130856Z return func(*args, **kwargs) 2025-08-14T21:46:38.5131124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5131225Z return func(*args, **kwargs) 2025-08-14T21:46:38.5131472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5131564Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5131863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5131938Z outputs = self.layoutlm( 2025-08-14T21:46:38.5132201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5132273Z return func(*args, **kwargs) 2025-08-14T21:46:38.5132543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5132615Z return func(*args, **kwargs) 2025-08-14T21:46:38.5132855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5132964Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5133279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5133360Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5133622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5133694Z return func(*args, **kwargs) 2025-08-14T21:46:38.5133956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5134028Z return func(*args, **kwargs) 2025-08-14T21:46:38.5134280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5134375Z return func(*args, **kwargs) 2025-08-14T21:46:38.5134458Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5134694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5134782Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5135073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5135154Z layer_outputs = layer_module( 2025-08-14T21:46:38.5135392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5135474Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5135733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5135805Z return func(*args, **kwargs) 2025-08-14T21:46:38.5136050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5136128Z return func(*args, **kwargs) 2025-08-14T21:46:38.5136364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5136436Z return func(*args, **kwargs) 2025-08-14T21:46:38.5136700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5136781Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5137023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5137090Z return func(*args, **kwargs) 2025-08-14T21:46:38.5137328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5137394Z return func(*args, **kwargs) 2025-08-14T21:46:38.5137798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5137952Z return func(*args, **kwargs) 2025-08-14T21:46:38.5138237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:46:38.5138375Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:38.5138663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:46:38.5138752Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.5138756Z 2025-08-14T21:46:38.5138871Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5139082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5139152Z return mod(**inputs) 2025-08-14T21:46:38.5139411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5139513Z return func(*args, **kwargs) 2025-08-14T21:46:38.5139785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5139866Z return func(*args, **kwargs) 2025-08-14T21:46:38.5140090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5140177Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5140476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5140547Z outputs = self.layoutlm( 2025-08-14T21:46:38.5140802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5140896Z return func(*args, **kwargs) 2025-08-14T21:46:38.5141154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5141228Z return func(*args, **kwargs) 2025-08-14T21:46:38.5141452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5141546Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5141808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5141881Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5142125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5142192Z return func(*args, **kwargs) 2025-08-14T21:46:38.5142433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5142500Z return func(*args, **kwargs) 2025-08-14T21:46:38.5142744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5142820Z return func(*args, **kwargs) 2025-08-14T21:46:38.5142901Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5143127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5143212Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5143513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5143594Z layer_outputs = layer_module( 2025-08-14T21:46:38.5143827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5143909Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5144164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5144255Z return func(*args, **kwargs) 2025-08-14T21:46:38.5144508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5144585Z return func(*args, **kwargs) 2025-08-14T21:46:38.5144835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5144909Z return func(*args, **kwargs) 2025-08-14T21:46:38.5145175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.5145259Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.5145527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.5145604Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.5145935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.5146079Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.5146362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:46:38.5146458Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.5146462Z 2025-08-14T21:46:38.5146576Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5146772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5146848Z return mod(**inputs) 2025-08-14T21:46:38.5147081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5147174Z return func(*args, **kwargs) 2025-08-14T21:46:38.5147411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5147482Z return func(*args, **kwargs) 2025-08-14T21:46:38.5147703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5147778Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5148040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5148117Z outputs = self.layoutlm( 2025-08-14T21:46:38.5148351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5148427Z return func(*args, **kwargs) 2025-08-14T21:46:38.5148661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5148726Z return func(*args, **kwargs) 2025-08-14T21:46:38.5148949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5149023Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5149294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5149368Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5149602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5149676Z return func(*args, **kwargs) 2025-08-14T21:46:38.5149910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5149976Z return func(*args, **kwargs) 2025-08-14T21:46:38.5150223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5150309Z return func(*args, **kwargs) 2025-08-14T21:46:38.5150395Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5150608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5150680Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5150950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5151021Z layer_outputs = layer_module( 2025-08-14T21:46:38.5151234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5151321Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5151552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5151626Z return func(*args, **kwargs) 2025-08-14T21:46:38.5151858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5151960Z return func(*args, **kwargs) 2025-08-14T21:46:38.5152207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5152273Z return func(*args, **kwargs) 2025-08-14T21:46:38.5152536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.5152628Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.5152887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.5152971Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.5153290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.5153411Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.5153683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:46:38.5153796Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:38.5154013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:38.5154083Z return self.act(input) 2025-08-14T21:46:38.5154087Z 2025-08-14T21:46:38.5154188Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5154394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5154464Z return mod(**inputs) 2025-08-14T21:46:38.5154713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5154792Z return func(*args, **kwargs) 2025-08-14T21:46:38.5155043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5155121Z return func(*args, **kwargs) 2025-08-14T21:46:38.5155345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5155424Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5155710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5155783Z outputs = self.layoutlm( 2025-08-14T21:46:38.5156036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5156106Z return func(*args, **kwargs) 2025-08-14T21:46:38.5156353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5156450Z return func(*args, **kwargs) 2025-08-14T21:46:38.5156685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5156763Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5157053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5157128Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5157391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5157460Z return func(*args, **kwargs) 2025-08-14T21:46:38.5157713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5157791Z return func(*args, **kwargs) 2025-08-14T21:46:38.5158046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5158135Z return func(*args, **kwargs) 2025-08-14T21:46:38.5158240Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5158468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5158552Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5158829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5158902Z layer_outputs = layer_module( 2025-08-14T21:46:38.5159140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5159222Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5159493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5159572Z return func(*args, **kwargs) 2025-08-14T21:46:38.5159823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5159902Z return func(*args, **kwargs) 2025-08-14T21:46:38.5160147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5160216Z return func(*args, **kwargs) 2025-08-14T21:46:38.5160498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.5160588Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.5160866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.5160947Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.5161265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:46:38.5161418Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:38.5161699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:46:38.5161786Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.5161790Z 2025-08-14T21:46:38.5161904Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5162114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5162192Z return mod(**inputs) 2025-08-14T21:46:38.5162437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5162508Z return func(*args, **kwargs) 2025-08-14T21:46:38.5162762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5162851Z return func(*args, **kwargs) 2025-08-14T21:46:38.5163086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5163171Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5163450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5163537Z outputs = self.layoutlm( 2025-08-14T21:46:38.5163787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5163857Z return func(*args, **kwargs) 2025-08-14T21:46:38.5164112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5164182Z return func(*args, **kwargs) 2025-08-14T21:46:38.5164416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5164515Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5164822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5164909Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5165210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5165281Z return func(*args, **kwargs) 2025-08-14T21:46:38.5165625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5165704Z return func(*args, **kwargs) 2025-08-14T21:46:38.5165979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5166075Z return func(*args, **kwargs) 2025-08-14T21:46:38.5166159Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5166406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5166488Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5166780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5166866Z layer_outputs = layer_module( 2025-08-14T21:46:38.5167105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5167210Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5167471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5167543Z return func(*args, **kwargs) 2025-08-14T21:46:38.5167808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5167881Z return func(*args, **kwargs) 2025-08-14T21:46:38.5168134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5168215Z return func(*args, **kwargs) 2025-08-14T21:46:38.5168510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5168611Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5168872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5168942Z return func(*args, **kwargs) 2025-08-14T21:46:38.5169222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5169292Z return func(*args, **kwargs) 2025-08-14T21:46:38.5169563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5169656Z return func(*args, **kwargs) 2025-08-14T21:46:38.5169955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.5170036Z self_outputs = self.self( 2025-08-14T21:46:38.5170297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5170367Z return func(*args, **kwargs) 2025-08-14T21:46:38.5170630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5170700Z return func(*args, **kwargs) 2025-08-14T21:46:38.5170969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5171040Z return func(*args, **kwargs) 2025-08-14T21:46:38.5171332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:46:38.5171534Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.5171539Z 2025-08-14T21:46:38.5171648Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5171878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5171946Z return mod(**inputs) 2025-08-14T21:46:38.5172201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5172280Z return func(*args, **kwargs) 2025-08-14T21:46:38.5172537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5172608Z return func(*args, **kwargs) 2025-08-14T21:46:38.5172855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5172940Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5173241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5173314Z outputs = self.layoutlm( 2025-08-14T21:46:38.5173571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5173647Z return func(*args, **kwargs) 2025-08-14T21:46:38.5173905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5173975Z return func(*args, **kwargs) 2025-08-14T21:46:38.5174217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5174297Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5174596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5174677Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5174923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5175003Z return func(*args, **kwargs) 2025-08-14T21:46:38.5175247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5175318Z return func(*args, **kwargs) 2025-08-14T21:46:38.5175574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5175646Z return func(*args, **kwargs) 2025-08-14T21:46:38.5175733Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5175958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5176056Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5176346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5176421Z layer_outputs = layer_module( 2025-08-14T21:46:38.5176646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5176736Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5176986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5177062Z return func(*args, **kwargs) 2025-08-14T21:46:38.5177296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5177362Z return func(*args, **kwargs) 2025-08-14T21:46:38.5177603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5177707Z return func(*args, **kwargs) 2025-08-14T21:46:38.5177997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5178085Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5178330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5178407Z return func(*args, **kwargs) 2025-08-14T21:46:38.5178651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5178719Z return func(*args, **kwargs) 2025-08-14T21:46:38.5178972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5179056Z return func(*args, **kwargs) 2025-08-14T21:46:38.5179360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.5179438Z self_outputs = self.self( 2025-08-14T21:46:38.5179680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5179758Z return func(*args, **kwargs) 2025-08-14T21:46:38.5180000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5180070Z return func(*args, **kwargs) 2025-08-14T21:46:38.5180322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5180389Z return func(*args, **kwargs) 2025-08-14T21:46:38.5180692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:46:38.5180838Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.5180844Z 2025-08-14T21:46:38.5180956Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5181173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5181242Z return mod(**inputs) 2025-08-14T21:46:38.5181496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5181566Z return func(*args, **kwargs) 2025-08-14T21:46:38.5181810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5181886Z return func(*args, **kwargs) 2025-08-14T21:46:38.5182108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5182187Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5182524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5182601Z outputs = self.layoutlm( 2025-08-14T21:46:38.5182860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5182934Z return func(*args, **kwargs) 2025-08-14T21:46:38.5183183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5183263Z return func(*args, **kwargs) 2025-08-14T21:46:38.5183494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5183574Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5183886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5183964Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5184277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5184349Z return func(*args, **kwargs) 2025-08-14T21:46:38.5184601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5184676Z return func(*args, **kwargs) 2025-08-14T21:46:38.5184907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5184973Z return func(*args, **kwargs) 2025-08-14T21:46:38.5185057Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5185268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5185347Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5185624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5185700Z layer_outputs = layer_module( 2025-08-14T21:46:38.5185920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5185998Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5186233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5186310Z return func(*args, **kwargs) 2025-08-14T21:46:38.5186553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5186630Z return func(*args, **kwargs) 2025-08-14T21:46:38.5186873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5186945Z return func(*args, **kwargs) 2025-08-14T21:46:38.5187230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5187316Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5187573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5187642Z return func(*args, **kwargs) 2025-08-14T21:46:38.5187889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5187967Z return func(*args, **kwargs) 2025-08-14T21:46:38.5188210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5188279Z return func(*args, **kwargs) 2025-08-14T21:46:38.5188565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.5188657Z self_outputs = self.self( 2025-08-14T21:46:38.5188916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5188987Z return func(*args, **kwargs) 2025-08-14T21:46:38.5189231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5189308Z return func(*args, **kwargs) 2025-08-14T21:46:38.5189553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5189623Z return func(*args, **kwargs) 2025-08-14T21:46:38.5189910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:46:38.5190062Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.5190068Z 2025-08-14T21:46:38.5190159Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.5190260Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.5190384Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5190608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5190678Z return mod(**inputs) 2025-08-14T21:46:38.5190935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5191006Z return func(*args, **kwargs) 2025-08-14T21:46:38.5191256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5191334Z return func(*args, **kwargs) 2025-08-14T21:46:38.5191561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5191655Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5191946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5192021Z outputs = self.layoutlm( 2025-08-14T21:46:38.5192280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5192351Z return func(*args, **kwargs) 2025-08-14T21:46:38.5192602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5192678Z return func(*args, **kwargs) 2025-08-14T21:46:38.5192901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5192980Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5193265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5193340Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5193599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5193669Z return func(*args, **kwargs) 2025-08-14T21:46:38.5193912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5193989Z return func(*args, **kwargs) 2025-08-14T21:46:38.5194235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5194304Z return func(*args, **kwargs) 2025-08-14T21:46:38.5194393Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5194617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5194701Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5194979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5195078Z layer_outputs = layer_module( 2025-08-14T21:46:38.5195319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5195401Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5195652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5195729Z return func(*args, **kwargs) 2025-08-14T21:46:38.5195978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5196054Z return func(*args, **kwargs) 2025-08-14T21:46:38.5196304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5196377Z return func(*args, **kwargs) 2025-08-14T21:46:38.5196682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5196800Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5197056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5197126Z return func(*args, **kwargs) 2025-08-14T21:46:38.5197370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5197449Z return func(*args, **kwargs) 2025-08-14T21:46:38.5197694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5197764Z return func(*args, **kwargs) 2025-08-14T21:46:38.5198095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:46:38.5198234Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:38.5198526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:46:38.5198616Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.5198620Z 2025-08-14T21:46:38.5198728Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5198950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5199024Z return mod(**inputs) 2025-08-14T21:46:38.5199283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5199354Z return func(*args, **kwargs) 2025-08-14T21:46:38.5199603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5199682Z return func(*args, **kwargs) 2025-08-14T21:46:38.5199911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5199986Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5200257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5200325Z outputs = self.layoutlm( 2025-08-14T21:46:38.5200566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5200633Z return func(*args, **kwargs) 2025-08-14T21:46:38.5200865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5200939Z return func(*args, **kwargs) 2025-08-14T21:46:38.5201153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5201229Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5201523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5201597Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5201837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5201903Z return func(*args, **kwargs) 2025-08-14T21:46:38.5202134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5202207Z return func(*args, **kwargs) 2025-08-14T21:46:38.5202435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5202501Z return func(*args, **kwargs) 2025-08-14T21:46:38.5202585Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5202797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5202907Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5203170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5203241Z layer_outputs = layer_module( 2025-08-14T21:46:38.5203465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5203542Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5203774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5203847Z return func(*args, **kwargs) 2025-08-14T21:46:38.5204463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5204543Z return func(*args, **kwargs) 2025-08-14T21:46:38.5204785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5204856Z return func(*args, **kwargs) 2025-08-14T21:46:38.5205130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.5205217Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.5205587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.5205682Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.5206008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.5206146Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.5206439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:46:38.5206534Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.5206547Z 2025-08-14T21:46:38.5206659Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5206880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5206955Z return mod(**inputs) 2025-08-14T21:46:38.5207194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5207261Z return func(*args, **kwargs) 2025-08-14T21:46:38.5207503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5207570Z return func(*args, **kwargs) 2025-08-14T21:46:38.5207793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5207894Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5208161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5208238Z outputs = self.layoutlm( 2025-08-14T21:46:38.5208475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5208542Z return func(*args, **kwargs) 2025-08-14T21:46:38.5208786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5208853Z return func(*args, **kwargs) 2025-08-14T21:46:38.5209073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5209146Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5209409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5209545Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5209792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5209860Z return func(*args, **kwargs) 2025-08-14T21:46:38.5210103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5210170Z return func(*args, **kwargs) 2025-08-14T21:46:38.5210410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5210475Z return func(*args, **kwargs) 2025-08-14T21:46:38.5210551Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5210772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5210861Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5211132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5211209Z layer_outputs = layer_module( 2025-08-14T21:46:38.5211427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5211512Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5211744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5211810Z return func(*args, **kwargs) 2025-08-14T21:46:38.5212051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5212117Z return func(*args, **kwargs) 2025-08-14T21:46:38.5212357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5212432Z return func(*args, **kwargs) 2025-08-14T21:46:38.5212698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.5212789Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.5213044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.5213120Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.5213424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.5213542Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.5213812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:46:38.5213925Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:38.5214154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:38.5214234Z return self.act(input) 2025-08-14T21:46:38.5214238Z 2025-08-14T21:46:38.5214338Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5214545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5214611Z return mod(**inputs) 2025-08-14T21:46:38.5214845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5214920Z return func(*args, **kwargs) 2025-08-14T21:46:38.5215154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5215219Z return func(*args, **kwargs) 2025-08-14T21:46:38.5215438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5215531Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5215817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5215888Z outputs = self.layoutlm( 2025-08-14T21:46:38.5216122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5216196Z return func(*args, **kwargs) 2025-08-14T21:46:38.5216425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5216490Z return func(*args, **kwargs) 2025-08-14T21:46:38.5216707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5216779Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5217061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5217139Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5217375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5217447Z return func(*args, **kwargs) 2025-08-14T21:46:38.5217681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5217746Z return func(*args, **kwargs) 2025-08-14T21:46:38.5217989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5218054Z return func(*args, **kwargs) 2025-08-14T21:46:38.5218137Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5218352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5218426Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5218700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5218772Z layer_outputs = layer_module( 2025-08-14T21:46:38.5218991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5219077Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5219311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5219385Z return func(*args, **kwargs) 2025-08-14T21:46:38.5219620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5219686Z return func(*args, **kwargs) 2025-08-14T21:46:38.5219926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5220013Z return func(*args, **kwargs) 2025-08-14T21:46:38.5220292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.5220378Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.5220639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.5220719Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.5221023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:46:38.5221156Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:38.5221432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:46:38.5221514Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.5221537Z 2025-08-14T21:46:38.5221663Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5221864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5221929Z return mod(**inputs) 2025-08-14T21:46:38.5222173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5222239Z return func(*args, **kwargs) 2025-08-14T21:46:38.5222483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5222552Z return func(*args, **kwargs) 2025-08-14T21:46:38.5222766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5222862Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5223124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5223197Z outputs = self.layoutlm( 2025-08-14T21:46:38.5223438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5223504Z return func(*args, **kwargs) 2025-08-14T21:46:38.5223744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5223810Z return func(*args, **kwargs) 2025-08-14T21:46:38.5224022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5224103Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5224364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5224436Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5224680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5224751Z return func(*args, **kwargs) 2025-08-14T21:46:38.5224991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5225057Z return func(*args, **kwargs) 2025-08-14T21:46:38.5225289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5225362Z return func(*args, **kwargs) 2025-08-14T21:46:38.5225437Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5225648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5225728Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5225990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5226083Z layer_outputs = layer_module( 2025-08-14T21:46:38.5226308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5226385Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5226630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5226698Z return func(*args, **kwargs) 2025-08-14T21:46:38.5226941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5227014Z return func(*args, **kwargs) 2025-08-14T21:46:38.5227256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5227332Z return func(*args, **kwargs) 2025-08-14T21:46:38.5227605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5227716Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5227966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5228033Z return func(*args, **kwargs) 2025-08-14T21:46:38.5228276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5228343Z return func(*args, **kwargs) 2025-08-14T21:46:38.5228577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5228650Z return func(*args, **kwargs) 2025-08-14T21:46:38.5228929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.5229001Z self_outputs = self.self( 2025-08-14T21:46:38.5229244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5229310Z return func(*args, **kwargs) 2025-08-14T21:46:38.5229552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5229619Z return func(*args, **kwargs) 2025-08-14T21:46:38.5229853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5229926Z return func(*args, **kwargs) 2025-08-14T21:46:38.5230190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:46:38.5230333Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.5230345Z 2025-08-14T21:46:38.5230447Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5230647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5230722Z return mod(**inputs) 2025-08-14T21:46:38.5230956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5231021Z return func(*args, **kwargs) 2025-08-14T21:46:38.5231259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5231324Z return func(*args, **kwargs) 2025-08-14T21:46:38.5231541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5231615Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5231913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5231990Z outputs = self.layoutlm( 2025-08-14T21:46:38.5232280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5232350Z return func(*args, **kwargs) 2025-08-14T21:46:38.5232615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5232685Z return func(*args, **kwargs) 2025-08-14T21:46:38.5232923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5233001Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5233290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5233374Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5233630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5233700Z return func(*args, **kwargs) 2025-08-14T21:46:38.5234007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5234077Z return func(*args, **kwargs) 2025-08-14T21:46:38.5234332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5234400Z return func(*args, **kwargs) 2025-08-14T21:46:38.5234479Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5234767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5234844Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5235131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5235229Z layer_outputs = layer_module( 2025-08-14T21:46:38.5235461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5235556Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5235801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5235876Z return func(*args, **kwargs) 2025-08-14T21:46:38.5236149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5236224Z return func(*args, **kwargs) 2025-08-14T21:46:38.5236489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5236564Z return func(*args, **kwargs) 2025-08-14T21:46:38.5236856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5236956Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5237211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5237284Z return func(*args, **kwargs) 2025-08-14T21:46:38.5237540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5237738Z return func(*args, **kwargs) 2025-08-14T21:46:38.5238006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5238078Z return func(*args, **kwargs) 2025-08-14T21:46:38.5238378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.5238462Z self_outputs = self.self( 2025-08-14T21:46:38.5238711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5238828Z return func(*args, **kwargs) 2025-08-14T21:46:38.5239090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5239162Z return func(*args, **kwargs) 2025-08-14T21:46:38.5239419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5239487Z return func(*args, **kwargs) 2025-08-14T21:46:38.5239787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:46:38.5239939Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.5239944Z 2025-08-14T21:46:38.5240051Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5240267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5240337Z return mod(**inputs) 2025-08-14T21:46:38.5240637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5240718Z return func(*args, **kwargs) 2025-08-14T21:46:38.5240971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5241042Z return func(*args, **kwargs) 2025-08-14T21:46:38.5241279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5241357Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5241658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5241730Z outputs = self.layoutlm( 2025-08-14T21:46:38.5242002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5242084Z return func(*args, **kwargs) 2025-08-14T21:46:38.5242336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5242407Z return func(*args, **kwargs) 2025-08-14T21:46:38.5242644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5242722Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5243023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5243100Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5243350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5243428Z return func(*args, **kwargs) 2025-08-14T21:46:38.5243679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5243754Z return func(*args, **kwargs) 2025-08-14T21:46:38.5244014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5244083Z return func(*args, **kwargs) 2025-08-14T21:46:38.5244174Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5244400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5244476Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5244764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5244837Z layer_outputs = layer_module( 2025-08-14T21:46:38.5245075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5245156Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5245435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5245560Z return func(*args, **kwargs) 2025-08-14T21:46:38.5245826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5245897Z return func(*args, **kwargs) 2025-08-14T21:46:38.5246160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5246231Z return func(*args, **kwargs) 2025-08-14T21:46:38.5246532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5246618Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5246864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5246966Z return func(*args, **kwargs) 2025-08-14T21:46:38.5247234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5247305Z return func(*args, **kwargs) 2025-08-14T21:46:38.5247563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5247634Z return func(*args, **kwargs) 2025-08-14T21:46:38.5247923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.5247997Z self_outputs = self.self( 2025-08-14T21:46:38.5248246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5248327Z return func(*args, **kwargs) 2025-08-14T21:46:38.5248590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5248667Z return func(*args, **kwargs) 2025-08-14T21:46:38.5248927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5248998Z return func(*args, **kwargs) 2025-08-14T21:46:38.5249284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:46:38.5249436Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.5249441Z 2025-08-14T21:46:38.5249524Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.5249614Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.5249722Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5249940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5250009Z return mod(**inputs) 2025-08-14T21:46:38.5250264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5250337Z return func(*args, **kwargs) 2025-08-14T21:46:38.5250565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5250630Z return func(*args, **kwargs) 2025-08-14T21:46:38.5250849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5250921Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5251187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5251255Z outputs = self.layoutlm( 2025-08-14T21:46:38.5251486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5251574Z return func(*args, **kwargs) 2025-08-14T21:46:38.5251809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5251874Z return func(*args, **kwargs) 2025-08-14T21:46:38.5252099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5252176Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5252461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5252539Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5252787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5252867Z return func(*args, **kwargs) 2025-08-14T21:46:38.5253118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5253207Z return func(*args, **kwargs) 2025-08-14T21:46:38.5253490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5253561Z return func(*args, **kwargs) 2025-08-14T21:46:38.5253650Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5253878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5253961Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5254230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5254299Z layer_outputs = layer_module( 2025-08-14T21:46:38.5254544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5254622Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5254855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5254927Z return func(*args, **kwargs) 2025-08-14T21:46:38.5255153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5255219Z return func(*args, **kwargs) 2025-08-14T21:46:38.5255451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5255515Z return func(*args, **kwargs) 2025-08-14T21:46:38.5255773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5255852Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5256084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5256159Z return func(*args, **kwargs) 2025-08-14T21:46:38.5256394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5256460Z return func(*args, **kwargs) 2025-08-14T21:46:38.5256697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5256762Z return func(*args, **kwargs) 2025-08-14T21:46:38.5257030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:46:38.5257157Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:38.5257419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:46:38.5257513Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.5257547Z 2025-08-14T21:46:38.5257659Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5257859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5257923Z return mod(**inputs) 2025-08-14T21:46:38.5258151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5258224Z return func(*args, **kwargs) 2025-08-14T21:46:38.5258450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5258515Z return func(*args, **kwargs) 2025-08-14T21:46:38.5258730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5258803Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5259067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5259154Z outputs = self.layoutlm( 2025-08-14T21:46:38.5259397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5259473Z return func(*args, **kwargs) 2025-08-14T21:46:38.5259698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5259763Z return func(*args, **kwargs) 2025-08-14T21:46:38.5259979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5260054Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5260324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5260411Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5260654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5260731Z return func(*args, **kwargs) 2025-08-14T21:46:38.5260957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5261021Z return func(*args, **kwargs) 2025-08-14T21:46:38.5261254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5261318Z return func(*args, **kwargs) 2025-08-14T21:46:38.5261398Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5261609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5261681Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5261951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5262023Z layer_outputs = layer_module( 2025-08-14T21:46:38.5262260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5262340Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5262588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5262662Z return func(*args, **kwargs) 2025-08-14T21:46:38.5262905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5262975Z return func(*args, **kwargs) 2025-08-14T21:46:38.5263228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5263298Z return func(*args, **kwargs) 2025-08-14T21:46:38.5263597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.5263705Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.5263988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.5264071Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.5264366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.5264484Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.5264754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:46:38.5264833Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.5264836Z 2025-08-14T21:46:38.5264944Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5265133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5265215Z return mod(**inputs) 2025-08-14T21:46:38.5265471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5265539Z return func(*args, **kwargs) 2025-08-14T21:46:38.5265780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5265847Z return func(*args, **kwargs) 2025-08-14T21:46:38.5266059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5266140Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5266405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5266489Z outputs = self.layoutlm( 2025-08-14T21:46:38.5266733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5266803Z return func(*args, **kwargs) 2025-08-14T21:46:38.5267043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5267110Z return func(*args, **kwargs) 2025-08-14T21:46:38.5267322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5267403Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5267668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5267742Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5267983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5268053Z return func(*args, **kwargs) 2025-08-14T21:46:38.5268309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5268375Z return func(*args, **kwargs) 2025-08-14T21:46:38.5268601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5268675Z return func(*args, **kwargs) 2025-08-14T21:46:38.5268749Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5268960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5269039Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5269305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5269384Z layer_outputs = layer_module( 2025-08-14T21:46:38.5269601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5269697Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5269941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5270008Z return func(*args, **kwargs) 2025-08-14T21:46:38.5270244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5270310Z return func(*args, **kwargs) 2025-08-14T21:46:38.5270541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5270614Z return func(*args, **kwargs) 2025-08-14T21:46:38.5270876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.5270960Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.5271222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.5271337Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.5271642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.5271761Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.5272026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:46:38.5272144Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:38.5272357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:38.5272436Z return self.act(input) 2025-08-14T21:46:38.5272440Z 2025-08-14T21:46:38.5272564Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5272777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5272858Z return mod(**inputs) 2025-08-14T21:46:38.5273113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5273180Z return func(*args, **kwargs) 2025-08-14T21:46:38.5273421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5273487Z return func(*args, **kwargs) 2025-08-14T21:46:38.5273707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5273783Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5274048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5274125Z outputs = self.layoutlm( 2025-08-14T21:46:38.5274362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5274440Z return func(*args, **kwargs) 2025-08-14T21:46:38.5274674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5274739Z return func(*args, **kwargs) 2025-08-14T21:46:38.5274952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5275023Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5275278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5275355Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5275585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5275667Z return func(*args, **kwargs) 2025-08-14T21:46:38.5275905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5275970Z return func(*args, **kwargs) 2025-08-14T21:46:38.5276206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5276269Z return func(*args, **kwargs) 2025-08-14T21:46:38.5276341Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5276558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5276627Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5276885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5276962Z layer_outputs = layer_module( 2025-08-14T21:46:38.5277172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5277291Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5277524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5277588Z return func(*args, **kwargs) 2025-08-14T21:46:38.5277821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5277885Z return func(*args, **kwargs) 2025-08-14T21:46:38.5278121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5278186Z return func(*args, **kwargs) 2025-08-14T21:46:38.5278457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.5278549Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.5278804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.5278880Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.5279173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:46:38.5279302Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:38.5279564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:46:38.5279642Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.5279646Z 2025-08-14T21:46:38.5279745Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5279944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5280009Z return mod(**inputs) 2025-08-14T21:46:38.5280248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5280316Z return func(*args, **kwargs) 2025-08-14T21:46:38.5280546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5280621Z return func(*args, **kwargs) 2025-08-14T21:46:38.5280832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5280907Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5281180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5281249Z outputs = self.layoutlm( 2025-08-14T21:46:38.5281493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5281579Z return func(*args, **kwargs) 2025-08-14T21:46:38.5281822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5281900Z return func(*args, **kwargs) 2025-08-14T21:46:38.5282120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5282194Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5282503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5282583Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5282845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5282915Z return func(*args, **kwargs) 2025-08-14T21:46:38.5283169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5283271Z return func(*args, **kwargs) 2025-08-14T21:46:38.5283535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5283607Z return func(*args, **kwargs) 2025-08-14T21:46:38.5283696Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5283924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5284010Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5284312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5284387Z layer_outputs = layer_module( 2025-08-14T21:46:38.5284652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5284737Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5284993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5285065Z return func(*args, **kwargs) 2025-08-14T21:46:38.5285312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5285389Z return func(*args, **kwargs) 2025-08-14T21:46:38.5285723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5285798Z return func(*args, **kwargs) 2025-08-14T21:46:38.5286091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5286181Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5286447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5286522Z return func(*args, **kwargs) 2025-08-14T21:46:38.5286779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5286860Z return func(*args, **kwargs) 2025-08-14T21:46:38.5287115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5287187Z return func(*args, **kwargs) 2025-08-14T21:46:38.5287481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.5287570Z self_outputs = self.self( 2025-08-14T21:46:38.5287811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5287879Z return func(*args, **kwargs) 2025-08-14T21:46:38.5288112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5288209Z return func(*args, **kwargs) 2025-08-14T21:46:38.5288446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5288511Z return func(*args, **kwargs) 2025-08-14T21:46:38.5288782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:46:38.5288936Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.5288941Z 2025-08-14T21:46:38.5289056Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5289265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5289333Z return mod(**inputs) 2025-08-14T21:46:38.5289596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5289667Z return func(*args, **kwargs) 2025-08-14T21:46:38.5289971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5290043Z return func(*args, **kwargs) 2025-08-14T21:46:38.5290268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5290354Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5290633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5290706Z outputs = self.layoutlm( 2025-08-14T21:46:38.5290971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5291040Z return func(*args, **kwargs) 2025-08-14T21:46:38.5291369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5291437Z return func(*args, **kwargs) 2025-08-14T21:46:38.5291655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5291736Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5292002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5292074Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5292317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5292382Z return func(*args, **kwargs) 2025-08-14T21:46:38.5292631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5292698Z return func(*args, **kwargs) 2025-08-14T21:46:38.5292933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5293012Z return func(*args, **kwargs) 2025-08-14T21:46:38.5293088Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5293315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5293392Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5293676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5293758Z layer_outputs = layer_module( 2025-08-14T21:46:38.5293990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5294071Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5294339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5294427Z return func(*args, **kwargs) 2025-08-14T21:46:38.5294687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5294758Z return func(*args, **kwargs) 2025-08-14T21:46:38.5295015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5295093Z return func(*args, **kwargs) 2025-08-14T21:46:38.5295374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5295463Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5295766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5295832Z return func(*args, **kwargs) 2025-08-14T21:46:38.5296076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5296210Z return func(*args, **kwargs) 2025-08-14T21:46:38.5296468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5296547Z return func(*args, **kwargs) 2025-08-14T21:46:38.5296826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.5296899Z self_outputs = self.self( 2025-08-14T21:46:38.5297162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5297230Z return func(*args, **kwargs) 2025-08-14T21:46:38.5297470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5297551Z return func(*args, **kwargs) 2025-08-14T21:46:38.5297786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5297866Z return func(*args, **kwargs) 2025-08-14T21:46:38.5298128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:46:38.5298280Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.5298284Z 2025-08-14T21:46:38.5298391Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5298600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5298676Z return mod(**inputs) 2025-08-14T21:46:38.5298932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5299004Z return func(*args, **kwargs) 2025-08-14T21:46:38.5299271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5299346Z return func(*args, **kwargs) 2025-08-14T21:46:38.5299578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5299658Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5299937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5300017Z outputs = self.layoutlm( 2025-08-14T21:46:38.5300261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5300330Z return func(*args, **kwargs) 2025-08-14T21:46:38.5300583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5300653Z return func(*args, **kwargs) 2025-08-14T21:46:38.5300882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5300978Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5301262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5301348Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5301599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5301676Z return func(*args, **kwargs) 2025-08-14T21:46:38.5301923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5301994Z return func(*args, **kwargs) 2025-08-14T21:46:38.5302250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5302319Z return func(*args, **kwargs) 2025-08-14T21:46:38.5302427Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5302676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5302754Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5303045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5303119Z layer_outputs = layer_module( 2025-08-14T21:46:38.5303353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5303443Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5303693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5303762Z return func(*args, **kwargs) 2025-08-14T21:46:38.5304039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5304114Z return func(*args, **kwargs) 2025-08-14T21:46:38.5304369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5304439Z return func(*args, **kwargs) 2025-08-14T21:46:38.5304718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5304810Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5305057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5305126Z return func(*args, **kwargs) 2025-08-14T21:46:38.5305383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5305455Z return func(*args, **kwargs) 2025-08-14T21:46:38.5305711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5305783Z return func(*args, **kwargs) 2025-08-14T21:46:38.5306060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.5306142Z self_outputs = self.self( 2025-08-14T21:46:38.5306392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5306468Z return func(*args, **kwargs) 2025-08-14T21:46:38.5306717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5306786Z return func(*args, **kwargs) 2025-08-14T21:46:38.5307041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5307114Z return func(*args, **kwargs) 2025-08-14T21:46:38.5307415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:46:38.5307575Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.5307579Z 2025-08-14T21:46:38.5307662Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.5307748Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.5307854Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5308063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5308138Z return mod(**inputs) 2025-08-14T21:46:38.5308387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5308455Z return func(*args, **kwargs) 2025-08-14T21:46:38.5308710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5308814Z return func(*args, **kwargs) 2025-08-14T21:46:38.5309055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5309136Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5309418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5309500Z outputs = self.layoutlm( 2025-08-14T21:46:38.5309752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5309823Z return func(*args, **kwargs) 2025-08-14T21:46:38.5310083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5310171Z return func(*args, **kwargs) 2025-08-14T21:46:38.5310410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5310493Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5310776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5310862Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5311117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5311196Z return func(*args, **kwargs) 2025-08-14T21:46:38.5311448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5311521Z return func(*args, **kwargs) 2025-08-14T21:46:38.5311785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5311854Z return func(*args, **kwargs) 2025-08-14T21:46:38.5311935Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5312169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5312248Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5312536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5312610Z layer_outputs = layer_module( 2025-08-14T21:46:38.5312840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5312933Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5313186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5313257Z return func(*args, **kwargs) 2025-08-14T21:46:38.5313527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5313617Z return func(*args, **kwargs) 2025-08-14T21:46:38.5313875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5313945Z return func(*args, **kwargs) 2025-08-14T21:46:38.5314225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5314320Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5314567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5314637Z return func(*args, **kwargs) 2025-08-14T21:46:38.5314892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5314963Z return func(*args, **kwargs) 2025-08-14T21:46:38.5315221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5315330Z return func(*args, **kwargs) 2025-08-14T21:46:38.5315609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:46:38.5315750Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:38.5316027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:46:38.5316122Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.5316127Z 2025-08-14T21:46:38.5316233Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5316447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5316540Z return mod(**inputs) 2025-08-14T21:46:38.5316789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5316862Z return func(*args, **kwargs) 2025-08-14T21:46:38.5317117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5317186Z return func(*args, **kwargs) 2025-08-14T21:46:38.5317415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5317494Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5317771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5317853Z outputs = self.layoutlm( 2025-08-14T21:46:38.5318099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5318171Z return func(*args, **kwargs) 2025-08-14T21:46:38.5318430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5318501Z return func(*args, **kwargs) 2025-08-14T21:46:38.5318730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5318808Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5319085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5319170Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5319416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5319493Z return func(*args, **kwargs) 2025-08-14T21:46:38.5319741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5319812Z return func(*args, **kwargs) 2025-08-14T21:46:38.5320088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5320159Z return func(*args, **kwargs) 2025-08-14T21:46:38.5320236Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5320467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5320542Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5320826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5320899Z layer_outputs = layer_module( 2025-08-14T21:46:38.5321127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5321218Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5321462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5321564Z return func(*args, **kwargs) 2025-08-14T21:46:38.5321817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5321888Z return func(*args, **kwargs) 2025-08-14T21:46:38.5322139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5322208Z return func(*args, **kwargs) 2025-08-14T21:46:38.5322483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.5322578Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.5322867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.5322948Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.5323274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.5323401Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.5323684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:46:38.5323770Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.5323774Z 2025-08-14T21:46:38.5323880Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5324097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5324165Z return mod(**inputs) 2025-08-14T21:46:38.5324421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5324494Z return func(*args, **kwargs) 2025-08-14T21:46:38.5324744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5324824Z return func(*args, **kwargs) 2025-08-14T21:46:38.5325049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5325127Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5325411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5325564Z outputs = self.layoutlm( 2025-08-14T21:46:38.5325832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5325903Z return func(*args, **kwargs) 2025-08-14T21:46:38.5326153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5326268Z return func(*args, **kwargs) 2025-08-14T21:46:38.5326501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5326581Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5326876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5326957Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5327225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5327298Z return func(*args, **kwargs) 2025-08-14T21:46:38.5327557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5327639Z return func(*args, **kwargs) 2025-08-14T21:46:38.5327898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5328002Z return func(*args, **kwargs) 2025-08-14T21:46:38.5328102Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5328347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5328434Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5328714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5328790Z layer_outputs = layer_module( 2025-08-14T21:46:38.5329028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5329111Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5329382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5329454Z return func(*args, **kwargs) 2025-08-14T21:46:38.5329707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5329785Z return func(*args, **kwargs) 2025-08-14T21:46:38.5330035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5330107Z return func(*args, **kwargs) 2025-08-14T21:46:38.5330397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.5330485Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.5330765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.5330849Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.5331175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.5331314Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.5331605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:46:38.5331735Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:38.5331965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:38.5332042Z return self.act(input) 2025-08-14T21:46:38.5332047Z 2025-08-14T21:46:38.5332164Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5332392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5332463Z return mod(**inputs) 2025-08-14T21:46:38.5332739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5332830Z return func(*args, **kwargs) 2025-08-14T21:46:38.5333100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5333172Z return func(*args, **kwargs) 2025-08-14T21:46:38.5333414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5333502Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5333792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5333865Z outputs = self.layoutlm( 2025-08-14T21:46:38.5334136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5334209Z return func(*args, **kwargs) 2025-08-14T21:46:38.5334480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5334569Z return func(*args, **kwargs) 2025-08-14T21:46:38.5334825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5334916Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5335218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5335303Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5335568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5335640Z return func(*args, **kwargs) 2025-08-14T21:46:38.5335907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5335978Z return func(*args, **kwargs) 2025-08-14T21:46:38.5336254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5336339Z return func(*args, **kwargs) 2025-08-14T21:46:38.5336422Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5336672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5336753Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5337055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5337138Z layer_outputs = layer_module( 2025-08-14T21:46:38.5337376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5337459Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5337871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5337953Z return func(*args, **kwargs) 2025-08-14T21:46:38.5338221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5338294Z return func(*args, **kwargs) 2025-08-14T21:46:38.5338618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5338698Z return func(*args, **kwargs) 2025-08-14T21:46:38.5338996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.5339088Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.5339378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.5339462Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.5339798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:46:38.5339991Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:38.5340288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:46:38.5340383Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.5340387Z 2025-08-14T21:46:38.5340497Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5340732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5340803Z return mod(**inputs) 2025-08-14T21:46:38.5341060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5341139Z return func(*args, **kwargs) 2025-08-14T21:46:38.5341393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5341493Z return func(*args, **kwargs) 2025-08-14T21:46:38.5341761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5341844Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5342138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5342213Z outputs = self.layoutlm( 2025-08-14T21:46:38.5342467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5342547Z return func(*args, **kwargs) 2025-08-14T21:46:38.5342803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5342899Z return func(*args, **kwargs) 2025-08-14T21:46:38.5343142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5343226Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5343519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5343600Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5343851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5343933Z return func(*args, **kwargs) 2025-08-14T21:46:38.5344179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5344252Z return func(*args, **kwargs) 2025-08-14T21:46:38.5344478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5344544Z return func(*args, **kwargs) 2025-08-14T21:46:38.5344627Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5344837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5344910Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5345172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5345242Z layer_outputs = layer_module( 2025-08-14T21:46:38.5345463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5345541Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5345774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5345848Z return func(*args, **kwargs) 2025-08-14T21:46:38.5346080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5346167Z return func(*args, **kwargs) 2025-08-14T21:46:38.5346414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5346481Z return func(*args, **kwargs) 2025-08-14T21:46:38.5346759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5346851Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5347089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5347162Z return func(*args, **kwargs) 2025-08-14T21:46:38.5347400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5347468Z return func(*args, **kwargs) 2025-08-14T21:46:38.5347711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5347809Z return func(*args, **kwargs) 2025-08-14T21:46:38.5348081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.5348152Z self_outputs = self.self( 2025-08-14T21:46:38.5348385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5348459Z return func(*args, **kwargs) 2025-08-14T21:46:38.5348692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5348767Z return func(*args, **kwargs) 2025-08-14T21:46:38.5349021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5349088Z return func(*args, **kwargs) 2025-08-14T21:46:38.5349358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:46:38.5349507Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.5349511Z 2025-08-14T21:46:38.5349613Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5349817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5349882Z return mod(**inputs) 2025-08-14T21:46:38.5350122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5350189Z return func(*args, **kwargs) 2025-08-14T21:46:38.5350421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5350496Z return func(*args, **kwargs) 2025-08-14T21:46:38.5350718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5350796Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5351060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5351127Z outputs = self.layoutlm( 2025-08-14T21:46:38.5351362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5351431Z return func(*args, **kwargs) 2025-08-14T21:46:38.5351663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5351736Z return func(*args, **kwargs) 2025-08-14T21:46:38.5351959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5352038Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5352313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5352386Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5352625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5352691Z return func(*args, **kwargs) 2025-08-14T21:46:38.5352923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5352997Z return func(*args, **kwargs) 2025-08-14T21:46:38.5353226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5353297Z return func(*args, **kwargs) 2025-08-14T21:46:38.5353371Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5353582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5353681Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5353963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5354033Z layer_outputs = layer_module( 2025-08-14T21:46:38.5354251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5354326Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5354559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5354623Z return func(*args, **kwargs) 2025-08-14T21:46:38.5354848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5354935Z return func(*args, **kwargs) 2025-08-14T21:46:38.5355164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5355231Z return func(*args, **kwargs) 2025-08-14T21:46:38.5355493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5355573Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5355807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5355871Z return func(*args, **kwargs) 2025-08-14T21:46:38.5356095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5356166Z return func(*args, **kwargs) 2025-08-14T21:46:38.5356392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5356464Z return func(*args, **kwargs) 2025-08-14T21:46:38.5356724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.5356791Z self_outputs = self.self( 2025-08-14T21:46:38.5357027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5357091Z return func(*args, **kwargs) 2025-08-14T21:46:38.5357315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5357388Z return func(*args, **kwargs) 2025-08-14T21:46:38.5357615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5357687Z return func(*args, **kwargs) 2025-08-14T21:46:38.5357943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:46:38.5358095Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.5358101Z 2025-08-14T21:46:38.5358207Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5358400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5358464Z return mod(**inputs) 2025-08-14T21:46:38.5358701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5358767Z return func(*args, **kwargs) 2025-08-14T21:46:38.5359005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5359070Z return func(*args, **kwargs) 2025-08-14T21:46:38.5359276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5359355Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5359645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5359721Z outputs = self.layoutlm( 2025-08-14T21:46:38.5359954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5360020Z return func(*args, **kwargs) 2025-08-14T21:46:38.5360256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5360321Z return func(*args, **kwargs) 2025-08-14T21:46:38.5360530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5360611Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5360885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5360966Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5361201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5361266Z return func(*args, **kwargs) 2025-08-14T21:46:38.5361501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5361564Z return func(*args, **kwargs) 2025-08-14T21:46:38.5361793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5361875Z return func(*args, **kwargs) 2025-08-14T21:46:38.5361948Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5362155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5362226Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5362476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5362555Z layer_outputs = layer_module( 2025-08-14T21:46:38.5362766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5362842Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5363077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5363144Z return func(*args, **kwargs) 2025-08-14T21:46:38.5363384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5363451Z return func(*args, **kwargs) 2025-08-14T21:46:38.5363684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5363758Z return func(*args, **kwargs) 2025-08-14T21:46:38.5364040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5364123Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5364377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5364443Z return func(*args, **kwargs) 2025-08-14T21:46:38.5364675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5364740Z return func(*args, **kwargs) 2025-08-14T21:46:38.5364971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5365045Z return func(*args, **kwargs) 2025-08-14T21:46:38.5365310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:46:38.5365405Z self_outputs = self.self( 2025-08-14T21:46:38.5365721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5365796Z return func(*args, **kwargs) 2025-08-14T21:46:38.5366039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5366108Z return func(*args, **kwargs) 2025-08-14T21:46:38.5366369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5366450Z return func(*args, **kwargs) 2025-08-14T21:46:38.5366755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:46:38.5366931Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:46:38.5366936Z 2025-08-14T21:46:38.5367022Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.5367108Z cudagraph partition due to non gpu ops 2025-08-14T21:46:38.5367224Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5367441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5367508Z return mod(**inputs) 2025-08-14T21:46:38.5367752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5367820Z return func(*args, **kwargs) 2025-08-14T21:46:38.5368062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5368130Z return func(*args, **kwargs) 2025-08-14T21:46:38.5368345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5368424Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5368678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5368753Z outputs = self.layoutlm( 2025-08-14T21:46:38.5368981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5369045Z return func(*args, **kwargs) 2025-08-14T21:46:38.5369278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5369340Z return func(*args, **kwargs) 2025-08-14T21:46:38.5369549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5369628Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5369886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5369980Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5370212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5370277Z return func(*args, **kwargs) 2025-08-14T21:46:38.5370512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5370576Z return func(*args, **kwargs) 2025-08-14T21:46:38.5370802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5370874Z return func(*args, **kwargs) 2025-08-14T21:46:38.5370947Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5371162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5371236Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5371490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5371610Z layer_outputs = layer_module( 2025-08-14T21:46:38.5371823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5371899Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5372138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5372203Z return func(*args, **kwargs) 2025-08-14T21:46:38.5372440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5372507Z return func(*args, **kwargs) 2025-08-14T21:46:38.5372758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5372835Z return func(*args, **kwargs) 2025-08-14T21:46:38.5373109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:46:38.5373192Z self_attention_outputs = self.attention( 2025-08-14T21:46:38.5373442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5373509Z return func(*args, **kwargs) 2025-08-14T21:46:38.5373754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5373821Z return func(*args, **kwargs) 2025-08-14T21:46:38.5374058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5374133Z return func(*args, **kwargs) 2025-08-14T21:46:38.5374410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:46:38.5374545Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:46:38.5374808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:46:38.5374891Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.5374894Z 2025-08-14T21:46:38.5375003Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5375204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5375271Z return mod(**inputs) 2025-08-14T21:46:38.5375518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5375586Z return func(*args, **kwargs) 2025-08-14T21:46:38.5375828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5375912Z return func(*args, **kwargs) 2025-08-14T21:46:38.5376129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5376209Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5376476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5376545Z outputs = self.layoutlm( 2025-08-14T21:46:38.5376787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5376853Z return func(*args, **kwargs) 2025-08-14T21:46:38.5377095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5377161Z return func(*args, **kwargs) 2025-08-14T21:46:38.5377376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5377461Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5377760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5377841Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5378079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5378145Z return func(*args, **kwargs) 2025-08-14T21:46:38.5378383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5378448Z return func(*args, **kwargs) 2025-08-14T21:46:38.5378686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5378762Z return func(*args, **kwargs) 2025-08-14T21:46:38.5378854Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5379076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5379151Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5379415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5379493Z layer_outputs = layer_module( 2025-08-14T21:46:38.5379709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5379786Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5380029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5380096Z return func(*args, **kwargs) 2025-08-14T21:46:38.5380336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5380405Z return func(*args, **kwargs) 2025-08-14T21:46:38.5380640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5380714Z return func(*args, **kwargs) 2025-08-14T21:46:38.5380981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.5381067Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.5381331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.5381407Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.5381707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.5381828Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.5382090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:46:38.5382202Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.5382206Z 2025-08-14T21:46:38.5382310Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5382525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5382594Z return mod(**inputs) 2025-08-14T21:46:38.5382842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5382919Z return func(*args, **kwargs) 2025-08-14T21:46:38.5383166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5383236Z return func(*args, **kwargs) 2025-08-14T21:46:38.5383469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5383568Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5383870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5383951Z outputs = self.layoutlm( 2025-08-14T21:46:38.5384186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5384262Z return func(*args, **kwargs) 2025-08-14T21:46:38.5384494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5384568Z return func(*args, **kwargs) 2025-08-14T21:46:38.5384780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5384853Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5385135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5385214Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5385446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5385520Z return func(*args, **kwargs) 2025-08-14T21:46:38.5385753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5385825Z return func(*args, **kwargs) 2025-08-14T21:46:38.5386058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5386123Z return func(*args, **kwargs) 2025-08-14T21:46:38.5386206Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5386421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5386493Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5386767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5386838Z layer_outputs = layer_module( 2025-08-14T21:46:38.5387065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5387143Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5387374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5387448Z return func(*args, **kwargs) 2025-08-14T21:46:38.5387680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5387746Z return func(*args, **kwargs) 2025-08-14T21:46:38.5387987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5388075Z return func(*args, **kwargs) 2025-08-14T21:46:38.5388346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.5388431Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.5388687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.5388773Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.5389066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:46:38.5389192Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:46:38.5389455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:46:38.5389564Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:46:38.5389819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:46:38.5389890Z return self.act(input) 2025-08-14T21:46:38.5389893Z 2025-08-14T21:46:38.5389994Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5390199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5390264Z return mod(**inputs) 2025-08-14T21:46:38.5390501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5390569Z return func(*args, **kwargs) 2025-08-14T21:46:38.5390803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5390893Z return func(*args, **kwargs) 2025-08-14T21:46:38.5391108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5391187Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5391460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 757, in forward 2025-08-14T21:46:38.5391530Z outputs = self.layoutlm( 2025-08-14T21:46:38.5391773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5391840Z return func(*args, **kwargs) 2025-08-14T21:46:38.5392071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5392147Z return func(*args, **kwargs) 2025-08-14T21:46:38.5392359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5392442Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5392707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:46:38.5392783Z encoder_outputs = self.encoder( 2025-08-14T21:46:38.5393028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5393094Z return func(*args, **kwargs) 2025-08-14T21:46:38.5393332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5393405Z return func(*args, **kwargs) 2025-08-14T21:46:38.5393640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5393712Z return func(*args, **kwargs) 2025-08-14T21:46:38.5393787Z [Previous line repeated 1 more time] 2025-08-14T21:46:38.5394002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5394098Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5394368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:46:38.5394441Z layer_outputs = layer_module( 2025-08-14T21:46:38.5394669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:46:38.5394747Z return super().__call__(*args, **kwargs) 2025-08-14T21:46:38.5394994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5395061Z return func(*args, **kwargs) 2025-08-14T21:46:38.5395298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5395374Z return func(*args, **kwargs) 2025-08-14T21:46:38.5395612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5395756Z return func(*args, **kwargs) 2025-08-14T21:46:38.5396028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:46:38.5396111Z layer_output = apply_chunking_to_forward( 2025-08-14T21:46:38.5396370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:46:38.5396446Z return forward_fn(*input_tensors) 2025-08-14T21:46:38.5396737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:46:38.5396875Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:46:38.5397160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:46:38.5397250Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.5397255Z 2025-08-14T21:46:38.5397356Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5397549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5397620Z return mod(**inputs) 2025-08-14T21:46:38.5397849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5397913Z return func(*args, **kwargs) 2025-08-14T21:46:38.5398150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5398214Z return func(*args, **kwargs) 2025-08-14T21:46:38.5398429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5398501Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5398759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 771, in forward 2025-08-14T21:46:38.5398859Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:46:38.5399119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 484, in forward 2025-08-14T21:46:38.5399236Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:46:38.5399499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 472, in forward 2025-08-14T21:46:38.5399588Z hidden_states = self.transform(hidden_states) 2025-08-14T21:46:38.5399849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 447, in forward 2025-08-14T21:46:38.5399929Z hidden_states = self.dense(hidden_states) 2025-08-14T21:46:38.5399932Z 2025-08-14T21:46:38.5400031Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5400247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5400311Z return mod(**inputs) 2025-08-14T21:46:38.5400554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5400623Z return func(*args, **kwargs) 2025-08-14T21:46:38.5400857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5400933Z return func(*args, **kwargs) 2025-08-14T21:46:38.5401148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5401230Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5401496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 771, in forward 2025-08-14T21:46:38.5401589Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:46:38.5401890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 484, in forward 2025-08-14T21:46:38.5402000Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:46:38.5402266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 473, in forward 2025-08-14T21:46:38.5402368Z hidden_states = self.decoder(hidden_states) 2025-08-14T21:46:38.5402373Z 2025-08-14T21:46:38.5402479Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:46:38.5402711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:46:38.5402781Z return mod(**inputs) 2025-08-14T21:46:38.5403057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5403141Z return func(*args, **kwargs) 2025-08-14T21:46:38.5403392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:46:38.5403462Z return func(*args, **kwargs) 2025-08-14T21:46:38.5403697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:46:38.5403774Z output = func(self, *args, **kwargs) 2025-08-14T21:46:38.5404058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 776, in forward 2025-08-14T21:46:38.5404132Z masked_lm_loss = loss_fct( 2025-08-14T21:46:38.5404136Z 2025-08-14T21:46:46.4698502Z Compilation time (from dynamo_timed): 14.887530595 2025-08-14T21:46:46.4729343Z pass 2025-08-14T21:46:46.4729905Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:46:46.4730805Z TIMING: _recursive_pre_grad_passes:0.00741 _recursive_joint_graph_passes:0.46217 _recursive_post_grad_passes:0.09212 async_compile.wait:0.59738 code_gen:7.18843 inductor_compile:8.42021 backend_compile:11.95489 gc:0.00022 entire_frame_compile:14.88753 total_wall_time:14.88753 2025-08-14T21:46:46.4731830Z STATS: call_* op count: 432 | FakeTensorMode.__torch_dispatch__:15442 | FakeTensor.__torch_dispatch__:4798 | ProxyTorchDispatchMode.__torch_dispatch__:5848 2025-08-14T21:46:46.4732307Z Dynamo produced 1 graphs covering 432 ops with 0 graph breaks (0 unique) 2025-08-14T21:46:51.4078129Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:46:51.4079064Z from pkg_resources import resource_filename 2025-08-14T21:46:52.0070217Z 2025-08-14T21:46:53.2186108Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:46:53.2186844Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:46:53.2202581Z cpu eval LayoutLMForSequenceClassification 2025-08-14T21:46:53.8925814Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:46:54.0987041Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:46:54.2870607Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:47:02.4021625Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4022201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4022537Z return mod(**inputs) 2025-08-14T21:47:02.4023098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4023452Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4024257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4024730Z outputs = self.layoutlm( 2025-08-14T21:47:02.4025081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4025438Z return func(*args, **kwargs) 2025-08-14T21:47:02.4025778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4026175Z return func(*args, **kwargs) 2025-08-14T21:47:02.4026501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4026849Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4027329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4027806Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4028169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4028514Z return func(*args, **kwargs) 2025-08-14T21:47:02.4028891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4029244Z return func(*args, **kwargs) 2025-08-14T21:47:02.4029585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4029932Z return func(*args, **kwargs) 2025-08-14T21:47:02.4030121Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4030463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4030815Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4031186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4031567Z layer_outputs = layer_module( 2025-08-14T21:47:02.4031899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4032245Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4032617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4032984Z return func(*args, **kwargs) 2025-08-14T21:47:02.4033318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4033657Z return func(*args, **kwargs) 2025-08-14T21:47:02.4033994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4034342Z return func(*args, **kwargs) 2025-08-14T21:47:02.4034762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4035172Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4035547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4035912Z return func(*args, **kwargs) 2025-08-14T21:47:02.4036254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4036623Z return func(*args, **kwargs) 2025-08-14T21:47:02.4036977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4037317Z return func(*args, **kwargs) 2025-08-14T21:47:02.4037911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4038304Z self_outputs = self.self( 2025-08-14T21:47:02.4038723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4039084Z return func(*args, **kwargs) 2025-08-14T21:47:02.4039442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4039807Z return func(*args, **kwargs) 2025-08-14T21:47:02.4040159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4040531Z return func(*args, **kwargs) 2025-08-14T21:47:02.4040924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:47:02.4041464Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4041675Z 2025-08-14T21:47:02.4041784Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4042149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4042475Z return mod(**inputs) 2025-08-14T21:47:02.4042812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4043163Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4043591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4044025Z outputs = self.layoutlm( 2025-08-14T21:47:02.4044394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4044779Z return func(*args, **kwargs) 2025-08-14T21:47:02.4045164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4045713Z return func(*args, **kwargs) 2025-08-14T21:47:02.4046077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4046463Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4046871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4047293Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4047659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4048026Z return func(*args, **kwargs) 2025-08-14T21:47:02.4048379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4048761Z return func(*args, **kwargs) 2025-08-14T21:47:02.4049181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4049578Z return func(*args, **kwargs) 2025-08-14T21:47:02.4049776Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4050117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4050469Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4050864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4051245Z layer_outputs = layer_module( 2025-08-14T21:47:02.4051582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4051931Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4052294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4052645Z return func(*args, **kwargs) 2025-08-14T21:47:02.4053041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4053399Z return func(*args, **kwargs) 2025-08-14T21:47:02.4053735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4054099Z return func(*args, **kwargs) 2025-08-14T21:47:02.4054487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4054907Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4055272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4055639Z return func(*args, **kwargs) 2025-08-14T21:47:02.4056014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4056377Z return func(*args, **kwargs) 2025-08-14T21:47:02.4056722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4057082Z return func(*args, **kwargs) 2025-08-14T21:47:02.4057464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4057866Z self_outputs = self.self( 2025-08-14T21:47:02.4058217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4058572Z return func(*args, **kwargs) 2025-08-14T21:47:02.4058915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4059262Z return func(*args, **kwargs) 2025-08-14T21:47:02.4059605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4059961Z return func(*args, **kwargs) 2025-08-14T21:47:02.4060322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:47:02.4060772Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4060965Z 2025-08-14T21:47:02.4061073Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4061431Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4061738Z return mod(**inputs) 2025-08-14T21:47:02.4062055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4062394Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4062775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4063189Z outputs = self.layoutlm( 2025-08-14T21:47:02.4063538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4063893Z return func(*args, **kwargs) 2025-08-14T21:47:02.4064231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4064587Z return func(*args, **kwargs) 2025-08-14T21:47:02.4064911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4065253Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4065632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4066025Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4066384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4066787Z return func(*args, **kwargs) 2025-08-14T21:47:02.4067145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4067510Z return func(*args, **kwargs) 2025-08-14T21:47:02.4067863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4068224Z return func(*args, **kwargs) 2025-08-14T21:47:02.4068416Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4068768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4069110Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4069524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4069924Z layer_outputs = layer_module( 2025-08-14T21:47:02.4070276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4070623Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4070996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4071366Z return func(*args, **kwargs) 2025-08-14T21:47:02.4071712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4072076Z return func(*args, **kwargs) 2025-08-14T21:47:02.4072428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4072793Z return func(*args, **kwargs) 2025-08-14T21:47:02.4073167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4073586Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4073965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4074319Z return func(*args, **kwargs) 2025-08-14T21:47:02.4074671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4075036Z return func(*args, **kwargs) 2025-08-14T21:47:02.4075384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4075741Z return func(*args, **kwargs) 2025-08-14T21:47:02.4076154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4076555Z self_outputs = self.self( 2025-08-14T21:47:02.4076907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4077308Z return func(*args, **kwargs) 2025-08-14T21:47:02.4077673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4078050Z return func(*args, **kwargs) 2025-08-14T21:47:02.4078403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4078773Z return func(*args, **kwargs) 2025-08-14T21:47:02.4079165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:47:02.4079643Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4079844Z 2025-08-14T21:47:02.4079930Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4080152Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4080604Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4080995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4081329Z return mod(**inputs) 2025-08-14T21:47:02.4081661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4082015Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4082408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4082832Z outputs = self.layoutlm( 2025-08-14T21:47:02.4083218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4083612Z return func(*args, **kwargs) 2025-08-14T21:47:02.4084011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4084405Z return func(*args, **kwargs) 2025-08-14T21:47:02.4084764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4085138Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4085667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4086112Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4086508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4086860Z return func(*args, **kwargs) 2025-08-14T21:47:02.4087210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4087573Z return func(*args, **kwargs) 2025-08-14T21:47:02.4087914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4088278Z return func(*args, **kwargs) 2025-08-14T21:47:02.4088469Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4088809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4089142Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4089593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4089992Z layer_outputs = layer_module( 2025-08-14T21:47:02.4090319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4090671Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4091037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4091465Z return func(*args, **kwargs) 2025-08-14T21:47:02.4091807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4092160Z return func(*args, **kwargs) 2025-08-14T21:47:02.4092505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4092859Z return func(*args, **kwargs) 2025-08-14T21:47:02.4093229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4093639Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4094021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4094376Z return func(*args, **kwargs) 2025-08-14T21:47:02.4094725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4095103Z return func(*args, **kwargs) 2025-08-14T21:47:02.4095473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4095834Z return func(*args, **kwargs) 2025-08-14T21:47:02.4096213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:47:02.4096666Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:47:02.4097119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:47:02.4097524Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4097669Z 2025-08-14T21:47:02.4097774Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4098154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4098497Z return mod(**inputs) 2025-08-14T21:47:02.4098841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4099192Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4099621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4100062Z outputs = self.layoutlm( 2025-08-14T21:47:02.4100426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4100794Z return func(*args, **kwargs) 2025-08-14T21:47:02.4101140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4101501Z return func(*args, **kwargs) 2025-08-14T21:47:02.4101835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4102190Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4102580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4102981Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4103349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4103703Z return func(*args, **kwargs) 2025-08-14T21:47:02.4104055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4104416Z return func(*args, **kwargs) 2025-08-14T21:47:02.4104768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4105124Z return func(*args, **kwargs) 2025-08-14T21:47:02.4105316Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4105691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4106035Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4106430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4106827Z layer_outputs = layer_module( 2025-08-14T21:47:02.4107248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4107604Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4107978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4108350Z return func(*args, **kwargs) 2025-08-14T21:47:02.4108700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4109065Z return func(*args, **kwargs) 2025-08-14T21:47:02.4109463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4109832Z return func(*args, **kwargs) 2025-08-14T21:47:02.4110217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4110653Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4111069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4111475Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4111903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4112397Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4112851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:47:02.4113258Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4113404Z 2025-08-14T21:47:02.4113510Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4113869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4114197Z return mod(**inputs) 2025-08-14T21:47:02.4114517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4114871Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4115270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4115677Z outputs = self.layoutlm( 2025-08-14T21:47:02.4116046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4116441Z return func(*args, **kwargs) 2025-08-14T21:47:02.4116816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4117210Z return func(*args, **kwargs) 2025-08-14T21:47:02.4117575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4117951Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4118371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4118786Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4119168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4119554Z return func(*args, **kwargs) 2025-08-14T21:47:02.4119921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4120336Z return func(*args, **kwargs) 2025-08-14T21:47:02.4120710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4121099Z return func(*args, **kwargs) 2025-08-14T21:47:02.4121297Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4121670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4122042Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4122462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4122904Z layer_outputs = layer_module( 2025-08-14T21:47:02.4123274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4123679Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4124085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4124473Z return func(*args, **kwargs) 2025-08-14T21:47:02.4124843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4125231Z return func(*args, **kwargs) 2025-08-14T21:47:02.4125696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4126106Z return func(*args, **kwargs) 2025-08-14T21:47:02.4126530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4127015Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4127442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4127865Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4128319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4128820Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4129299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:47:02.4129765Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:02.4130167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:47:02.4130521Z return self.act(input) 2025-08-14T21:47:02.4130646Z 2025-08-14T21:47:02.4130756Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4131139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4131481Z return mod(**inputs) 2025-08-14T21:47:02.4131829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4132202Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4132625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4133040Z outputs = self.layoutlm( 2025-08-14T21:47:02.4133411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4133797Z return func(*args, **kwargs) 2025-08-14T21:47:02.4134163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4134562Z return func(*args, **kwargs) 2025-08-14T21:47:02.4134912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4135304Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4135717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4136152Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4136528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4136915Z return func(*args, **kwargs) 2025-08-14T21:47:02.4137280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4137811Z return func(*args, **kwargs) 2025-08-14T21:47:02.4138197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4138637Z return func(*args, **kwargs) 2025-08-14T21:47:02.4138903Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4139307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4139675Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4140107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4140524Z layer_outputs = layer_module( 2025-08-14T21:47:02.4140885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4141269Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4141654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4142078Z return func(*args, **kwargs) 2025-08-14T21:47:02.4142438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4142805Z return func(*args, **kwargs) 2025-08-14T21:47:02.4143164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4143555Z return func(*args, **kwargs) 2025-08-14T21:47:02.4143939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4144328Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4144713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4145093Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4145499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:47:02.4145977Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:47:02.4146440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:47:02.4146835Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4146970Z 2025-08-14T21:47:02.4147078Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4147429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4147772Z return mod(**inputs) 2025-08-14T21:47:02.4148106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4148475Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4148900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4149310Z outputs = self.layoutlm( 2025-08-14T21:47:02.4149686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4150046Z return func(*args, **kwargs) 2025-08-14T21:47:02.4150391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4150746Z return func(*args, **kwargs) 2025-08-14T21:47:02.4151061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4151393Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4151769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4152145Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4152490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4152837Z return func(*args, **kwargs) 2025-08-14T21:47:02.4153203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4153553Z return func(*args, **kwargs) 2025-08-14T21:47:02.4153886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4154232Z return func(*args, **kwargs) 2025-08-14T21:47:02.4154411Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4154744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4155077Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4155445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4155841Z layer_outputs = layer_module( 2025-08-14T21:47:02.4156172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4156520Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4156876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4157232Z return func(*args, **kwargs) 2025-08-14T21:47:02.4157577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4157937Z return func(*args, **kwargs) 2025-08-14T21:47:02.4158274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4158634Z return func(*args, **kwargs) 2025-08-14T21:47:02.4159002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4159391Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4159761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4160115Z return func(*args, **kwargs) 2025-08-14T21:47:02.4160456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4160808Z return func(*args, **kwargs) 2025-08-14T21:47:02.4161151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4161509Z return func(*args, **kwargs) 2025-08-14T21:47:02.4161878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4162280Z self_outputs = self.self( 2025-08-14T21:47:02.4162648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4163022Z return func(*args, **kwargs) 2025-08-14T21:47:02.4163359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4163709Z return func(*args, **kwargs) 2025-08-14T21:47:02.4164049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4164397Z return func(*args, **kwargs) 2025-08-14T21:47:02.4164768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:47:02.4165308Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4165606Z 2025-08-14T21:47:02.4165734Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4166119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4166465Z return mod(**inputs) 2025-08-14T21:47:02.4166902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4167266Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4167646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4168030Z outputs = self.layoutlm( 2025-08-14T21:47:02.4168373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4168721Z return func(*args, **kwargs) 2025-08-14T21:47:02.4169066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4169416Z return func(*args, **kwargs) 2025-08-14T21:47:02.4169751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4170093Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4170487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4170886Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4171236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4171592Z return func(*args, **kwargs) 2025-08-14T21:47:02.4171935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4172297Z return func(*args, **kwargs) 2025-08-14T21:47:02.4172632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4172985Z return func(*args, **kwargs) 2025-08-14T21:47:02.4173177Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4173509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4173853Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4174247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4174637Z layer_outputs = layer_module( 2025-08-14T21:47:02.4174969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4175324Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4175689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4176044Z return func(*args, **kwargs) 2025-08-14T21:47:02.4176387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4176744Z return func(*args, **kwargs) 2025-08-14T21:47:02.4177977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4178332Z return func(*args, **kwargs) 2025-08-14T21:47:02.4178723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4179159Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4179520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4179858Z return func(*args, **kwargs) 2025-08-14T21:47:02.4180192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4180541Z return func(*args, **kwargs) 2025-08-14T21:47:02.4180870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4181239Z return func(*args, **kwargs) 2025-08-14T21:47:02.4181627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4182019Z self_outputs = self.self( 2025-08-14T21:47:02.4182370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4182719Z return func(*args, **kwargs) 2025-08-14T21:47:02.4183054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4183394Z return func(*args, **kwargs) 2025-08-14T21:47:02.4183728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4184073Z return func(*args, **kwargs) 2025-08-14T21:47:02.4184449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:47:02.4184899Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4185095Z 2025-08-14T21:47:02.4185198Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4185554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4185867Z return mod(**inputs) 2025-08-14T21:47:02.4186177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4186517Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4186903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4187282Z outputs = self.layoutlm( 2025-08-14T21:47:02.4187632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4187996Z return func(*args, **kwargs) 2025-08-14T21:47:02.4188350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4188706Z return func(*args, **kwargs) 2025-08-14T21:47:02.4189041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4189395Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4189784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4190182Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4190547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4190909Z return func(*args, **kwargs) 2025-08-14T21:47:02.4191252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4191631Z return func(*args, **kwargs) 2025-08-14T21:47:02.4191986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4192323Z return func(*args, **kwargs) 2025-08-14T21:47:02.4192510Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4192845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4193178Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4193548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4193929Z layer_outputs = layer_module( 2025-08-14T21:47:02.4194258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4194592Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4194987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4195333Z return func(*args, **kwargs) 2025-08-14T21:47:02.4195662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4195996Z return func(*args, **kwargs) 2025-08-14T21:47:02.4196326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4196668Z return func(*args, **kwargs) 2025-08-14T21:47:02.4197033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4197422Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4197797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4198158Z return func(*args, **kwargs) 2025-08-14T21:47:02.4198500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4198850Z return func(*args, **kwargs) 2025-08-14T21:47:02.4199190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4199540Z return func(*args, **kwargs) 2025-08-14T21:47:02.4199897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4200280Z self_outputs = self.self( 2025-08-14T21:47:02.4200631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4200985Z return func(*args, **kwargs) 2025-08-14T21:47:02.4201330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4201688Z return func(*args, **kwargs) 2025-08-14T21:47:02.4202032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4202392Z return func(*args, **kwargs) 2025-08-14T21:47:02.4202765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:47:02.4203230Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4203423Z 2025-08-14T21:47:02.4203508Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4203710Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4203936Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4204287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4204616Z return mod(**inputs) 2025-08-14T21:47:02.4204941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4205286Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4205748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4206152Z outputs = self.layoutlm( 2025-08-14T21:47:02.4206538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4206931Z return func(*args, **kwargs) 2025-08-14T21:47:02.4207304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4207691Z return func(*args, **kwargs) 2025-08-14T21:47:02.4208026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4208376Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4208797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4209282Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4209637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4209986Z return func(*args, **kwargs) 2025-08-14T21:47:02.4210328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4210682Z return func(*args, **kwargs) 2025-08-14T21:47:02.4211023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4211369Z return func(*args, **kwargs) 2025-08-14T21:47:02.4211577Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4211924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4212259Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4212649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4213039Z layer_outputs = layer_module( 2025-08-14T21:47:02.4213377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4213723Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4214090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4214455Z return func(*args, **kwargs) 2025-08-14T21:47:02.4214796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4215153Z return func(*args, **kwargs) 2025-08-14T21:47:02.4215499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4215859Z return func(*args, **kwargs) 2025-08-14T21:47:02.4216228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4216632Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4217012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4217353Z return func(*args, **kwargs) 2025-08-14T21:47:02.4217694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4218046Z return func(*args, **kwargs) 2025-08-14T21:47:02.4218382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4218746Z return func(*args, **kwargs) 2025-08-14T21:47:02.4219110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:47:02.4219539Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:47:02.4219964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:47:02.4220342Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4220482Z 2025-08-14T21:47:02.4220580Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4220923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4221221Z return mod(**inputs) 2025-08-14T21:47:02.4221533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4221868Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4222290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4222666Z outputs = self.layoutlm( 2025-08-14T21:47:02.4223014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4223370Z return func(*args, **kwargs) 2025-08-14T21:47:02.4223705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4224056Z return func(*args, **kwargs) 2025-08-14T21:47:02.4224380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4224722Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4225111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4225499Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4225844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4226186Z return func(*args, **kwargs) 2025-08-14T21:47:02.4226511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4226859Z return func(*args, **kwargs) 2025-08-14T21:47:02.4227188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4227525Z return func(*args, **kwargs) 2025-08-14T21:47:02.4227708Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4228043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4228377Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4228750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4229127Z layer_outputs = layer_module( 2025-08-14T21:47:02.4229462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4229796Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4230148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4230491Z return func(*args, **kwargs) 2025-08-14T21:47:02.4230825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4231165Z return func(*args, **kwargs) 2025-08-14T21:47:02.4231501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4231869Z return func(*args, **kwargs) 2025-08-14T21:47:02.4232231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4232628Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4233018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4233394Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4233804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4234273Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4234708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:47:02.4235111Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4235253Z 2025-08-14T21:47:02.4235385Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4235747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4236067Z return mod(**inputs) 2025-08-14T21:47:02.4236379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4236726Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4237113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4237498Z outputs = self.layoutlm( 2025-08-14T21:47:02.4238015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4238382Z return func(*args, **kwargs) 2025-08-14T21:47:02.4238776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4239132Z return func(*args, **kwargs) 2025-08-14T21:47:02.4239457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4239803Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4240194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4240586Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4240944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4241302Z return func(*args, **kwargs) 2025-08-14T21:47:02.4241640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4242008Z return func(*args, **kwargs) 2025-08-14T21:47:02.4242362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4242725Z return func(*args, **kwargs) 2025-08-14T21:47:02.4242912Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4243261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4243614Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4244006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4244406Z layer_outputs = layer_module( 2025-08-14T21:47:02.4244757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4245140Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4245599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4246069Z return func(*args, **kwargs) 2025-08-14T21:47:02.4246454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4246834Z return func(*args, **kwargs) 2025-08-14T21:47:02.4247176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4247535Z return func(*args, **kwargs) 2025-08-14T21:47:02.4247911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4248310Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4248705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4249097Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4249519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4250043Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4250477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:47:02.4250903Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:02.4251270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:47:02.4251590Z return self.act(input) 2025-08-14T21:47:02.4251706Z 2025-08-14T21:47:02.4251808Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4252152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4252449Z return mod(**inputs) 2025-08-14T21:47:02.4252778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4253111Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4253478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4253841Z outputs = self.layoutlm( 2025-08-14T21:47:02.4254171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4254510Z return func(*args, **kwargs) 2025-08-14T21:47:02.4254833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4255180Z return func(*args, **kwargs) 2025-08-14T21:47:02.4255497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4255832Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4256209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4256586Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4256943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4257293Z return func(*args, **kwargs) 2025-08-14T21:47:02.4257641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4257995Z return func(*args, **kwargs) 2025-08-14T21:47:02.4258346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4258675Z return func(*args, **kwargs) 2025-08-14T21:47:02.4258856Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4259184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4259524Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4259892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4260261Z layer_outputs = layer_module( 2025-08-14T21:47:02.4260575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4260899Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4261243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4261580Z return func(*args, **kwargs) 2025-08-14T21:47:02.4261905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4262238Z return func(*args, **kwargs) 2025-08-14T21:47:02.4262565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4262942Z return func(*args, **kwargs) 2025-08-14T21:47:02.4263284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4263662Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4264031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4264393Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4264779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:47:02.4265228Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:47:02.4265655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:47:02.4266040Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4266170Z 2025-08-14T21:47:02.4266269Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4266603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4266911Z return mod(**inputs) 2025-08-14T21:47:02.4267212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4267546Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4267922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4268308Z outputs = self.layoutlm( 2025-08-14T21:47:02.4268636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4268994Z return func(*args, **kwargs) 2025-08-14T21:47:02.4269334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4269678Z return func(*args, **kwargs) 2025-08-14T21:47:02.4269992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4270322Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4270696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4271071Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4271419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4271766Z return func(*args, **kwargs) 2025-08-14T21:47:02.4272094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4272459Z return func(*args, **kwargs) 2025-08-14T21:47:02.4272799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4273146Z return func(*args, **kwargs) 2025-08-14T21:47:02.4273324Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4273654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4273988Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4274355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4274739Z layer_outputs = layer_module( 2025-08-14T21:47:02.4275068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4275413Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4275763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4276151Z return func(*args, **kwargs) 2025-08-14T21:47:02.4276485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4276829Z return func(*args, **kwargs) 2025-08-14T21:47:02.4277157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4277501Z return func(*args, **kwargs) 2025-08-14T21:47:02.4277858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4278239Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4278610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4278960Z return func(*args, **kwargs) 2025-08-14T21:47:02.4279301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4279656Z return func(*args, **kwargs) 2025-08-14T21:47:02.4280000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4280357Z return func(*args, **kwargs) 2025-08-14T21:47:02.4280717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4281112Z self_outputs = self.self( 2025-08-14T21:47:02.4281454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4281803Z return func(*args, **kwargs) 2025-08-14T21:47:02.4282132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4282479Z return func(*args, **kwargs) 2025-08-14T21:47:02.4282821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4283169Z return func(*args, **kwargs) 2025-08-14T21:47:02.4283540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:47:02.4284009Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4284206Z 2025-08-14T21:47:02.4284318Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4284671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4284994Z return mod(**inputs) 2025-08-14T21:47:02.4285318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4285791Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4286280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4286728Z outputs = self.layoutlm( 2025-08-14T21:47:02.4287120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4287494Z return func(*args, **kwargs) 2025-08-14T21:47:02.4287858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4288231Z return func(*args, **kwargs) 2025-08-14T21:47:02.4288574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4288932Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4289356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4289773Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4290180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4290554Z return func(*args, **kwargs) 2025-08-14T21:47:02.4290911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4291281Z return func(*args, **kwargs) 2025-08-14T21:47:02.4291629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4291998Z return func(*args, **kwargs) 2025-08-14T21:47:02.4292194Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4292536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4292910Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4293313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4293717Z layer_outputs = layer_module( 2025-08-14T21:47:02.4294053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4294411Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4294786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4295148Z return func(*args, **kwargs) 2025-08-14T21:47:02.4295477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4295815Z return func(*args, **kwargs) 2025-08-14T21:47:02.4296143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4296472Z return func(*args, **kwargs) 2025-08-14T21:47:02.4296833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4297213Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4297570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4297909Z return func(*args, **kwargs) 2025-08-14T21:47:02.4298256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4298610Z return func(*args, **kwargs) 2025-08-14T21:47:02.4298942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4299294Z return func(*args, **kwargs) 2025-08-14T21:47:02.4299663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4300068Z self_outputs = self.self( 2025-08-14T21:47:02.4300409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4300763Z return func(*args, **kwargs) 2025-08-14T21:47:02.4301110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4301469Z return func(*args, **kwargs) 2025-08-14T21:47:02.4301830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4302184Z return func(*args, **kwargs) 2025-08-14T21:47:02.4302551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:47:02.4302990Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4303185Z 2025-08-14T21:47:02.4303293Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4303676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4303989Z return mod(**inputs) 2025-08-14T21:47:02.4304291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4304644Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4305009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4305377Z outputs = self.layoutlm( 2025-08-14T21:47:02.4305716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4306061Z return func(*args, **kwargs) 2025-08-14T21:47:02.4306427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4306761Z return func(*args, **kwargs) 2025-08-14T21:47:02.4307075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4307409Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4307789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4308179Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4308523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4308866Z return func(*args, **kwargs) 2025-08-14T21:47:02.4309206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4309537Z return func(*args, **kwargs) 2025-08-14T21:47:02.4309863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4310201Z return func(*args, **kwargs) 2025-08-14T21:47:02.4310386Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4310710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4311047Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4311403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4311771Z layer_outputs = layer_module( 2025-08-14T21:47:02.4312088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4312414Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4312765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4313104Z return func(*args, **kwargs) 2025-08-14T21:47:02.4313457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4313784Z return func(*args, **kwargs) 2025-08-14T21:47:02.4314107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4314441Z return func(*args, **kwargs) 2025-08-14T21:47:02.4314781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4315164Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4315508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4315842Z return func(*args, **kwargs) 2025-08-14T21:47:02.4316158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4316522Z return func(*args, **kwargs) 2025-08-14T21:47:02.4316872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4317223Z return func(*args, **kwargs) 2025-08-14T21:47:02.4317576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4317950Z self_outputs = self.self( 2025-08-14T21:47:02.4318288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4318630Z return func(*args, **kwargs) 2025-08-14T21:47:02.4318968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4319312Z return func(*args, **kwargs) 2025-08-14T21:47:02.4319694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4320039Z return func(*args, **kwargs) 2025-08-14T21:47:02.4320405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:47:02.4320854Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4321040Z 2025-08-14T21:47:02.4321117Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4321320Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4321544Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4321892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4322195Z return mod(**inputs) 2025-08-14T21:47:02.4322505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4322837Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4323210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4323592Z outputs = self.layoutlm( 2025-08-14T21:47:02.4323928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4324279Z return func(*args, **kwargs) 2025-08-14T21:47:02.4324613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4324954Z return func(*args, **kwargs) 2025-08-14T21:47:02.4325276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4325708Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4326116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4326556Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4326915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4327267Z return func(*args, **kwargs) 2025-08-14T21:47:02.4327612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4327967Z return func(*args, **kwargs) 2025-08-14T21:47:02.4328312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4328662Z return func(*args, **kwargs) 2025-08-14T21:47:02.4328851Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4329190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4329528Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4329934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4330345Z layer_outputs = layer_module( 2025-08-14T21:47:02.4330682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4331029Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4331403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4331755Z return func(*args, **kwargs) 2025-08-14T21:47:02.4332095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4332452Z return func(*args, **kwargs) 2025-08-14T21:47:02.4332826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4333184Z return func(*args, **kwargs) 2025-08-14T21:47:02.4333559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4333957Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4334330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4334679Z return func(*args, **kwargs) 2025-08-14T21:47:02.4335036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4335394Z return func(*args, **kwargs) 2025-08-14T21:47:02.4335761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4336149Z return func(*args, **kwargs) 2025-08-14T21:47:02.4336553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:47:02.4337041Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:47:02.4337516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:47:02.4338116Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4338288Z 2025-08-14T21:47:02.4338399Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4338790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4339142Z return mod(**inputs) 2025-08-14T21:47:02.4339491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4339870Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4340266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4340704Z outputs = self.layoutlm( 2025-08-14T21:47:02.4341059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4341423Z return func(*args, **kwargs) 2025-08-14T21:47:02.4341761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4342119Z return func(*args, **kwargs) 2025-08-14T21:47:02.4342445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4342787Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4343166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4343728Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4344106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4361683Z return func(*args, **kwargs) 2025-08-14T21:47:02.4362261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4362657Z return func(*args, **kwargs) 2025-08-14T21:47:02.4363032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4363397Z return func(*args, **kwargs) 2025-08-14T21:47:02.4363605Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4363972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4364345Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4364785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4365190Z layer_outputs = layer_module( 2025-08-14T21:47:02.4365643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4366012Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4366435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4366841Z return func(*args, **kwargs) 2025-08-14T21:47:02.4367221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4367611Z return func(*args, **kwargs) 2025-08-14T21:47:02.4367971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4368339Z return func(*args, **kwargs) 2025-08-14T21:47:02.4368721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4369145Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4369565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4369949Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4370358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4370823Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4371259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:47:02.4371660Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4371799Z 2025-08-14T21:47:02.4371905Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4372263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4372621Z return mod(**inputs) 2025-08-14T21:47:02.4372945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4373304Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4373696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4374081Z outputs = self.layoutlm( 2025-08-14T21:47:02.4374431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4374795Z return func(*args, **kwargs) 2025-08-14T21:47:02.4375135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4375499Z return func(*args, **kwargs) 2025-08-14T21:47:02.4375827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4376197Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4376596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4376989Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4377348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4377696Z return func(*args, **kwargs) 2025-08-14T21:47:02.4378042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4378396Z return func(*args, **kwargs) 2025-08-14T21:47:02.4378739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4379107Z return func(*args, **kwargs) 2025-08-14T21:47:02.4379309Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4379644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4379971Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4380344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4380718Z layer_outputs = layer_module( 2025-08-14T21:47:02.4381047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4381386Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4381742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4382100Z return func(*args, **kwargs) 2025-08-14T21:47:02.4382440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4382797Z return func(*args, **kwargs) 2025-08-14T21:47:02.4383147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4383509Z return func(*args, **kwargs) 2025-08-14T21:47:02.4383863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4384255Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4384638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4385012Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4385408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4385861Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4386300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:47:02.4386718Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:02.4387085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:47:02.4387416Z return self.act(input) 2025-08-14T21:47:02.4387525Z 2025-08-14T21:47:02.4387636Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4387982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4388308Z return mod(**inputs) 2025-08-14T21:47:02.4388619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4388950Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4389321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4389737Z outputs = self.layoutlm( 2025-08-14T21:47:02.4390075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4390419Z return func(*args, **kwargs) 2025-08-14T21:47:02.4390755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4391103Z return func(*args, **kwargs) 2025-08-14T21:47:02.4391415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4391741Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4392115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4392510Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4392853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4393209Z return func(*args, **kwargs) 2025-08-14T21:47:02.4393542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4393888Z return func(*args, **kwargs) 2025-08-14T21:47:02.4394216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4394559Z return func(*args, **kwargs) 2025-08-14T21:47:02.4394741Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4395063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4395391Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4395767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4396146Z layer_outputs = layer_module( 2025-08-14T21:47:02.4396470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4396817Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4397179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4397526Z return func(*args, **kwargs) 2025-08-14T21:47:02.4397868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4398218Z return func(*args, **kwargs) 2025-08-14T21:47:02.4398564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4398900Z return func(*args, **kwargs) 2025-08-14T21:47:02.4399271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4399694Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4400083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4400459Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4400870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:47:02.4401347Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:47:02.4401783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:47:02.4402183Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4402327Z 2025-08-14T21:47:02.4402434Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4402792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4403154Z return mod(**inputs) 2025-08-14T21:47:02.4403480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4403825Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4404205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4404593Z outputs = self.layoutlm( 2025-08-14T21:47:02.4404939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4405297Z return func(*args, **kwargs) 2025-08-14T21:47:02.4405761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4406197Z return func(*args, **kwargs) 2025-08-14T21:47:02.4406558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4406927Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4407317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4407719Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4408097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4408446Z return func(*args, **kwargs) 2025-08-14T21:47:02.4408794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4409151Z return func(*args, **kwargs) 2025-08-14T21:47:02.4409494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4409840Z return func(*args, **kwargs) 2025-08-14T21:47:02.4410033Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4410376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4410708Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4411092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4411478Z layer_outputs = layer_module( 2025-08-14T21:47:02.4411811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4412152Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4412514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4412869Z return func(*args, **kwargs) 2025-08-14T21:47:02.4413212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4413578Z return func(*args, **kwargs) 2025-08-14T21:47:02.4413913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4414262Z return func(*args, **kwargs) 2025-08-14T21:47:02.4414616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4415008Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4415370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4415798Z return func(*args, **kwargs) 2025-08-14T21:47:02.4416118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4416454Z return func(*args, **kwargs) 2025-08-14T21:47:02.4416800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4417142Z return func(*args, **kwargs) 2025-08-14T21:47:02.4417502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4417878Z self_outputs = self.self( 2025-08-14T21:47:02.4418215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4418556Z return func(*args, **kwargs) 2025-08-14T21:47:02.4418891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4419234Z return func(*args, **kwargs) 2025-08-14T21:47:02.4419570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4419922Z return func(*args, **kwargs) 2025-08-14T21:47:02.4420291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:47:02.4420741Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4420931Z 2025-08-14T21:47:02.4421032Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4421375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4421686Z return mod(**inputs) 2025-08-14T21:47:02.4421988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4422324Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4422705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4423084Z outputs = self.layoutlm( 2025-08-14T21:47:02.4423422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4423770Z return func(*args, **kwargs) 2025-08-14T21:47:02.4424111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4424460Z return func(*args, **kwargs) 2025-08-14T21:47:02.4424773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4425118Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4425512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4425885Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4426240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4426606Z return func(*args, **kwargs) 2025-08-14T21:47:02.4426952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4427302Z return func(*args, **kwargs) 2025-08-14T21:47:02.4427644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4427995Z return func(*args, **kwargs) 2025-08-14T21:47:02.4428176Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4428526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4428859Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4429236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4429609Z layer_outputs = layer_module( 2025-08-14T21:47:02.4429938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4430311Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4430662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4431020Z return func(*args, **kwargs) 2025-08-14T21:47:02.4431364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4431724Z return func(*args, **kwargs) 2025-08-14T21:47:02.4432059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4432413Z return func(*args, **kwargs) 2025-08-14T21:47:02.4432804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4433201Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4433573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4433930Z return func(*args, **kwargs) 2025-08-14T21:47:02.4434271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4434623Z return func(*args, **kwargs) 2025-08-14T21:47:02.4434967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4435322Z return func(*args, **kwargs) 2025-08-14T21:47:02.4435693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4436070Z self_outputs = self.self( 2025-08-14T21:47:02.4436422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4436777Z return func(*args, **kwargs) 2025-08-14T21:47:02.4437117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4437472Z return func(*args, **kwargs) 2025-08-14T21:47:02.4438025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4438422Z return func(*args, **kwargs) 2025-08-14T21:47:02.4438843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:47:02.4439334Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4439538Z 2025-08-14T21:47:02.4439659Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4440047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4440378Z return mod(**inputs) 2025-08-14T21:47:02.4440745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4441146Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4441525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4441918Z outputs = self.layoutlm( 2025-08-14T21:47:02.4442272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4442638Z return func(*args, **kwargs) 2025-08-14T21:47:02.4442980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4443336Z return func(*args, **kwargs) 2025-08-14T21:47:02.4443663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4444005Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4444443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4444838Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4445197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4445608Z return func(*args, **kwargs) 2025-08-14T21:47:02.4445961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4446390Z return func(*args, **kwargs) 2025-08-14T21:47:02.4446729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4447139Z return func(*args, **kwargs) 2025-08-14T21:47:02.4447363Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4447712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4448055Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4448447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4448838Z layer_outputs = layer_module( 2025-08-14T21:47:02.4449169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4449524Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4449889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4450248Z return func(*args, **kwargs) 2025-08-14T21:47:02.4450577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4450927Z return func(*args, **kwargs) 2025-08-14T21:47:02.4451268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4451616Z return func(*args, **kwargs) 2025-08-14T21:47:02.4451972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4452364Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4452723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4453062Z return func(*args, **kwargs) 2025-08-14T21:47:02.4453400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4453743Z return func(*args, **kwargs) 2025-08-14T21:47:02.4454082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4454438Z return func(*args, **kwargs) 2025-08-14T21:47:02.4454805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4455179Z self_outputs = self.self( 2025-08-14T21:47:02.4455511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4455855Z return func(*args, **kwargs) 2025-08-14T21:47:02.4456186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4456531Z return func(*args, **kwargs) 2025-08-14T21:47:02.4456853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4457200Z return func(*args, **kwargs) 2025-08-14T21:47:02.4457565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:47:02.4458081Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4458285Z 2025-08-14T21:47:02.4458365Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4458571Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4458798Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4459145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4459474Z return mod(**inputs) 2025-08-14T21:47:02.4459789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4460123Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4460526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4460904Z outputs = self.layoutlm( 2025-08-14T21:47:02.4461244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4461583Z return func(*args, **kwargs) 2025-08-14T21:47:02.4461919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4462264Z return func(*args, **kwargs) 2025-08-14T21:47:02.4462578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4462902Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4463278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4463655Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4464003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4464340Z return func(*args, **kwargs) 2025-08-14T21:47:02.4464668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4464999Z return func(*args, **kwargs) 2025-08-14T21:47:02.4465314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4465652Z return func(*args, **kwargs) 2025-08-14T21:47:02.4465829Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4466143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4466464Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4466827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4467197Z layer_outputs = layer_module( 2025-08-14T21:47:02.4467519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4467874Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4468223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4468552Z return func(*args, **kwargs) 2025-08-14T21:47:02.4468882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4469218Z return func(*args, **kwargs) 2025-08-14T21:47:02.4469544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4469873Z return func(*args, **kwargs) 2025-08-14T21:47:02.4470233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4470623Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4471012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4471358Z return func(*args, **kwargs) 2025-08-14T21:47:02.4471691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4472034Z return func(*args, **kwargs) 2025-08-14T21:47:02.4472365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4472699Z return func(*args, **kwargs) 2025-08-14T21:47:02.4473049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:47:02.4473463Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:47:02.4473891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:47:02.4474275Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4474402Z 2025-08-14T21:47:02.4474503Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4474825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4475124Z return mod(**inputs) 2025-08-14T21:47:02.4475423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4475750Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4476109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4476474Z outputs = self.layoutlm( 2025-08-14T21:47:02.4476810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4477149Z return func(*args, **kwargs) 2025-08-14T21:47:02.4477486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4477829Z return func(*args, **kwargs) 2025-08-14T21:47:02.4478140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4478474Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4478839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4479211Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4479551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4479890Z return func(*args, **kwargs) 2025-08-14T21:47:02.4480226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4480601Z return func(*args, **kwargs) 2025-08-14T21:47:02.4480935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4481288Z return func(*args, **kwargs) 2025-08-14T21:47:02.4481480Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4481816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4482147Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4482533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4482918Z layer_outputs = layer_module( 2025-08-14T21:47:02.4483244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4483592Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4483979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4484343Z return func(*args, **kwargs) 2025-08-14T21:47:02.4484671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4485013Z return func(*args, **kwargs) 2025-08-14T21:47:02.4485348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4485790Z return func(*args, **kwargs) 2025-08-14T21:47:02.4486180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4486593Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4487026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4487414Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4487838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4488324Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4488753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:47:02.4489142Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4489283Z 2025-08-14T21:47:02.4489381Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4489723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4490027Z return mod(**inputs) 2025-08-14T21:47:02.4490339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4490679Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4491066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4491445Z outputs = self.layoutlm( 2025-08-14T21:47:02.4491787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4492143Z return func(*args, **kwargs) 2025-08-14T21:47:02.4492475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4492831Z return func(*args, **kwargs) 2025-08-14T21:47:02.4493150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4493485Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4493861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4494260Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4494607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4494958Z return func(*args, **kwargs) 2025-08-14T21:47:02.4495292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4495627Z return func(*args, **kwargs) 2025-08-14T21:47:02.4495959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4496304Z return func(*args, **kwargs) 2025-08-14T21:47:02.4496483Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4496813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4497140Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4497530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4497924Z layer_outputs = layer_module( 2025-08-14T21:47:02.4498250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4498589Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4498934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4499279Z return func(*args, **kwargs) 2025-08-14T21:47:02.4499611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4499958Z return func(*args, **kwargs) 2025-08-14T21:47:02.4500300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4500647Z return func(*args, **kwargs) 2025-08-14T21:47:02.4501013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4501399Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4501770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4502139Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4502539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4502986Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4503396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:47:02.4503805Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:02.4504163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:47:02.4504476Z return self.act(input) 2025-08-14T21:47:02.4504588Z 2025-08-14T21:47:02.4504687Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4505031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4505341Z return mod(**inputs) 2025-08-14T21:47:02.4505641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4505981Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4506347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4506707Z outputs = self.layoutlm( 2025-08-14T21:47:02.4507045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4507435Z return func(*args, **kwargs) 2025-08-14T21:47:02.4507776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4508118Z return func(*args, **kwargs) 2025-08-14T21:47:02.4508435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4508772Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4509135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4509509Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4509849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4510186Z return func(*args, **kwargs) 2025-08-14T21:47:02.4510506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4510879Z return func(*args, **kwargs) 2025-08-14T21:47:02.4511211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4511545Z return func(*args, **kwargs) 2025-08-14T21:47:02.4511726Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4512051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4512372Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4512733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4513102Z layer_outputs = layer_module( 2025-08-14T21:47:02.4513438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4513767Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4514117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4514456Z return func(*args, **kwargs) 2025-08-14T21:47:02.4514783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4515123Z return func(*args, **kwargs) 2025-08-14T21:47:02.4515463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4515816Z return func(*args, **kwargs) 2025-08-14T21:47:02.4516173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4516568Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4516952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4517332Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4517734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:47:02.4518199Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:47:02.4518629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:47:02.4519024Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4519155Z 2025-08-14T21:47:02.4519253Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4519595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4519905Z return mod(**inputs) 2025-08-14T21:47:02.4520207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4520560Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4520937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4521317Z outputs = self.layoutlm( 2025-08-14T21:47:02.4521649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4522001Z return func(*args, **kwargs) 2025-08-14T21:47:02.4522340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4522688Z return func(*args, **kwargs) 2025-08-14T21:47:02.4522998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4523333Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4523725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4524120Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4524478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4524832Z return func(*args, **kwargs) 2025-08-14T21:47:02.4525175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4525588Z return func(*args, **kwargs) 2025-08-14T21:47:02.4525933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4526307Z return func(*args, **kwargs) 2025-08-14T21:47:02.4526498Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4526883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4527268Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4527693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4528109Z layer_outputs = layer_module( 2025-08-14T21:47:02.4528475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4528856Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4529252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4529636Z return func(*args, **kwargs) 2025-08-14T21:47:02.4529979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4530330Z return func(*args, **kwargs) 2025-08-14T21:47:02.4530665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4531022Z return func(*args, **kwargs) 2025-08-14T21:47:02.4531393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4531783Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4532149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4532524Z return func(*args, **kwargs) 2025-08-14T21:47:02.4532877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4533229Z return func(*args, **kwargs) 2025-08-14T21:47:02.4533589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4533941Z return func(*args, **kwargs) 2025-08-14T21:47:02.4534765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4535144Z self_outputs = self.self( 2025-08-14T21:47:02.4535491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4535843Z return func(*args, **kwargs) 2025-08-14T21:47:02.4536172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4536527Z return func(*args, **kwargs) 2025-08-14T21:47:02.4536867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4537228Z return func(*args, **kwargs) 2025-08-14T21:47:02.4537601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:47:02.4538238Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4538492Z 2025-08-14T21:47:02.4538640Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4539023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4539360Z return mod(**inputs) 2025-08-14T21:47:02.4539707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4540080Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4540471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4540859Z outputs = self.layoutlm( 2025-08-14T21:47:02.4541227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4541587Z return func(*args, **kwargs) 2025-08-14T21:47:02.4541928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4542288Z return func(*args, **kwargs) 2025-08-14T21:47:02.4542611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4542947Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4543335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4543728Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4544082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4544434Z return func(*args, **kwargs) 2025-08-14T21:47:02.4544781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4545135Z return func(*args, **kwargs) 2025-08-14T21:47:02.4545482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4545830Z return func(*args, **kwargs) 2025-08-14T21:47:02.4546011Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4546368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4546686Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4547062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4547440Z layer_outputs = layer_module( 2025-08-14T21:47:02.4547759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4548102Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4548468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4548833Z return func(*args, **kwargs) 2025-08-14T21:47:02.4549154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4549492Z return func(*args, **kwargs) 2025-08-14T21:47:02.4549817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4550154Z return func(*args, **kwargs) 2025-08-14T21:47:02.4550500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4550878Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4551222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4551547Z return func(*args, **kwargs) 2025-08-14T21:47:02.4551890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4552244Z return func(*args, **kwargs) 2025-08-14T21:47:02.4552569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4552897Z return func(*args, **kwargs) 2025-08-14T21:47:02.4553248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4553613Z self_outputs = self.self( 2025-08-14T21:47:02.4553934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4554269Z return func(*args, **kwargs) 2025-08-14T21:47:02.4554622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4554966Z return func(*args, **kwargs) 2025-08-14T21:47:02.4555287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4555619Z return func(*args, **kwargs) 2025-08-14T21:47:02.4555968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:47:02.4556385Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4556570Z 2025-08-14T21:47:02.4556668Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4557005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4557315Z return mod(**inputs) 2025-08-14T21:47:02.4557618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4557958Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4558348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4558740Z outputs = self.layoutlm( 2025-08-14T21:47:02.4559070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4559418Z return func(*args, **kwargs) 2025-08-14T21:47:02.4559639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4559712Z return func(*args, **kwargs) 2025-08-14T21:47:02.4559914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4559986Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4560251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4560342Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4560576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4560639Z return func(*args, **kwargs) 2025-08-14T21:47:02.4560859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4560930Z return func(*args, **kwargs) 2025-08-14T21:47:02.4561147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4561217Z return func(*args, **kwargs) 2025-08-14T21:47:02.4561290Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4561490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4561568Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4561837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4561918Z layer_outputs = layer_module( 2025-08-14T21:47:02.4562137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4562213Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4562442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4562505Z return func(*args, **kwargs) 2025-08-14T21:47:02.4562726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4562798Z return func(*args, **kwargs) 2025-08-14T21:47:02.4563040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4563105Z return func(*args, **kwargs) 2025-08-14T21:47:02.4563373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4563453Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4563682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4563746Z return func(*args, **kwargs) 2025-08-14T21:47:02.4563969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4564039Z return func(*args, **kwargs) 2025-08-14T21:47:02.4564264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4564327Z return func(*args, **kwargs) 2025-08-14T21:47:02.4564587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4564656Z self_outputs = self.self( 2025-08-14T21:47:02.4564888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4564951Z return func(*args, **kwargs) 2025-08-14T21:47:02.4565175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4565244Z return func(*args, **kwargs) 2025-08-14T21:47:02.4565531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4565607Z return func(*args, **kwargs) 2025-08-14T21:47:02.4565855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:47:02.4565996Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4566000Z 2025-08-14T21:47:02.4566108Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4566189Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4566291Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4566495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4566561Z return mod(**inputs) 2025-08-14T21:47:02.4566777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4566850Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4567110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4567187Z outputs = self.layoutlm( 2025-08-14T21:47:02.4567417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4567483Z return func(*args, **kwargs) 2025-08-14T21:47:02.4567737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4567821Z return func(*args, **kwargs) 2025-08-14T21:47:02.4568042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4568114Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4568370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4568450Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4568687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4568750Z return func(*args, **kwargs) 2025-08-14T21:47:02.4569002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4569068Z return func(*args, **kwargs) 2025-08-14T21:47:02.4569299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4569362Z return func(*args, **kwargs) 2025-08-14T21:47:02.4569433Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4569647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4569716Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4569967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4570042Z layer_outputs = layer_module( 2025-08-14T21:47:02.4570249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4570332Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4570553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4570622Z return func(*args, **kwargs) 2025-08-14T21:47:02.4570852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4570916Z return func(*args, **kwargs) 2025-08-14T21:47:02.4571147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4571210Z return func(*args, **kwargs) 2025-08-14T21:47:02.4571461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4571544Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4571769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4571833Z return func(*args, **kwargs) 2025-08-14T21:47:02.4572091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4572155Z return func(*args, **kwargs) 2025-08-14T21:47:02.4572382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4572445Z return func(*args, **kwargs) 2025-08-14T21:47:02.4572693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:47:02.4572822Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:47:02.4573071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:47:02.4573150Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4573162Z 2025-08-14T21:47:02.4573260Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4573485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4573557Z return mod(**inputs) 2025-08-14T21:47:02.4573763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4573834Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4574093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4574158Z outputs = self.layoutlm( 2025-08-14T21:47:02.4574392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4574456Z return func(*args, **kwargs) 2025-08-14T21:47:02.4574695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4574773Z return func(*args, **kwargs) 2025-08-14T21:47:02.4574979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4575051Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4575307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4575376Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4575605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4575668Z return func(*args, **kwargs) 2025-08-14T21:47:02.4575889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4575960Z return func(*args, **kwargs) 2025-08-14T21:47:02.4576184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4576252Z return func(*args, **kwargs) 2025-08-14T21:47:02.4576333Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4576536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4576613Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4576867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4576933Z layer_outputs = layer_module( 2025-08-14T21:47:02.4577152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4577225Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4577452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4577525Z return func(*args, **kwargs) 2025-08-14T21:47:02.4577766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4577840Z return func(*args, **kwargs) 2025-08-14T21:47:02.4578061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4578127Z return func(*args, **kwargs) 2025-08-14T21:47:02.4578385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4578467Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4578718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4578791Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4579072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4579236Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4579488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:47:02.4579565Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4579576Z 2025-08-14T21:47:02.4579673Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4579860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4579930Z return mod(**inputs) 2025-08-14T21:47:02.4580134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4580202Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4580501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4580569Z outputs = self.layoutlm( 2025-08-14T21:47:02.4580800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4580866Z return func(*args, **kwargs) 2025-08-14T21:47:02.4581086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4581157Z return func(*args, **kwargs) 2025-08-14T21:47:02.4581356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4581424Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4581679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4581748Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4581986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4582050Z return func(*args, **kwargs) 2025-08-14T21:47:02.4582264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4582334Z return func(*args, **kwargs) 2025-08-14T21:47:02.4582549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4582611Z return func(*args, **kwargs) 2025-08-14T21:47:02.4582688Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4582882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4582957Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4583203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4583271Z layer_outputs = layer_module( 2025-08-14T21:47:02.4583505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4583580Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4583804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4583878Z return func(*args, **kwargs) 2025-08-14T21:47:02.4584101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4584171Z return func(*args, **kwargs) 2025-08-14T21:47:02.4584393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4584455Z return func(*args, **kwargs) 2025-08-14T21:47:02.4584714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4584796Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4585076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4585152Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4585437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4585559Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4585807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:47:02.4585916Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:02.4586139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:47:02.4586208Z return self.act(input) 2025-08-14T21:47:02.4586214Z 2025-08-14T21:47:02.4586318Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4586509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4586574Z return mod(**inputs) 2025-08-14T21:47:02.4586790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4586863Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4587127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4587197Z outputs = self.layoutlm( 2025-08-14T21:47:02.4587431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4587504Z return func(*args, **kwargs) 2025-08-14T21:47:02.4587735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4587803Z return func(*args, **kwargs) 2025-08-14T21:47:02.4588028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4588098Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4588355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4588424Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4588649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4588719Z return func(*args, **kwargs) 2025-08-14T21:47:02.4588943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4589007Z return func(*args, **kwargs) 2025-08-14T21:47:02.4589238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4589322Z return func(*args, **kwargs) 2025-08-14T21:47:02.4589402Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4589606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4589675Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4589931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4589997Z layer_outputs = layer_module( 2025-08-14T21:47:02.4590204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4590285Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4590510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4590582Z return func(*args, **kwargs) 2025-08-14T21:47:02.4590832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4590898Z return func(*args, **kwargs) 2025-08-14T21:47:02.4591130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4591193Z return func(*args, **kwargs) 2025-08-14T21:47:02.4591442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4591530Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4591772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4591854Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4592148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:47:02.4592279Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:47:02.4592547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:47:02.4592624Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4592627Z 2025-08-14T21:47:02.4592731Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4592913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4592974Z return mod(**inputs) 2025-08-14T21:47:02.4593178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4593246Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4593494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4593564Z outputs = self.layoutlm( 2025-08-14T21:47:02.4593783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4593852Z return func(*args, **kwargs) 2025-08-14T21:47:02.4594071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4594133Z return func(*args, **kwargs) 2025-08-14T21:47:02.4594336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4594404Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4594654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4594723Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4594940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4595029Z return func(*args, **kwargs) 2025-08-14T21:47:02.4595243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4595304Z return func(*args, **kwargs) 2025-08-14T21:47:02.4595523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4595584Z return func(*args, **kwargs) 2025-08-14T21:47:02.4595662Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4595855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4595921Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4596169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4596250Z layer_outputs = layer_module( 2025-08-14T21:47:02.4596462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4596545Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4596769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4596840Z return func(*args, **kwargs) 2025-08-14T21:47:02.4597061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4597124Z return func(*args, **kwargs) 2025-08-14T21:47:02.4597353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4597417Z return func(*args, **kwargs) 2025-08-14T21:47:02.4597681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4597774Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4598002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4598075Z return func(*args, **kwargs) 2025-08-14T21:47:02.4598315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4598375Z return func(*args, **kwargs) 2025-08-14T21:47:02.4598599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4598660Z return func(*args, **kwargs) 2025-08-14T21:47:02.4598916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4598986Z self_outputs = self.self( 2025-08-14T21:47:02.4599212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4599285Z return func(*args, **kwargs) 2025-08-14T21:47:02.4599509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4599573Z return func(*args, **kwargs) 2025-08-14T21:47:02.4599805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4599868Z return func(*args, **kwargs) 2025-08-14T21:47:02.4600126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:47:02.4600264Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4600267Z 2025-08-14T21:47:02.4600364Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4600574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4600641Z return mod(**inputs) 2025-08-14T21:47:02.4600844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4600924Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4601172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4601246Z outputs = self.layoutlm( 2025-08-14T21:47:02.4601467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4601530Z return func(*args, **kwargs) 2025-08-14T21:47:02.4601756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4601819Z return func(*args, **kwargs) 2025-08-14T21:47:02.4602048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4602132Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4602387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4602463Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4602687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4602749Z return func(*args, **kwargs) 2025-08-14T21:47:02.4602979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4603042Z return func(*args, **kwargs) 2025-08-14T21:47:02.4603291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4603354Z return func(*args, **kwargs) 2025-08-14T21:47:02.4603430Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4603639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4603708Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4603957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4604032Z layer_outputs = layer_module( 2025-08-14T21:47:02.4604239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4604317Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4604541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4604605Z return func(*args, **kwargs) 2025-08-14T21:47:02.4604836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4604904Z return func(*args, **kwargs) 2025-08-14T21:47:02.4605130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4605204Z return func(*args, **kwargs) 2025-08-14T21:47:02.4605538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4605631Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4605863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4605928Z return func(*args, **kwargs) 2025-08-14T21:47:02.4606175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4606242Z return func(*args, **kwargs) 2025-08-14T21:47:02.4606508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4606577Z return func(*args, **kwargs) 2025-08-14T21:47:02.4606847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4606926Z self_outputs = self.self( 2025-08-14T21:47:02.4607152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4607217Z return func(*args, **kwargs) 2025-08-14T21:47:02.4607450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4607515Z return func(*args, **kwargs) 2025-08-14T21:47:02.4607749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4607814Z return func(*args, **kwargs) 2025-08-14T21:47:02.4608105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:47:02.4608248Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4608253Z 2025-08-14T21:47:02.4608351Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4608559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4608622Z return mod(**inputs) 2025-08-14T21:47:02.4608825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4608900Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4609158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4609225Z outputs = self.layoutlm( 2025-08-14T21:47:02.4609456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4609520Z return func(*args, **kwargs) 2025-08-14T21:47:02.4609746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4609809Z return func(*args, **kwargs) 2025-08-14T21:47:02.4610010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4610088Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4610336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4610403Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4610636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4610701Z return func(*args, **kwargs) 2025-08-14T21:47:02.4610933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4610995Z return func(*args, **kwargs) 2025-08-14T21:47:02.4611217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4611287Z return func(*args, **kwargs) 2025-08-14T21:47:02.4611359Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4611560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4611637Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4611884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4611960Z layer_outputs = layer_module( 2025-08-14T21:47:02.4612164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4612257Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4612487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4612551Z return func(*args, **kwargs) 2025-08-14T21:47:02.4612775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4612848Z return func(*args, **kwargs) 2025-08-14T21:47:02.4613069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4613139Z return func(*args, **kwargs) 2025-08-14T21:47:02.4613392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4613470Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4613735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4613801Z return func(*args, **kwargs) 2025-08-14T21:47:02.4614045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4614111Z return func(*args, **kwargs) 2025-08-14T21:47:02.4614341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4614424Z return func(*args, **kwargs) 2025-08-14T21:47:02.4614674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4614740Z self_outputs = self.self( 2025-08-14T21:47:02.4614986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4615053Z return func(*args, **kwargs) 2025-08-14T21:47:02.4615284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4615347Z return func(*args, **kwargs) 2025-08-14T21:47:02.4615567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4615637Z return func(*args, **kwargs) 2025-08-14T21:47:02.4615884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:47:02.4616029Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4616041Z 2025-08-14T21:47:02.4616116Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4616190Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4616294Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4616488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4616555Z return mod(**inputs) 2025-08-14T21:47:02.4616772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4616844Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4617110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4617186Z outputs = self.layoutlm( 2025-08-14T21:47:02.4617424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4617502Z return func(*args, **kwargs) 2025-08-14T21:47:02.4617734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4617800Z return func(*args, **kwargs) 2025-08-14T21:47:02.4618044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4618120Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4618399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4618470Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4618701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4618773Z return func(*args, **kwargs) 2025-08-14T21:47:02.4619000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4619064Z return func(*args, **kwargs) 2025-08-14T21:47:02.4619299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4619365Z return func(*args, **kwargs) 2025-08-14T21:47:02.4619469Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4619690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4619763Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4620028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4620097Z layer_outputs = layer_module( 2025-08-14T21:47:02.4620310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4620393Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4620622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4620706Z return func(*args, **kwargs) 2025-08-14T21:47:02.4620936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4621006Z return func(*args, **kwargs) 2025-08-14T21:47:02.4621245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4621310Z return func(*args, **kwargs) 2025-08-14T21:47:02.4621568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4621655Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4621879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4621952Z return func(*args, **kwargs) 2025-08-14T21:47:02.4622180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4622243Z return func(*args, **kwargs) 2025-08-14T21:47:02.4622480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4622545Z return func(*args, **kwargs) 2025-08-14T21:47:02.4622808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:47:02.4622932Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:47:02.4623188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:47:02.4623274Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4623277Z 2025-08-14T21:47:02.4623374Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4623567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4623638Z return mod(**inputs) 2025-08-14T21:47:02.4623871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4623952Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4624206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4624273Z outputs = self.layoutlm( 2025-08-14T21:47:02.4624503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4624569Z return func(*args, **kwargs) 2025-08-14T21:47:02.4624792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4624863Z return func(*args, **kwargs) 2025-08-14T21:47:02.4625072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4625151Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4625438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4625511Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4625747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4625813Z return func(*args, **kwargs) 2025-08-14T21:47:02.4626045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4626109Z return func(*args, **kwargs) 2025-08-14T21:47:02.4626333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4626407Z return func(*args, **kwargs) 2025-08-14T21:47:02.4627407Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4627633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4627718Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4627978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4628058Z layer_outputs = layer_module( 2025-08-14T21:47:02.4628273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4628351Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4628594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4628661Z return func(*args, **kwargs) 2025-08-14T21:47:02.4628895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4628971Z return func(*args, **kwargs) 2025-08-14T21:47:02.4629206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4629283Z return func(*args, **kwargs) 2025-08-14T21:47:02.4629548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4629644Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4629898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4629973Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4630255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4630378Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4630632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:47:02.4630745Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4630749Z 2025-08-14T21:47:02.4630847Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4631039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4631111Z return mod(**inputs) 2025-08-14T21:47:02.4631319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4631397Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4631649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4631717Z outputs = self.layoutlm( 2025-08-14T21:47:02.4631953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4632020Z return func(*args, **kwargs) 2025-08-14T21:47:02.4632279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4632353Z return func(*args, **kwargs) 2025-08-14T21:47:02.4632558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4632636Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4632890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4632961Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4633196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4633261Z return func(*args, **kwargs) 2025-08-14T21:47:02.4633507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4633576Z return func(*args, **kwargs) 2025-08-14T21:47:02.4633805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4633875Z return func(*args, **kwargs) 2025-08-14T21:47:02.4633950Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4634156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4634235Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4634489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4634566Z layer_outputs = layer_module( 2025-08-14T21:47:02.4634776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4634853Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4635089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4635155Z return func(*args, **kwargs) 2025-08-14T21:47:02.4635380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4635454Z return func(*args, **kwargs) 2025-08-14T21:47:02.4635680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4635750Z return func(*args, **kwargs) 2025-08-14T21:47:02.4636005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4636085Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4636342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4636434Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4636725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4636852Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4637116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:47:02.4637234Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:02.4637442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:47:02.4637512Z return self.act(input) 2025-08-14T21:47:02.4637516Z 2025-08-14T21:47:02.4637777Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4637992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4638069Z return mod(**inputs) 2025-08-14T21:47:02.4638354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4638430Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4638710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4638786Z outputs = self.layoutlm( 2025-08-14T21:47:02.4639037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4639114Z return func(*args, **kwargs) 2025-08-14T21:47:02.4639364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4639443Z return func(*args, **kwargs) 2025-08-14T21:47:02.4639706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4639790Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4640081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4640160Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4640410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4640489Z return func(*args, **kwargs) 2025-08-14T21:47:02.4640736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4640815Z return func(*args, **kwargs) 2025-08-14T21:47:02.4641063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4641135Z return func(*args, **kwargs) 2025-08-14T21:47:02.4641223Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4641452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4641539Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4641825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4641902Z layer_outputs = layer_module( 2025-08-14T21:47:02.4642138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4642218Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4642468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4642548Z return func(*args, **kwargs) 2025-08-14T21:47:02.4642797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4642905Z return func(*args, **kwargs) 2025-08-14T21:47:02.4643164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4643235Z return func(*args, **kwargs) 2025-08-14T21:47:02.4643523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4643614Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4643885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4643973Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4644289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:47:02.4644436Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:47:02.4644737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:47:02.4644843Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4644848Z 2025-08-14T21:47:02.4644964Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4645178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4645254Z return mod(**inputs) 2025-08-14T21:47:02.4645635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4645724Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4646010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4646087Z outputs = self.layoutlm( 2025-08-14T21:47:02.4646372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4646461Z return func(*args, **kwargs) 2025-08-14T21:47:02.4646705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4646781Z return func(*args, **kwargs) 2025-08-14T21:47:02.4646989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4647063Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4647337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4647407Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4647628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4647703Z return func(*args, **kwargs) 2025-08-14T21:47:02.4647932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4648012Z return func(*args, **kwargs) 2025-08-14T21:47:02.4648242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4648309Z return func(*args, **kwargs) 2025-08-14T21:47:02.4648393Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4648600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4648671Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4648938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4649017Z layer_outputs = layer_module( 2025-08-14T21:47:02.4649230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4649320Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4649546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4649616Z return func(*args, **kwargs) 2025-08-14T21:47:02.4649838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4649907Z return func(*args, **kwargs) 2025-08-14T21:47:02.4650129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4650194Z return func(*args, **kwargs) 2025-08-14T21:47:02.4650453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4650533Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4650758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4650862Z return func(*args, **kwargs) 2025-08-14T21:47:02.4651085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4651155Z return func(*args, **kwargs) 2025-08-14T21:47:02.4651382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4651448Z return func(*args, **kwargs) 2025-08-14T21:47:02.4651709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4651778Z self_outputs = self.self( 2025-08-14T21:47:02.4652006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4652099Z return func(*args, **kwargs) 2025-08-14T21:47:02.4652333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4652409Z return func(*args, **kwargs) 2025-08-14T21:47:02.4652635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4652698Z return func(*args, **kwargs) 2025-08-14T21:47:02.4652962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:47:02.4653104Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4653109Z 2025-08-14T21:47:02.4653211Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4653403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4653467Z return mod(**inputs) 2025-08-14T21:47:02.4653681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4653756Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4654012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4654086Z outputs = self.layoutlm( 2025-08-14T21:47:02.4654316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4654388Z return func(*args, **kwargs) 2025-08-14T21:47:02.4654617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4654681Z return func(*args, **kwargs) 2025-08-14T21:47:02.4654895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4654968Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4655248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4655326Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4655554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4655628Z return func(*args, **kwargs) 2025-08-14T21:47:02.4655854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4655919Z return func(*args, **kwargs) 2025-08-14T21:47:02.4656149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4656214Z return func(*args, **kwargs) 2025-08-14T21:47:02.4656288Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4656502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4656595Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4656881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4656952Z layer_outputs = layer_module( 2025-08-14T21:47:02.4657161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4657244Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4657471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4657544Z return func(*args, **kwargs) 2025-08-14T21:47:02.4657771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4657851Z return func(*args, **kwargs) 2025-08-14T21:47:02.4658087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4658157Z return func(*args, **kwargs) 2025-08-14T21:47:02.4658416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4658503Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4658731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4658802Z return func(*args, **kwargs) 2025-08-14T21:47:02.4659031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4659094Z return func(*args, **kwargs) 2025-08-14T21:47:02.4659331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4659395Z return func(*args, **kwargs) 2025-08-14T21:47:02.4659654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4659730Z self_outputs = self.self( 2025-08-14T21:47:02.4659962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4660031Z return func(*args, **kwargs) 2025-08-14T21:47:02.4660256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4660320Z return func(*args, **kwargs) 2025-08-14T21:47:02.4660554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4660621Z return func(*args, **kwargs) 2025-08-14T21:47:02.4660888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:47:02.4661040Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4661046Z 2025-08-14T21:47:02.4661146Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4661347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4661410Z return mod(**inputs) 2025-08-14T21:47:02.4661621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4661702Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4661964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4662038Z outputs = self.layoutlm( 2025-08-14T21:47:02.4662271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4662336Z return func(*args, **kwargs) 2025-08-14T21:47:02.4662599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4662677Z return func(*args, **kwargs) 2025-08-14T21:47:02.4662878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4662957Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4663204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4663279Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4663499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4663562Z return func(*args, **kwargs) 2025-08-14T21:47:02.4663806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4663871Z return func(*args, **kwargs) 2025-08-14T21:47:02.4664104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4664167Z return func(*args, **kwargs) 2025-08-14T21:47:02.4664240Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4664450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4664519Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4664769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4664842Z layer_outputs = layer_module( 2025-08-14T21:47:02.4665048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4665129Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4665352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4665419Z return func(*args, **kwargs) 2025-08-14T21:47:02.4665647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4665710Z return func(*args, **kwargs) 2025-08-14T21:47:02.4665931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4666000Z return func(*args, **kwargs) 2025-08-14T21:47:02.4666249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4666333Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4666557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4666619Z return func(*args, **kwargs) 2025-08-14T21:47:02.4666863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4666925Z return func(*args, **kwargs) 2025-08-14T21:47:02.4667145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4667215Z return func(*args, **kwargs) 2025-08-14T21:47:02.4667462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4667536Z self_outputs = self.self( 2025-08-14T21:47:02.4667756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4667818Z return func(*args, **kwargs) 2025-08-14T21:47:02.4668044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4668109Z return func(*args, **kwargs) 2025-08-14T21:47:02.4668365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4668435Z return func(*args, **kwargs) 2025-08-14T21:47:02.4668690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:47:02.4668834Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4668838Z 2025-08-14T21:47:02.4668912Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4668986Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4669090Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4669283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4669368Z return mod(**inputs) 2025-08-14T21:47:02.4669587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4669661Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4669917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4669984Z outputs = self.layoutlm( 2025-08-14T21:47:02.4670208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4670279Z return func(*args, **kwargs) 2025-08-14T21:47:02.4670501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4670570Z return func(*args, **kwargs) 2025-08-14T21:47:02.4670772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4670844Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4671102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4671172Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4671392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4671463Z return func(*args, **kwargs) 2025-08-14T21:47:02.4671685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4671755Z return func(*args, **kwargs) 2025-08-14T21:47:02.4671975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4672038Z return func(*args, **kwargs) 2025-08-14T21:47:02.4672119Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4672324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4672412Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4672673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4672741Z layer_outputs = layer_module( 2025-08-14T21:47:02.4672956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4673030Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4673252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4673323Z return func(*args, **kwargs) 2025-08-14T21:47:02.4673545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4673615Z return func(*args, **kwargs) 2025-08-14T21:47:02.4673836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4673958Z return func(*args, **kwargs) 2025-08-14T21:47:02.4674228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4674305Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4674527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4674597Z return func(*args, **kwargs) 2025-08-14T21:47:02.4674818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4674887Z return func(*args, **kwargs) 2025-08-14T21:47:02.4675127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4675190Z return func(*args, **kwargs) 2025-08-14T21:47:02.4675448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:47:02.4675569Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:47:02.4675821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:47:02.4675906Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4675909Z 2025-08-14T21:47:02.4676004Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4676198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4676261Z return mod(**inputs) 2025-08-14T21:47:02.4676467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4676548Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4676800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4676875Z outputs = self.layoutlm( 2025-08-14T21:47:02.4677100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4677163Z return func(*args, **kwargs) 2025-08-14T21:47:02.4677396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4677459Z return func(*args, **kwargs) 2025-08-14T21:47:02.4677662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4677738Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4677990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4678065Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4678309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4678372Z return func(*args, **kwargs) 2025-08-14T21:47:02.4678600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4678664Z return func(*args, **kwargs) 2025-08-14T21:47:02.4678883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4678951Z return func(*args, **kwargs) 2025-08-14T21:47:02.4679021Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4679228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4679297Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4679546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4679648Z layer_outputs = layer_module( 2025-08-14T21:47:02.4679859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4679933Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4680162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4680226Z return func(*args, **kwargs) 2025-08-14T21:47:02.4680456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4680517Z return func(*args, **kwargs) 2025-08-14T21:47:02.4680739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4680820Z return func(*args, **kwargs) 2025-08-14T21:47:02.4681073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4681160Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4681402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4681473Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4681756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4681864Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4682111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:47:02.4682194Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4682200Z 2025-08-14T21:47:02.4682293Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4682488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4682550Z return mod(**inputs) 2025-08-14T21:47:02.4682750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4682827Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4683079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4683153Z outputs = self.layoutlm( 2025-08-14T21:47:02.4683382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4683448Z return func(*args, **kwargs) 2025-08-14T21:47:02.4683682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4683765Z return func(*args, **kwargs) 2025-08-14T21:47:02.4683982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4684061Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4684322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4684402Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4684635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4684698Z return func(*args, **kwargs) 2025-08-14T21:47:02.4684934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4684998Z return func(*args, **kwargs) 2025-08-14T21:47:02.4685227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4685321Z return func(*args, **kwargs) 2025-08-14T21:47:02.4685494Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4685744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4685819Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4686099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4686183Z layer_outputs = layer_module( 2025-08-14T21:47:02.4686411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4686494Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4686764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4686839Z return func(*args, **kwargs) 2025-08-14T21:47:02.4687099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4687180Z return func(*args, **kwargs) 2025-08-14T21:47:02.4687409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4687487Z return func(*args, **kwargs) 2025-08-14T21:47:02.4687744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4687837Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4688087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4688161Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4688456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4688575Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4688834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:47:02.4688954Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:02.4689162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:47:02.4689239Z return self.act(input) 2025-08-14T21:47:02.4689243Z 2025-08-14T21:47:02.4689345Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4689541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4689614Z return mod(**inputs) 2025-08-14T21:47:02.4689828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4689929Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4690196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4690264Z outputs = self.layoutlm( 2025-08-14T21:47:02.4690511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4690581Z return func(*args, **kwargs) 2025-08-14T21:47:02.4690820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4690895Z return func(*args, **kwargs) 2025-08-14T21:47:02.4691114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4691197Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4691466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4691566Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4691817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4691883Z return func(*args, **kwargs) 2025-08-14T21:47:02.4692112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4692185Z return func(*args, **kwargs) 2025-08-14T21:47:02.4692413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4692485Z return func(*args, **kwargs) 2025-08-14T21:47:02.4692560Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4692768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4692863Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4693128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4693198Z layer_outputs = layer_module( 2025-08-14T21:47:02.4693416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4693491Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4693725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4693790Z return func(*args, **kwargs) 2025-08-14T21:47:02.4694016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4694086Z return func(*args, **kwargs) 2025-08-14T21:47:02.4694314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4694380Z return func(*args, **kwargs) 2025-08-14T21:47:02.4694642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4694721Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4694972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4695044Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4695325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:47:02.4695459Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:47:02.4695714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:47:02.4695800Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4695817Z 2025-08-14T21:47:02.4695920Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4696122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4696199Z return mod(**inputs) 2025-08-14T21:47:02.4696412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4696486Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4696755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4696825Z outputs = self.layoutlm( 2025-08-14T21:47:02.4697067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4697138Z return func(*args, **kwargs) 2025-08-14T21:47:02.4697379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4697473Z return func(*args, **kwargs) 2025-08-14T21:47:02.4697703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4697785Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4698053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4698127Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4698370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4698437Z return func(*args, **kwargs) 2025-08-14T21:47:02.4698681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4698768Z return func(*args, **kwargs) 2025-08-14T21:47:02.4698997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4699072Z return func(*args, **kwargs) 2025-08-14T21:47:02.4699147Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4699356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4699434Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4699686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4699755Z layer_outputs = layer_module( 2025-08-14T21:47:02.4699971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4700046Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4700276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4700343Z return func(*args, **kwargs) 2025-08-14T21:47:02.4700574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4700650Z return func(*args, **kwargs) 2025-08-14T21:47:02.4700875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4700940Z return func(*args, **kwargs) 2025-08-14T21:47:02.4701213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4701291Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4701521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4701584Z return func(*args, **kwargs) 2025-08-14T21:47:02.4701805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4701900Z return func(*args, **kwargs) 2025-08-14T21:47:02.4702126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4702197Z return func(*args, **kwargs) 2025-08-14T21:47:02.4702450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4702518Z self_outputs = self.self( 2025-08-14T21:47:02.4702750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4702815Z return func(*args, **kwargs) 2025-08-14T21:47:02.4703038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4703108Z return func(*args, **kwargs) 2025-08-14T21:47:02.4703330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4703432Z return func(*args, **kwargs) 2025-08-14T21:47:02.4703688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:47:02.4703825Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4703829Z 2025-08-14T21:47:02.4703932Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4704120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4704184Z return mod(**inputs) 2025-08-14T21:47:02.4704399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4704469Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4704744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4704815Z outputs = self.layoutlm( 2025-08-14T21:47:02.4705038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4705110Z return func(*args, **kwargs) 2025-08-14T21:47:02.4705332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4705403Z return func(*args, **kwargs) 2025-08-14T21:47:02.4705608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4705677Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4705932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4706002Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4706226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4706300Z return func(*args, **kwargs) 2025-08-14T21:47:02.4706521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4706591Z return func(*args, **kwargs) 2025-08-14T21:47:02.4706811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4706876Z return func(*args, **kwargs) 2025-08-14T21:47:02.4706953Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4707156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4707223Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4707479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4707571Z layer_outputs = layer_module( 2025-08-14T21:47:02.4707784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4707857Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4708080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4708151Z return func(*args, **kwargs) 2025-08-14T21:47:02.4708370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4708433Z return func(*args, **kwargs) 2025-08-14T21:47:02.4708659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4708720Z return func(*args, **kwargs) 2025-08-14T21:47:02.4708973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4709081Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4709300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4709371Z return func(*args, **kwargs) 2025-08-14T21:47:02.4709591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4709660Z return func(*args, **kwargs) 2025-08-14T21:47:02.4709878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4709940Z return func(*args, **kwargs) 2025-08-14T21:47:02.4710212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4710280Z self_outputs = self.self( 2025-08-14T21:47:02.4710507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4710578Z return func(*args, **kwargs) 2025-08-14T21:47:02.4710801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4710871Z return func(*args, **kwargs) 2025-08-14T21:47:02.4711093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4711154Z return func(*args, **kwargs) 2025-08-14T21:47:02.4711412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:47:02.4711543Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4711547Z 2025-08-14T21:47:02.4711643Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4711839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4711907Z return mod(**inputs) 2025-08-14T21:47:02.4712118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4712191Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4712448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4712523Z outputs = self.layoutlm( 2025-08-14T21:47:02.4712753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4712824Z return func(*args, **kwargs) 2025-08-14T21:47:02.4713059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4713124Z return func(*args, **kwargs) 2025-08-14T21:47:02.4713372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4713447Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4713711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4713789Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4714012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4714081Z return func(*args, **kwargs) 2025-08-14T21:47:02.4714306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4714371Z return func(*args, **kwargs) 2025-08-14T21:47:02.4714600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4714664Z return func(*args, **kwargs) 2025-08-14T21:47:02.4714757Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4714982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4715053Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4715311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4715378Z layer_outputs = layer_module( 2025-08-14T21:47:02.4715583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4715663Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4715886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4715973Z return func(*args, **kwargs) 2025-08-14T21:47:02.4716207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4716273Z return func(*args, **kwargs) 2025-08-14T21:47:02.4716499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4716562Z return func(*args, **kwargs) 2025-08-14T21:47:02.4716815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4716900Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4717121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4717192Z return func(*args, **kwargs) 2025-08-14T21:47:02.4717420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4717483Z return func(*args, **kwargs) 2025-08-14T21:47:02.4717715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4717780Z return func(*args, **kwargs) 2025-08-14T21:47:02.4718032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4718106Z self_outputs = self.self( 2025-08-14T21:47:02.4718330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4718400Z return func(*args, **kwargs) 2025-08-14T21:47:02.4718627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4718689Z return func(*args, **kwargs) 2025-08-14T21:47:02.4718920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4718998Z return func(*args, **kwargs) 2025-08-14T21:47:02.4719253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:47:02.4719397Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4719401Z 2025-08-14T21:47:02.4719477Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4719557Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4719654Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4719842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4719913Z return mod(**inputs) 2025-08-14T21:47:02.4720112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4720184Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4720441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4720535Z outputs = self.layoutlm( 2025-08-14T21:47:02.4720767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4720832Z return func(*args, **kwargs) 2025-08-14T21:47:02.4721054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4721124Z return func(*args, **kwargs) 2025-08-14T21:47:02.4721326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4721401Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4721652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4721739Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4721970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4722035Z return func(*args, **kwargs) 2025-08-14T21:47:02.4722253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4722324Z return func(*args, **kwargs) 2025-08-14T21:47:02.4722543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4722613Z return func(*args, **kwargs) 2025-08-14T21:47:02.4722685Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4722884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4722959Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4723209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4723279Z layer_outputs = layer_module( 2025-08-14T21:47:02.4723492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4723563Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4723792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4723854Z return func(*args, **kwargs) 2025-08-14T21:47:02.4724073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4724144Z return func(*args, **kwargs) 2025-08-14T21:47:02.4724362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4724427Z return func(*args, **kwargs) 2025-08-14T21:47:02.4724687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4724786Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4725015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4725079Z return func(*args, **kwargs) 2025-08-14T21:47:02.4725306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4725379Z return func(*args, **kwargs) 2025-08-14T21:47:02.4725683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4725758Z return func(*args, **kwargs) 2025-08-14T21:47:02.4726048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:47:02.4726184Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:47:02.4726513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:47:02.4726605Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4726609Z 2025-08-14T21:47:02.4726716Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4726934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4727012Z return mod(**inputs) 2025-08-14T21:47:02.4727230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4727305Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4727568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4727661Z outputs = self.layoutlm( 2025-08-14T21:47:02.4727902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4727971Z return func(*args, **kwargs) 2025-08-14T21:47:02.4728212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4728280Z return func(*args, **kwargs) 2025-08-14T21:47:02.4728503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4728578Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4728841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4728922Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4729158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4729232Z return func(*args, **kwargs) 2025-08-14T21:47:02.4729473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4729540Z return func(*args, **kwargs) 2025-08-14T21:47:02.4729784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4729850Z return func(*args, **kwargs) 2025-08-14T21:47:02.4729926Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4730146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4730220Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4730504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4730573Z layer_outputs = layer_module( 2025-08-14T21:47:02.4730793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4730896Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4731131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4731198Z return func(*args, **kwargs) 2025-08-14T21:47:02.4731448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4731513Z return func(*args, **kwargs) 2025-08-14T21:47:02.4731747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4731812Z return func(*args, **kwargs) 2025-08-14T21:47:02.4732069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4732160Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4732429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4732519Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4732815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4732929Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4733194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:47:02.4733276Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4733279Z 2025-08-14T21:47:02.4733380Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4733605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4733671Z return mod(**inputs) 2025-08-14T21:47:02.4733890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4733963Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4734219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4734296Z outputs = self.layoutlm( 2025-08-14T21:47:02.4734528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4734596Z return func(*args, **kwargs) 2025-08-14T21:47:02.4734835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4734902Z return func(*args, **kwargs) 2025-08-14T21:47:02.4735117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4735189Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4735456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4735537Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4735768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4735834Z return func(*args, **kwargs) 2025-08-14T21:47:02.4736071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4736136Z return func(*args, **kwargs) 2025-08-14T21:47:02.4736371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4736436Z return func(*args, **kwargs) 2025-08-14T21:47:02.4736511Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4736730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4736821Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4737093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4737164Z layer_outputs = layer_module( 2025-08-14T21:47:02.4737379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4737462Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4737877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4737955Z return func(*args, **kwargs) 2025-08-14T21:47:02.4738920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4739099Z return func(*args, **kwargs) 2025-08-14T21:47:02.4739607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4739740Z return func(*args, **kwargs) 2025-08-14T21:47:02.4740017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4740116Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4740390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4740471Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4740789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4740923Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4741272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:47:02.4741405Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:02.4741630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:47:02.4741715Z return self.act(input) 2025-08-14T21:47:02.4741723Z 2025-08-14T21:47:02.4741839Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4742140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4742217Z return mod(**inputs) 2025-08-14T21:47:02.4742453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4742542Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4742843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4742925Z outputs = self.layoutlm( 2025-08-14T21:47:02.4743212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4743288Z return func(*args, **kwargs) 2025-08-14T21:47:02.4743563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4743635Z return func(*args, **kwargs) 2025-08-14T21:47:02.4743868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4743958Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4744255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4744338Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4744616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4744733Z return func(*args, **kwargs) 2025-08-14T21:47:02.4744996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4745067Z return func(*args, **kwargs) 2025-08-14T21:47:02.4745327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4745405Z return func(*args, **kwargs) 2025-08-14T21:47:02.4745493Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4745771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4745859Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4746154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4746240Z layer_outputs = layer_module( 2025-08-14T21:47:02.4746506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4746614Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4746923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4746995Z return func(*args, **kwargs) 2025-08-14T21:47:02.4747262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4747334Z return func(*args, **kwargs) 2025-08-14T21:47:02.4747585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4747659Z return func(*args, **kwargs) 2025-08-14T21:47:02.4747943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4748032Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4748301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4748382Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4748686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:47:02.4748822Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:47:02.4749091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:47:02.4749182Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4749187Z 2025-08-14T21:47:02.4749296Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4749509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4749575Z return mod(**inputs) 2025-08-14T21:47:02.4749784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4749862Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4750110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4750178Z outputs = self.layoutlm( 2025-08-14T21:47:02.4750410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4750473Z return func(*args, **kwargs) 2025-08-14T21:47:02.4750701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4750765Z return func(*args, **kwargs) 2025-08-14T21:47:02.4750968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4751067Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4751317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4751392Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4751626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4751689Z return func(*args, **kwargs) 2025-08-14T21:47:02.4751919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4751983Z return func(*args, **kwargs) 2025-08-14T21:47:02.4752207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4752279Z return func(*args, **kwargs) 2025-08-14T21:47:02.4752354Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4752575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4752668Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4752917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4752995Z layer_outputs = layer_module( 2025-08-14T21:47:02.4753202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4753279Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4753510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4753573Z return func(*args, **kwargs) 2025-08-14T21:47:02.4753815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4753880Z return func(*args, **kwargs) 2025-08-14T21:47:02.4754107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4754174Z return func(*args, **kwargs) 2025-08-14T21:47:02.4754421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4754502Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4754733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4754796Z return func(*args, **kwargs) 2025-08-14T21:47:02.4755023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4755086Z return func(*args, **kwargs) 2025-08-14T21:47:02.4755311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4755384Z return func(*args, **kwargs) 2025-08-14T21:47:02.4755640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4755710Z self_outputs = self.self( 2025-08-14T21:47:02.4755949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4756013Z return func(*args, **kwargs) 2025-08-14T21:47:02.4756248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4756313Z return func(*args, **kwargs) 2025-08-14T21:47:02.4756538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4756612Z return func(*args, **kwargs) 2025-08-14T21:47:02.4756873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:47:02.4757049Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4757054Z 2025-08-14T21:47:02.4757162Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4757376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4757446Z return mod(**inputs) 2025-08-14T21:47:02.4757654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4757724Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4757987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4758054Z outputs = self.layoutlm( 2025-08-14T21:47:02.4758292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4758406Z return func(*args, **kwargs) 2025-08-14T21:47:02.4758634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4758709Z return func(*args, **kwargs) 2025-08-14T21:47:02.4758914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4758983Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4759241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4759313Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4759544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4759625Z return func(*args, **kwargs) 2025-08-14T21:47:02.4759851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4759928Z return func(*args, **kwargs) 2025-08-14T21:47:02.4760148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4760212Z return func(*args, **kwargs) 2025-08-14T21:47:02.4760292Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4760496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4760570Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4760818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4760886Z layer_outputs = layer_module( 2025-08-14T21:47:02.4761103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4761179Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4761411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4761473Z return func(*args, **kwargs) 2025-08-14T21:47:02.4761699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4761767Z return func(*args, **kwargs) 2025-08-14T21:47:02.4761990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4762051Z return func(*args, **kwargs) 2025-08-14T21:47:02.4762305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4762384Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4762615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4762700Z return func(*args, **kwargs) 2025-08-14T21:47:02.4762927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4762999Z return func(*args, **kwargs) 2025-08-14T21:47:02.4763229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4763292Z return func(*args, **kwargs) 2025-08-14T21:47:02.4763555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4763624Z self_outputs = self.self( 2025-08-14T21:47:02.4763859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4763925Z return func(*args, **kwargs) 2025-08-14T21:47:02.4764173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4764262Z return func(*args, **kwargs) 2025-08-14T21:47:02.4764500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4764566Z return func(*args, **kwargs) 2025-08-14T21:47:02.4764838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:47:02.4764972Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4764976Z 2025-08-14T21:47:02.4765083Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4765285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4765366Z return mod(**inputs) 2025-08-14T21:47:02.4765784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4765875Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4766147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4766217Z outputs = self.layoutlm( 2025-08-14T21:47:02.4766452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4766526Z return func(*args, **kwargs) 2025-08-14T21:47:02.4766760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4766827Z return func(*args, **kwargs) 2025-08-14T21:47:02.4767046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4767122Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4767397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4767472Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4767707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4767783Z return func(*args, **kwargs) 2025-08-14T21:47:02.4768017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4768085Z return func(*args, **kwargs) 2025-08-14T21:47:02.4768330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4768396Z return func(*args, **kwargs) 2025-08-14T21:47:02.4768479Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4768693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4768788Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4769064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4769133Z layer_outputs = layer_module( 2025-08-14T21:47:02.4769349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4769426Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4769660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4769734Z return func(*args, **kwargs) 2025-08-14T21:47:02.4769965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4770032Z return func(*args, **kwargs) 2025-08-14T21:47:02.4770271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4770405Z return func(*args, **kwargs) 2025-08-14T21:47:02.4770678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4770761Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4771011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4771088Z return func(*args, **kwargs) 2025-08-14T21:47:02.4771369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4771440Z return func(*args, **kwargs) 2025-08-14T21:47:02.4771716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4771789Z return func(*args, **kwargs) 2025-08-14T21:47:02.4772083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4772159Z self_outputs = self.self( 2025-08-14T21:47:02.4772408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4772487Z return func(*args, **kwargs) 2025-08-14T21:47:02.4772735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4772804Z return func(*args, **kwargs) 2025-08-14T21:47:02.4773066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4773138Z return func(*args, **kwargs) 2025-08-14T21:47:02.4773436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:47:02.4773586Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4773594Z 2025-08-14T21:47:02.4773675Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4773760Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4773862Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4774079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4774144Z return mod(**inputs) 2025-08-14T21:47:02.4774352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4774432Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4774688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4774756Z outputs = self.layoutlm( 2025-08-14T21:47:02.4774994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4775082Z return func(*args, **kwargs) 2025-08-14T21:47:02.4775324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4775392Z return func(*args, **kwargs) 2025-08-14T21:47:02.4775604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4775684Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4775941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4776012Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4776252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4776318Z return func(*args, **kwargs) 2025-08-14T21:47:02.4776584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4776665Z return func(*args, **kwargs) 2025-08-14T21:47:02.4776898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4776969Z return func(*args, **kwargs) 2025-08-14T21:47:02.4777043Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4777253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4777331Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4777589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4777665Z layer_outputs = layer_module( 2025-08-14T21:47:02.4777895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4777974Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4778211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4778276Z return func(*args, **kwargs) 2025-08-14T21:47:02.4778511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4778576Z return func(*args, **kwargs) 2025-08-14T21:47:02.4778804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4778879Z return func(*args, **kwargs) 2025-08-14T21:47:02.4779143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4779222Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4779460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4779530Z return func(*args, **kwargs) 2025-08-14T21:47:02.4779765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4779829Z return func(*args, **kwargs) 2025-08-14T21:47:02.4780059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4780131Z return func(*args, **kwargs) 2025-08-14T21:47:02.4780386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:47:02.4780522Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:47:02.4780781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:47:02.4780879Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4780884Z 2025-08-14T21:47:02.4780992Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4781185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4781248Z return mod(**inputs) 2025-08-14T21:47:02.4781461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4781532Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4781795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4781862Z outputs = self.layoutlm( 2025-08-14T21:47:02.4782094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4782168Z return func(*args, **kwargs) 2025-08-14T21:47:02.4782396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4782505Z return func(*args, **kwargs) 2025-08-14T21:47:02.4782724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4782797Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4783061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4783133Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4783362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4783432Z return func(*args, **kwargs) 2025-08-14T21:47:02.4783676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4783743Z return func(*args, **kwargs) 2025-08-14T21:47:02.4783984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4784048Z return func(*args, **kwargs) 2025-08-14T21:47:02.4784128Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4784336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4784405Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4784668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4784737Z layer_outputs = layer_module( 2025-08-14T21:47:02.4784950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4785033Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4785263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4785338Z return func(*args, **kwargs) 2025-08-14T21:47:02.4785569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4785632Z return func(*args, **kwargs) 2025-08-14T21:47:02.4785868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4785932Z return func(*args, **kwargs) 2025-08-14T21:47:02.4786201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4786284Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4786537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4786620Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4786937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4787060Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4787324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:47:02.4787404Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4787407Z 2025-08-14T21:47:02.4787515Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4787710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4787775Z return mod(**inputs) 2025-08-14T21:47:02.4787993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4788068Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4788357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4788446Z outputs = self.layoutlm( 2025-08-14T21:47:02.4788685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4788773Z return func(*args, **kwargs) 2025-08-14T21:47:02.4789008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4789073Z return func(*args, **kwargs) 2025-08-14T21:47:02.4789290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4789361Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4789640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4789715Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4789953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4790027Z return func(*args, **kwargs) 2025-08-14T21:47:02.4790256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4790320Z return func(*args, **kwargs) 2025-08-14T21:47:02.4790553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4790617Z return func(*args, **kwargs) 2025-08-14T21:47:02.4790716Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4790925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4790995Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4791258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4791330Z layer_outputs = layer_module( 2025-08-14T21:47:02.4791542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4791625Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4791854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4791927Z return func(*args, **kwargs) 2025-08-14T21:47:02.4792154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4792219Z return func(*args, **kwargs) 2025-08-14T21:47:02.4792455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4792522Z return func(*args, **kwargs) 2025-08-14T21:47:02.4792804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4792895Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4793151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4793235Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4793532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4793650Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4793920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:47:02.4794028Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:02.4794245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:47:02.4794334Z return self.act(input) 2025-08-14T21:47:02.4794353Z 2025-08-14T21:47:02.4794455Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4794656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4794722Z return mod(**inputs) 2025-08-14T21:47:02.4794929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4795011Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4795268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4795343Z outputs = self.layoutlm( 2025-08-14T21:47:02.4795589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4795657Z return func(*args, **kwargs) 2025-08-14T21:47:02.4795900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4795967Z return func(*args, **kwargs) 2025-08-14T21:47:02.4796191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4796263Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4796519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4796596Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4796826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4796892Z return func(*args, **kwargs) 2025-08-14T21:47:02.4797135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4797203Z return func(*args, **kwargs) 2025-08-14T21:47:02.4797451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4797514Z return func(*args, **kwargs) 2025-08-14T21:47:02.4797587Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4797798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4797868Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4798135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4798211Z layer_outputs = layer_module( 2025-08-14T21:47:02.4798427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4798510Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4798762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4798828Z return func(*args, **kwargs) 2025-08-14T21:47:02.4799061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4799128Z return func(*args, **kwargs) 2025-08-14T21:47:02.4799374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4799451Z return func(*args, **kwargs) 2025-08-14T21:47:02.4799729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4799823Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4800093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4800177Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4800536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:47:02.4800680Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:47:02.4800970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:47:02.4801057Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4801061Z 2025-08-14T21:47:02.4801170Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4801389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4801458Z return mod(**inputs) 2025-08-14T21:47:02.4801699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4801788Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4802070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4802150Z outputs = self.layoutlm( 2025-08-14T21:47:02.4802415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4802485Z return func(*args, **kwargs) 2025-08-14T21:47:02.4802741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4802811Z return func(*args, **kwargs) 2025-08-14T21:47:02.4803046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4803123Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4803402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4803493Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4803744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4803816Z return func(*args, **kwargs) 2025-08-14T21:47:02.4804080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4804151Z return func(*args, **kwargs) 2025-08-14T21:47:02.4804417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4804487Z return func(*args, **kwargs) 2025-08-14T21:47:02.4804569Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4804807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4804886Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4805200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4805286Z layer_outputs = layer_module( 2025-08-14T21:47:02.4805670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4805774Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4806027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4806100Z return func(*args, **kwargs) 2025-08-14T21:47:02.4806363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4806433Z return func(*args, **kwargs) 2025-08-14T21:47:02.4806692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4806775Z return func(*args, **kwargs) 2025-08-14T21:47:02.4807102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4807195Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4807426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4807491Z return func(*args, **kwargs) 2025-08-14T21:47:02.4807725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4807790Z return func(*args, **kwargs) 2025-08-14T21:47:02.4808018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4808090Z return func(*args, **kwargs) 2025-08-14T21:47:02.4808373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4808457Z self_outputs = self.self( 2025-08-14T21:47:02.4808694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4808760Z return func(*args, **kwargs) 2025-08-14T21:47:02.4809004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4809068Z return func(*args, **kwargs) 2025-08-14T21:47:02.4809301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4809367Z return func(*args, **kwargs) 2025-08-14T21:47:02.4809622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:47:02.4809771Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4809777Z 2025-08-14T21:47:02.4809880Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4810083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4810154Z return mod(**inputs) 2025-08-14T21:47:02.4810356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4810432Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4810683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4810750Z outputs = self.layoutlm( 2025-08-14T21:47:02.4810979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4811043Z return func(*args, **kwargs) 2025-08-14T21:47:02.4811270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4811356Z return func(*args, **kwargs) 2025-08-14T21:47:02.4811558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4811635Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4811884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4811953Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4812184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4812249Z return func(*args, **kwargs) 2025-08-14T21:47:02.4812483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4812550Z return func(*args, **kwargs) 2025-08-14T21:47:02.4812780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4812903Z return func(*args, **kwargs) 2025-08-14T21:47:02.4812978Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4813179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4813255Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4813505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4813581Z layer_outputs = layer_module( 2025-08-14T21:47:02.4813791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4813865Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4814114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4814184Z return func(*args, **kwargs) 2025-08-14T21:47:02.4814414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4814489Z return func(*args, **kwargs) 2025-08-14T21:47:02.4814718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4814791Z return func(*args, **kwargs) 2025-08-14T21:47:02.4815050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4815130Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4815368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4815435Z return func(*args, **kwargs) 2025-08-14T21:47:02.4815667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4815744Z return func(*args, **kwargs) 2025-08-14T21:47:02.4815978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4816050Z return func(*args, **kwargs) 2025-08-14T21:47:02.4816312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4816382Z self_outputs = self.self( 2025-08-14T21:47:02.4816632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4816698Z return func(*args, **kwargs) 2025-08-14T21:47:02.4816941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4817009Z return func(*args, **kwargs) 2025-08-14T21:47:02.4817257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4817358Z return func(*args, **kwargs) 2025-08-14T21:47:02.4817641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:47:02.4817787Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4817791Z 2025-08-14T21:47:02.4817909Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4818121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4818198Z return mod(**inputs) 2025-08-14T21:47:02.4818426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4818508Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4818802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4818912Z outputs = self.layoutlm( 2025-08-14T21:47:02.4819164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4819241Z return func(*args, **kwargs) 2025-08-14T21:47:02.4819476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4819550Z return func(*args, **kwargs) 2025-08-14T21:47:02.4819763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4819837Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4820106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4820199Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4820448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4820520Z return func(*args, **kwargs) 2025-08-14T21:47:02.4820754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4820828Z return func(*args, **kwargs) 2025-08-14T21:47:02.4821060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4821126Z return func(*args, **kwargs) 2025-08-14T21:47:02.4821213Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4821429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4821508Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4821774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4821849Z layer_outputs = layer_module( 2025-08-14T21:47:02.4822078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4822158Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4822399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4822472Z return func(*args, **kwargs) 2025-08-14T21:47:02.4822709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4822783Z return func(*args, **kwargs) 2025-08-14T21:47:02.4823021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4823088Z return func(*args, **kwargs) 2025-08-14T21:47:02.4823361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4823467Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4823704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4823779Z return func(*args, **kwargs) 2025-08-14T21:47:02.4824020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4824094Z return func(*args, **kwargs) 2025-08-14T21:47:02.4824327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4824391Z return func(*args, **kwargs) 2025-08-14T21:47:02.4824661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4824729Z self_outputs = self.self( 2025-08-14T21:47:02.4824984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4825065Z return func(*args, **kwargs) 2025-08-14T21:47:02.4825300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4825372Z return func(*args, **kwargs) 2025-08-14T21:47:02.4825610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4825677Z return func(*args, **kwargs) 2025-08-14T21:47:02.4825952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:47:02.4826095Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4826099Z 2025-08-14T21:47:02.4826208Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4826289Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4826394Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4826595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4826660Z return mod(**inputs) 2025-08-14T21:47:02.4826874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4826954Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4827214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4827291Z outputs = self.layoutlm( 2025-08-14T21:47:02.4827550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4827620Z return func(*args, **kwargs) 2025-08-14T21:47:02.4827876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4827961Z return func(*args, **kwargs) 2025-08-14T21:47:02.4828173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4828254Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4828516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4828597Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4828832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4828899Z return func(*args, **kwargs) 2025-08-14T21:47:02.4829139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4829206Z return func(*args, **kwargs) 2025-08-14T21:47:02.4829449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4829534Z return func(*args, **kwargs) 2025-08-14T21:47:02.4829611Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4829836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4829909Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4830175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4830253Z layer_outputs = layer_module( 2025-08-14T21:47:02.4830467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4830551Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4830787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4830874Z return func(*args, **kwargs) 2025-08-14T21:47:02.4831135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4831204Z return func(*args, **kwargs) 2025-08-14T21:47:02.4831442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4831519Z return func(*args, **kwargs) 2025-08-14T21:47:02.4831785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4831876Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4832154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4832237Z return func(*args, **kwargs) 2025-08-14T21:47:02.4832483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4832555Z return func(*args, **kwargs) 2025-08-14T21:47:02.4832789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4832863Z return func(*args, **kwargs) 2025-08-14T21:47:02.4833125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:47:02.4833263Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:47:02.4833523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:47:02.4833606Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4833609Z 2025-08-14T21:47:02.4833723Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4833921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4834000Z return mod(**inputs) 2025-08-14T21:47:02.4834212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4834285Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4834556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4834625Z outputs = self.layoutlm( 2025-08-14T21:47:02.4834859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4834932Z return func(*args, **kwargs) 2025-08-14T21:47:02.4835167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4835242Z return func(*args, **kwargs) 2025-08-14T21:47:02.4835452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4835549Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4835819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4835893Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4836129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4836203Z return func(*args, **kwargs) 2025-08-14T21:47:02.4836436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4836508Z return func(*args, **kwargs) 2025-08-14T21:47:02.4836740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4836807Z return func(*args, **kwargs) 2025-08-14T21:47:02.4836893Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4837144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4837224Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4837485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4837554Z layer_outputs = layer_module( 2025-08-14T21:47:02.4838111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4838221Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4838466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4838542Z return func(*args, **kwargs) 2025-08-14T21:47:02.4838833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4838917Z return func(*args, **kwargs) 2025-08-14T21:47:02.4839158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4839229Z return func(*args, **kwargs) 2025-08-14T21:47:02.4839517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4839607Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4839877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4839968Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4840285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4840420Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4840706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:47:02.4840793Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4840797Z 2025-08-14T21:47:02.4840917Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4841128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4841206Z return mod(**inputs) 2025-08-14T21:47:02.4841431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4841508Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4841796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4841872Z outputs = self.layoutlm( 2025-08-14T21:47:02.4842125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4842235Z return func(*args, **kwargs) 2025-08-14T21:47:02.4842485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4842563Z return func(*args, **kwargs) 2025-08-14T21:47:02.4842789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4842865Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4843150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4843226Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4843477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4843557Z return func(*args, **kwargs) 2025-08-14T21:47:02.4843857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4843935Z return func(*args, **kwargs) 2025-08-14T21:47:02.4844186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4844255Z return func(*args, **kwargs) 2025-08-14T21:47:02.4844342Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4844577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4844653Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4844943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4845018Z layer_outputs = layer_module( 2025-08-14T21:47:02.4845275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4845363Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4845686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4845772Z return func(*args, **kwargs) 2025-08-14T21:47:02.4846059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4846135Z return func(*args, **kwargs) 2025-08-14T21:47:02.4846396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4846466Z return func(*args, **kwargs) 2025-08-14T21:47:02.4846764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4846851Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4847113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4847199Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4847496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4847622Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4847887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:47:02.4848004Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:02.4848225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:47:02.4848297Z return self.act(input) 2025-08-14T21:47:02.4848303Z 2025-08-14T21:47:02.4848415Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4848642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4848712Z return mod(**inputs) 2025-08-14T21:47:02.4848932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4849006Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4849269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4849345Z outputs = self.layoutlm( 2025-08-14T21:47:02.4849584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4849660Z return func(*args, **kwargs) 2025-08-14T21:47:02.4849898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4849964Z return func(*args, **kwargs) 2025-08-14T21:47:02.4850229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4850305Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4850566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4850650Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4850947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4851019Z return func(*args, **kwargs) 2025-08-14T21:47:02.4851249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4851314Z return func(*args, **kwargs) 2025-08-14T21:47:02.4851564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4851632Z return func(*args, **kwargs) 2025-08-14T21:47:02.4851710Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4851924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4851996Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4852257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4852326Z layer_outputs = layer_module( 2025-08-14T21:47:02.4852536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4852622Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4852852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4852925Z return func(*args, **kwargs) 2025-08-14T21:47:02.4853159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4853226Z return func(*args, **kwargs) 2025-08-14T21:47:02.4853463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4853528Z return func(*args, **kwargs) 2025-08-14T21:47:02.4853784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4853873Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4854120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4854202Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4854495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:47:02.4854645Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:47:02.4854915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:47:02.4854992Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4854996Z 2025-08-14T21:47:02.4855103Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4855294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4855358Z return mod(**inputs) 2025-08-14T21:47:02.4855574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4855645Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4855902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4855981Z outputs = self.layoutlm( 2025-08-14T21:47:02.4856242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4856316Z return func(*args, **kwargs) 2025-08-14T21:47:02.4856544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4856608Z return func(*args, **kwargs) 2025-08-14T21:47:02.4856820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4856892Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4857152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4857231Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4857479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4857557Z return func(*args, **kwargs) 2025-08-14T21:47:02.4857794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4857860Z return func(*args, **kwargs) 2025-08-14T21:47:02.4858100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4858166Z return func(*args, **kwargs) 2025-08-14T21:47:02.4858242Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4858461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4858533Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4858812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4858880Z layer_outputs = layer_module( 2025-08-14T21:47:02.4859093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4859178Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4859409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4859481Z return func(*args, **kwargs) 2025-08-14T21:47:02.4859711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4859777Z return func(*args, **kwargs) 2025-08-14T21:47:02.4860010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4860074Z return func(*args, **kwargs) 2025-08-14T21:47:02.4860331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4860441Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4860671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4860742Z return func(*args, **kwargs) 2025-08-14T21:47:02.4860968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4861032Z return func(*args, **kwargs) 2025-08-14T21:47:02.4861267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4861330Z return func(*args, **kwargs) 2025-08-14T21:47:02.4861585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4861661Z self_outputs = self.self( 2025-08-14T21:47:02.4861889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4861980Z return func(*args, **kwargs) 2025-08-14T21:47:02.4862598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4862673Z return func(*args, **kwargs) 2025-08-14T21:47:02.4862913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4862979Z return func(*args, **kwargs) 2025-08-14T21:47:02.4863239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:47:02.4863392Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4863396Z 2025-08-14T21:47:02.4863496Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4863718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4863789Z return mod(**inputs) 2025-08-14T21:47:02.4864000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4864082Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4864340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4864418Z outputs = self.layoutlm( 2025-08-14T21:47:02.4864648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4864715Z return func(*args, **kwargs) 2025-08-14T21:47:02.4864950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4865015Z return func(*args, **kwargs) 2025-08-14T21:47:02.4865224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4865309Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4865564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4865644Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4865889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4865952Z return func(*args, **kwargs) 2025-08-14T21:47:02.4866183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4866247Z return func(*args, **kwargs) 2025-08-14T21:47:02.4866471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4866544Z return func(*args, **kwargs) 2025-08-14T21:47:02.4866617Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4866889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4866962Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4867219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4867296Z layer_outputs = layer_module( 2025-08-14T21:47:02.4867507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4867582Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4867821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4867888Z return func(*args, **kwargs) 2025-08-14T21:47:02.4868128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4868196Z return func(*args, **kwargs) 2025-08-14T21:47:02.4868472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4868549Z return func(*args, **kwargs) 2025-08-14T21:47:02.4868811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4868896Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4869125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4869190Z return func(*args, **kwargs) 2025-08-14T21:47:02.4869428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4869495Z return func(*args, **kwargs) 2025-08-14T21:47:02.4869749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4869830Z return func(*args, **kwargs) 2025-08-14T21:47:02.4870093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4870171Z self_outputs = self.self( 2025-08-14T21:47:02.4870403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4870470Z return func(*args, **kwargs) 2025-08-14T21:47:02.4870708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4870774Z return func(*args, **kwargs) 2025-08-14T21:47:02.4871015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4871087Z return func(*args, **kwargs) 2025-08-14T21:47:02.4871343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:47:02.4871487Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4871492Z 2025-08-14T21:47:02.4871591Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4871783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4871854Z return mod(**inputs) 2025-08-14T21:47:02.4872056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4872133Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4872386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4872454Z outputs = self.layoutlm( 2025-08-14T21:47:02.4872689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4872774Z return func(*args, **kwargs) 2025-08-14T21:47:02.4873005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4873078Z return func(*args, **kwargs) 2025-08-14T21:47:02.4873286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4873364Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4873622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4873693Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4873929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4873995Z return func(*args, **kwargs) 2025-08-14T21:47:02.4874251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4874338Z return func(*args, **kwargs) 2025-08-14T21:47:02.4874567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4874638Z return func(*args, **kwargs) 2025-08-14T21:47:02.4874713Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4874921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4874999Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4875250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4875318Z layer_outputs = layer_module( 2025-08-14T21:47:02.4875553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4875634Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4875875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4875939Z return func(*args, **kwargs) 2025-08-14T21:47:02.4876164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4876236Z return func(*args, **kwargs) 2025-08-14T21:47:02.4876458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4876529Z return func(*args, **kwargs) 2025-08-14T21:47:02.4876782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4876862Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4877097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4877168Z return func(*args, **kwargs) 2025-08-14T21:47:02.4877394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4877468Z return func(*args, **kwargs) 2025-08-14T21:47:02.4877697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4877769Z return func(*args, **kwargs) 2025-08-14T21:47:02.4878025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4878096Z self_outputs = self.self( 2025-08-14T21:47:02.4878336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4878405Z return func(*args, **kwargs) 2025-08-14T21:47:02.4878661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4878742Z return func(*args, **kwargs) 2025-08-14T21:47:02.4878987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4879064Z return func(*args, **kwargs) 2025-08-14T21:47:02.4879339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:47:02.4879490Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4879495Z 2025-08-14T21:47:02.4879585Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4879667Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4879783Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4879992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4880089Z return mod(**inputs) 2025-08-14T21:47:02.4880328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4880404Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4880676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4880751Z outputs = self.layoutlm( 2025-08-14T21:47:02.4880980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4881052Z return func(*args, **kwargs) 2025-08-14T21:47:02.4881282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4881350Z return func(*args, **kwargs) 2025-08-14T21:47:02.4881583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4881661Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4881924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4882004Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4882239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4882312Z return func(*args, **kwargs) 2025-08-14T21:47:02.4882545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4882610Z return func(*args, **kwargs) 2025-08-14T21:47:02.4882853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4882922Z return func(*args, **kwargs) 2025-08-14T21:47:02.4882998Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4883235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4883311Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4883600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4883676Z layer_outputs = layer_module( 2025-08-14T21:47:02.4883907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4883997Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4884246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4884324Z return func(*args, **kwargs) 2025-08-14T21:47:02.4884567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4884651Z return func(*args, **kwargs) 2025-08-14T21:47:02.4884899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4884970Z return func(*args, **kwargs) 2025-08-14T21:47:02.4885247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4885340Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4885678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4885764Z return func(*args, **kwargs) 2025-08-14T21:47:02.4886017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4886091Z return func(*args, **kwargs) 2025-08-14T21:47:02.4886347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4886459Z return func(*args, **kwargs) 2025-08-14T21:47:02.4886745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:47:02.4886889Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:47:02.4887168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:47:02.4887264Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4887269Z 2025-08-14T21:47:02.4887375Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4887586Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4887665Z return mod(**inputs) 2025-08-14T21:47:02.4887909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4887992Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4888284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4888357Z outputs = self.layoutlm( 2025-08-14T21:47:02.4888616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4888687Z return func(*args, **kwargs) 2025-08-14T21:47:02.4888934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4889013Z return func(*args, **kwargs) 2025-08-14T21:47:02.4889238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4889325Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4889607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4889688Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4889944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4890013Z return func(*args, **kwargs) 2025-08-14T21:47:02.4890259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4890335Z return func(*args, **kwargs) 2025-08-14T21:47:02.4890581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4890656Z return func(*args, **kwargs) 2025-08-14T21:47:02.4890735Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4890960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4891066Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4891348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4891424Z layer_outputs = layer_module( 2025-08-14T21:47:02.4891662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4891744Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4891995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4892066Z return func(*args, **kwargs) 2025-08-14T21:47:02.4892311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4892391Z return func(*args, **kwargs) 2025-08-14T21:47:02.4892636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4892727Z return func(*args, **kwargs) 2025-08-14T21:47:02.4893038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4893129Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4893407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4893488Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4893801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4893937Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4894228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:47:02.4894325Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4894330Z 2025-08-14T21:47:02.4894439Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4894647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4894725Z return mod(**inputs) 2025-08-14T21:47:02.4894951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4895027Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4895310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4895382Z outputs = self.layoutlm( 2025-08-14T21:47:02.4895639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4895711Z return func(*args, **kwargs) 2025-08-14T21:47:02.4895959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4896042Z return func(*args, **kwargs) 2025-08-14T21:47:02.4896266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4896349Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4896633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4896702Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4896932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4896995Z return func(*args, **kwargs) 2025-08-14T21:47:02.4897214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4897287Z return func(*args, **kwargs) 2025-08-14T21:47:02.4897554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4897631Z return func(*args, **kwargs) 2025-08-14T21:47:02.4897707Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4897919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4897998Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4898263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4898334Z layer_outputs = layer_module( 2025-08-14T21:47:02.4898565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4898647Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4898903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4899030Z return func(*args, **kwargs) 2025-08-14T21:47:02.4899278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4899354Z return func(*args, **kwargs) 2025-08-14T21:47:02.4899583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4899648Z return func(*args, **kwargs) 2025-08-14T21:47:02.4899914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4899996Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4900256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4900348Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4900640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4900764Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4901033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:47:02.4901145Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:02.4901343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:47:02.4901411Z return self.act(input) 2025-08-14T21:47:02.4901414Z 2025-08-14T21:47:02.4901517Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4901705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4901769Z return mod(**inputs) 2025-08-14T21:47:02.4901979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4902054Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4902312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4902379Z outputs = self.layoutlm( 2025-08-14T21:47:02.4902599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4902671Z return func(*args, **kwargs) 2025-08-14T21:47:02.4902892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4902954Z return func(*args, **kwargs) 2025-08-14T21:47:02.4903161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4903230Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4903506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4903576Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4903798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4903872Z return func(*args, **kwargs) 2025-08-14T21:47:02.4904093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4904161Z return func(*args, **kwargs) 2025-08-14T21:47:02.4904383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4904446Z return func(*args, **kwargs) 2025-08-14T21:47:02.4904527Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4904730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4904835Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4905092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4905158Z layer_outputs = layer_module( 2025-08-14T21:47:02.4905372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4905445Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4905664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4905734Z return func(*args, **kwargs) 2025-08-14T21:47:02.4905954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4906033Z return func(*args, **kwargs) 2025-08-14T21:47:02.4906266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4906333Z return func(*args, **kwargs) 2025-08-14T21:47:02.4906591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4906672Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4906922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4907006Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4907303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:47:02.4907445Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:47:02.4907712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:47:02.4907797Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4907801Z 2025-08-14T21:47:02.4907913Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4908113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4908178Z return mod(**inputs) 2025-08-14T21:47:02.4908399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4908472Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4908742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4908820Z outputs = self.layoutlm( 2025-08-14T21:47:02.4909051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4909141Z return func(*args, **kwargs) 2025-08-14T21:47:02.4909370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4909435Z return func(*args, **kwargs) 2025-08-14T21:47:02.4909657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4909726Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4909984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4910053Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4910279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4910350Z return func(*args, **kwargs) 2025-08-14T21:47:02.4910578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4910676Z return func(*args, **kwargs) 2025-08-14T21:47:02.4910926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4910992Z return func(*args, **kwargs) 2025-08-14T21:47:02.4911073Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4911283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4911353Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4911621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4911690Z layer_outputs = layer_module( 2025-08-14T21:47:02.4911910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4912003Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4912236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4912314Z return func(*args, **kwargs) 2025-08-14T21:47:02.4912544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4912609Z return func(*args, **kwargs) 2025-08-14T21:47:02.4912889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4912954Z return func(*args, **kwargs) 2025-08-14T21:47:02.4913220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4913300Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4913531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4913606Z return func(*args, **kwargs) 2025-08-14T21:47:02.4913839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4913905Z return func(*args, **kwargs) 2025-08-14T21:47:02.4914141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4914206Z return func(*args, **kwargs) 2025-08-14T21:47:02.4914469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4914538Z self_outputs = self.self( 2025-08-14T21:47:02.4914769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4914841Z return func(*args, **kwargs) 2025-08-14T21:47:02.4915072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4915163Z return func(*args, **kwargs) 2025-08-14T21:47:02.4915401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4915466Z return func(*args, **kwargs) 2025-08-14T21:47:02.4915737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 191, in forward 2025-08-14T21:47:02.4915877Z query_states = self.query(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4915881Z 2025-08-14T21:47:02.4915987Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4916200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4916265Z return mod(**inputs) 2025-08-14T21:47:02.4916492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4916566Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4916866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4916945Z outputs = self.layoutlm( 2025-08-14T21:47:02.4917183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4917251Z return func(*args, **kwargs) 2025-08-14T21:47:02.4917513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4917582Z return func(*args, **kwargs) 2025-08-14T21:47:02.4917809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4917886Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4918192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4918281Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4918532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4918610Z return func(*args, **kwargs) 2025-08-14T21:47:02.4918858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4918929Z return func(*args, **kwargs) 2025-08-14T21:47:02.4919182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4919253Z return func(*args, **kwargs) 2025-08-14T21:47:02.4919333Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4919565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4919642Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4919928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4920005Z layer_outputs = layer_module( 2025-08-14T21:47:02.4920233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4920321Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4920567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4920637Z return func(*args, **kwargs) 2025-08-14T21:47:02.4920892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4920963Z return func(*args, **kwargs) 2025-08-14T21:47:02.4921218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4921311Z return func(*args, **kwargs) 2025-08-14T21:47:02.4921602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4921700Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4921959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4922032Z return func(*args, **kwargs) 2025-08-14T21:47:02.4922312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4922381Z return func(*args, **kwargs) 2025-08-14T21:47:02.4922643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4922712Z return func(*args, **kwargs) 2025-08-14T21:47:02.4922998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4923125Z self_outputs = self.self( 2025-08-14T21:47:02.4923377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4923454Z return func(*args, **kwargs) 2025-08-14T21:47:02.4923700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4923770Z return func(*args, **kwargs) 2025-08-14T21:47:02.4924021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4924092Z return func(*args, **kwargs) 2025-08-14T21:47:02.4924371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 192, in forward 2025-08-14T21:47:02.4924542Z key_states = self.key(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4924548Z 2025-08-14T21:47:02.4924659Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4924878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4924948Z return mod(**inputs) 2025-08-14T21:47:02.4925178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4925262Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4925631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4925712Z outputs = self.layoutlm( 2025-08-14T21:47:02.4925971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4926042Z return func(*args, **kwargs) 2025-08-14T21:47:02.4926300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4926376Z return func(*args, **kwargs) 2025-08-14T21:47:02.4926600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4926687Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4926963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4927049Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4927293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4927365Z return func(*args, **kwargs) 2025-08-14T21:47:02.4927621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4927695Z return func(*args, **kwargs) 2025-08-14T21:47:02.4927940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4928045Z return func(*args, **kwargs) 2025-08-14T21:47:02.4928129Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4928363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4928440Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4928723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4928803Z layer_outputs = layer_module( 2025-08-14T21:47:02.4929036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4929116Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4929375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4929468Z return func(*args, **kwargs) 2025-08-14T21:47:02.4929740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4929813Z return func(*args, **kwargs) 2025-08-14T21:47:02.4930061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4930139Z return func(*args, **kwargs) 2025-08-14T21:47:02.4930419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4930506Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4930760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4930848Z return func(*args, **kwargs) 2025-08-14T21:47:02.4931102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4931177Z return func(*args, **kwargs) 2025-08-14T21:47:02.4931421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4931496Z return func(*args, **kwargs) 2025-08-14T21:47:02.4931774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 271, in forward 2025-08-14T21:47:02.4931853Z self_outputs = self.self( 2025-08-14T21:47:02.4932097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4932167Z return func(*args, **kwargs) 2025-08-14T21:47:02.4932419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4932489Z return func(*args, **kwargs) 2025-08-14T21:47:02.4932739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4932817Z return func(*args, **kwargs) 2025-08-14T21:47:02.4933097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 193, in forward 2025-08-14T21:47:02.4933255Z value_states = self.value(hidden_states).view(hidden_shape).transpose(1, 2) 2025-08-14T21:47:02.4933259Z 2025-08-14T21:47:02.4933342Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4933423Z cudagraph partition due to non gpu ops 2025-08-14T21:47:02.4933535Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4933748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4933817Z return mod(**inputs) 2025-08-14T21:47:02.4934053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4934152Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4934441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4934515Z outputs = self.layoutlm( 2025-08-14T21:47:02.4934765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4934843Z return func(*args, **kwargs) 2025-08-14T21:47:02.4935090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4935162Z return func(*args, **kwargs) 2025-08-14T21:47:02.4935394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4935472Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4935757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4935871Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4936120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4936199Z return func(*args, **kwargs) 2025-08-14T21:47:02.4936444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4936522Z return func(*args, **kwargs) 2025-08-14T21:47:02.4936769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4936841Z return func(*args, **kwargs) 2025-08-14T21:47:02.4936928Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4937172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4937251Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4937544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4937768Z layer_outputs = layer_module( 2025-08-14T21:47:02.4938185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4938274Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4938530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4938610Z return func(*args, **kwargs) 2025-08-14T21:47:02.4938859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4938930Z return func(*args, **kwargs) 2025-08-14T21:47:02.4939195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4939271Z return func(*args, **kwargs) 2025-08-14T21:47:02.4939568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 338, in forward 2025-08-14T21:47:02.4939657Z self_attention_outputs = self.attention( 2025-08-14T21:47:02.4939907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4939987Z return func(*args, **kwargs) 2025-08-14T21:47:02.4940236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4940307Z return func(*args, **kwargs) 2025-08-14T21:47:02.4940566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4940636Z return func(*args, **kwargs) 2025-08-14T21:47:02.4940927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 278, in forward 2025-08-14T21:47:02.4941116Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:47:02.4941397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 225, in forward 2025-08-14T21:47:02.4941492Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4941496Z 2025-08-14T21:47:02.4941606Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4941821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4941891Z return mod(**inputs) 2025-08-14T21:47:02.4942116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4942201Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4942483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4942586Z outputs = self.layoutlm( 2025-08-14T21:47:02.4942878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4942954Z return func(*args, **kwargs) 2025-08-14T21:47:02.4943211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4943283Z return func(*args, **kwargs) 2025-08-14T21:47:02.4943508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4943595Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4943879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4943975Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4944210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4944281Z return func(*args, **kwargs) 2025-08-14T21:47:02.4944512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4944578Z return func(*args, **kwargs) 2025-08-14T21:47:02.4944804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4944879Z return func(*args, **kwargs) 2025-08-14T21:47:02.4944954Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4945159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4945240Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4945495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4945571Z layer_outputs = layer_module( 2025-08-14T21:47:02.4945785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4945860Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4946095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4946159Z return func(*args, **kwargs) 2025-08-14T21:47:02.4946390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4946455Z return func(*args, **kwargs) 2025-08-14T21:47:02.4946678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4946751Z return func(*args, **kwargs) 2025-08-14T21:47:02.4947003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4947107Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4947364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4947436Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4947731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4947846Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4948102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 294, in forward 2025-08-14T21:47:02.4948190Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4948194Z 2025-08-14T21:47:02.4948297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4948498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4948601Z return mod(**inputs) 2025-08-14T21:47:02.4948817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4948897Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4949159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4949229Z outputs = self.layoutlm( 2025-08-14T21:47:02.4949475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4949541Z return func(*args, **kwargs) 2025-08-14T21:47:02.4949789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4949872Z return func(*args, **kwargs) 2025-08-14T21:47:02.4950087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4950172Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4950434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4950506Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4950756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4950821Z return func(*args, **kwargs) 2025-08-14T21:47:02.4951055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4951119Z return func(*args, **kwargs) 2025-08-14T21:47:02.4951347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4951421Z return func(*args, **kwargs) 2025-08-14T21:47:02.4951497Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4951706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4951785Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4952041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4952117Z layer_outputs = layer_module( 2025-08-14T21:47:02.4952331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4952406Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4952640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4952706Z return func(*args, **kwargs) 2025-08-14T21:47:02.4952941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4953026Z return func(*args, **kwargs) 2025-08-14T21:47:02.4953252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4953326Z return func(*args, **kwargs) 2025-08-14T21:47:02.4953582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4953663Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4953924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4953998Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4954294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 356, in feed_forward_chunk 2025-08-14T21:47:02.4954408Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:47:02.4954695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 295, in forward 2025-08-14T21:47:02.4954815Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:47:02.4955019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:47:02.4955087Z return self.act(input) 2025-08-14T21:47:02.4955099Z 2025-08-14T21:47:02.4955200Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4955398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4955469Z return mod(**inputs) 2025-08-14T21:47:02.4955679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4955766Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4956036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4956109Z outputs = self.layoutlm( 2025-08-14T21:47:02.4956350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4956417Z return func(*args, **kwargs) 2025-08-14T21:47:02.4956649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4956721Z return func(*args, **kwargs) 2025-08-14T21:47:02.4956928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4957001Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4957263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 645, in forward 2025-08-14T21:47:02.4957335Z encoder_outputs = self.encoder( 2025-08-14T21:47:02.4957577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4957643Z return func(*args, **kwargs) 2025-08-14T21:47:02.4957870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4957943Z return func(*args, **kwargs) 2025-08-14T21:47:02.4958168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4958231Z return func(*args, **kwargs) 2025-08-14T21:47:02.4958312Z [Previous line repeated 1 more time] 2025-08-14T21:47:02.4958516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4958596Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4958851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 397, in forward 2025-08-14T21:47:02.4958951Z layer_outputs = layer_module( 2025-08-14T21:47:02.4959171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:02.4959247Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:02.4959479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4959553Z return func(*args, **kwargs) 2025-08-14T21:47:02.4959782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4959854Z return func(*args, **kwargs) 2025-08-14T21:47:02.4960084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4960150Z return func(*args, **kwargs) 2025-08-14T21:47:02.4960414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 348, in forward 2025-08-14T21:47:02.4960532Z layer_output = apply_chunking_to_forward( 2025-08-14T21:47:02.4960792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:47:02.4960868Z return forward_fn(*input_tensors) 2025-08-14T21:47:02.4961161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 357, in feed_forward_chunk 2025-08-14T21:47:02.4961299Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:47:02.4961560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 308, in forward 2025-08-14T21:47:02.4961642Z hidden_states = self.dense(hidden_states) 2025-08-14T21:47:02.4961700Z 2025-08-14T21:47:02.4961814Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4962012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4962080Z return mod(**inputs) 2025-08-14T21:47:02.4962346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4962422Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4962700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4962771Z outputs = self.layoutlm( 2025-08-14T21:47:02.4963022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4963089Z return func(*args, **kwargs) 2025-08-14T21:47:02.4963333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4963408Z return func(*args, **kwargs) 2025-08-14T21:47:02.4963631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4963704Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4963980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 654, in forward 2025-08-14T21:47:02.4964071Z pooled_output = self.pooler(sequence_output) 2025-08-14T21:47:02.4964346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 430, in forward 2025-08-14T21:47:02.4964442Z pooled_output = self.dense(first_token_tensor) 2025-08-14T21:47:02.4964446Z 2025-08-14T21:47:02.4964547Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4964756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4964821Z return mod(**inputs) 2025-08-14T21:47:02.4965058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4965139Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4965467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 875, in forward 2025-08-14T21:47:02.4965558Z outputs = self.layoutlm( 2025-08-14T21:47:02.4965818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4965891Z return func(*args, **kwargs) 2025-08-14T21:47:02.4966154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:47:02.4966226Z return func(*args, **kwargs) 2025-08-14T21:47:02.4966470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4966548Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4966878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 654, in forward 2025-08-14T21:47:02.4966999Z pooled_output = self.pooler(sequence_output) 2025-08-14T21:47:02.4967261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 431, in forward 2025-08-14T21:47:02.4967359Z pooled_output = self.activation(pooled_output) 2025-08-14T21:47:02.4967373Z 2025-08-14T21:47:02.4967477Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4967675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4967750Z return mod(**inputs) 2025-08-14T21:47:02.4967965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4968056Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4968333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 891, in forward 2025-08-14T21:47:02.4968419Z logits = self.classifier(pooled_output) 2025-08-14T21:47:02.4968423Z 2025-08-14T21:47:02.4968531Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4968731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4968794Z return mod(**inputs) 2025-08-14T21:47:02.4969013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4969088Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4969349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 911, in forward 2025-08-14T21:47:02.4969495Z loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1)) 2025-08-14T21:47:02.4969499Z 2025-08-14T21:47:02.4969609Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4969813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4969879Z return mod(**inputs) 2025-08-14T21:47:02.4970095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4970178Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4970442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 911, in forward 2025-08-14T21:47:02.4970575Z loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1)) 2025-08-14T21:47:02.4970578Z 2025-08-14T21:47:02.4970679Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:02.4970879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:02.4970952Z return mod(**inputs) 2025-08-14T21:47:02.4971190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:47:02.4971265Z output = func(self, *args, **kwargs) 2025-08-14T21:47:02.4971536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/layoutlm/modeling_layoutlm.py", line 911, in forward 2025-08-14T21:47:02.4971659Z loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1)) 2025-08-14T21:47:02.4971662Z 2025-08-14T21:47:13.8454086Z Compilation time (from dynamo_timed): 18.334340768 2025-08-14T21:47:13.8454658Z pass 2025-08-14T21:47:13.8455153Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:47:13.8456122Z TIMING: _recursive_pre_grad_passes:0.01308 _recursive_joint_graph_passes:0.4377 _recursive_post_grad_passes:0.07972 async_compile.wait:0.69301 code_gen:7.14753 inductor_compile:8.2903 backend_compile:12.57648 gc:0.00184 entire_frame_compile:18.33434 total_wall_time:18.33434 2025-08-14T21:47:13.8457392Z STATS: call_* op count: 860 | FakeTensorMode.__torch_dispatch__:16781 | FakeTensor.__torch_dispatch__:4682 | ProxyTorchDispatchMode.__torch_dispatch__:5774 2025-08-14T21:47:13.8457867Z Dynamo produced 2 graphs covering 860 ops with 0 graph breaks (0 unique) 2025-08-14T21:47:19.3403670Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:47:19.3404638Z from pkg_resources import resource_filename 2025-08-14T21:47:20.0309885Z 2025-08-14T21:47:27.8163468Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:47:27.8163907Z loading model: 0it [00:07, ?it/s] 2025-08-14T21:47:27.8188951Z cpu eval M2M100ForConditionalGeneration 2025-08-14T21:47:28.7122343Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:47:29.0792387Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:47:29.4670953Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:47:45.8936696Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.8937851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.8938526Z return mod(**inputs) 2025-08-14T21:47:45.8939211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.8939912Z outputs = self.model( 2025-08-14T21:47:45.8940580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.8941243Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.8941920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 844, in forward 2025-08-14T21:47:45.8942744Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-08-14T21:47:45.8943469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-08-14T21:47:45.8944130Z return func(*args, **kwargs) 2025-08-14T21:47:45.8944852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-08-14T21:47:45.8945842Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-08-14T21:47:45.8946975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 80, in create_position_ids_from_input_ids 2025-08-14T21:47:45.8947806Z mask = input_ids.ne(padding_idx).int() 2025-08-14T21:47:45.8948497Z 2025-08-14T21:47:45.8948668Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.8949317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.8949923Z return mod(**inputs) 2025-08-14T21:47:45.8950621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.8951345Z outputs = self.model( 2025-08-14T21:47:45.8951959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:45.8952374Z decoder_outputs = self.decoder( 2025-08-14T21:47:45.8952782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1095, in forward 2025-08-14T21:47:45.8953381Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-08-14T21:47:45.8953958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-08-14T21:47:45.8954649Z return func(*args, **kwargs) 2025-08-14T21:47:45.8955362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-08-14T21:47:45.8955964Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-08-14T21:47:45.8956576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 80, in create_position_ids_from_input_ids 2025-08-14T21:47:45.8957050Z mask = input_ids.ne(padding_idx).int() 2025-08-14T21:47:45.8957209Z 2025-08-14T21:47:45.8957298Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.8957527Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.8957843Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.8958069Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.8958294Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.8958540Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.8958868Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.8959219Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.8959574Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.8959929Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.8960280Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.8960624Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.8961007Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.8961628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.8962193Z return mod(**inputs) 2025-08-14T21:47:45.8962837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.8963533Z outputs = self.model( 2025-08-14T21:47:45.8964208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.8964929Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.8965765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 844, in forward 2025-08-14T21:47:45.8966564Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-08-14T21:47:45.8967295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-08-14T21:47:45.8967930Z return func(*args, **kwargs) 2025-08-14T21:47:45.8968598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-08-14T21:47:45.8969555Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-08-14T21:47:45.8970692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-08-14T21:47:45.8971706Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-14T21:47:45.8972149Z 2025-08-14T21:47:45.8972343Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.8972977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.8973567Z return mod(**inputs) 2025-08-14T21:47:45.8974199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.8974899Z outputs = self.model( 2025-08-14T21:47:45.8975534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.8976221Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.8976950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 844, in forward 2025-08-14T21:47:45.8977710Z embed_pos = self.embed_positions(input_ids, inputs_embeds) 2025-08-14T21:47:45.8978417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-08-14T21:47:45.8979054Z return func(*args, **kwargs) 2025-08-14T21:47:45.8979686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-08-14T21:47:45.8980665Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-08-14T21:47:45.8981821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-08-14T21:47:45.8982807Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-14T21:47:45.8983237Z 2025-08-14T21:47:45.8983398Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.8984038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.8984560Z return mod(**inputs) 2025-08-14T21:47:45.8985156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.8985808Z outputs = self.model( 2025-08-14T21:47:45.8986390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.8987054Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.8987684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.8988382Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.8989013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.8989665Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.8990326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.8991046Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.8991740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:45.8992511Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:45.8992867Z 2025-08-14T21:47:45.8993033Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.8993633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.8994196Z return mod(**inputs) 2025-08-14T21:47:45.8994869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.8995533Z outputs = self.model( 2025-08-14T21:47:45.8996154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.8996840Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.8997548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.8998274Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.8998926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.8999591Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9000321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9001049Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9001844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:45.9002523Z key_states = self.k_proj(current_states) 2025-08-14T21:47:45.9002760Z 2025-08-14T21:47:45.9002935Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9003563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9004124Z return mod(**inputs) 2025-08-14T21:47:45.9004771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9005536Z outputs = self.model( 2025-08-14T21:47:45.9006222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9006896Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9007581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9008248Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9008857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9009484Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9010191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9010858Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9011509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:45.9012156Z value_states = self.v_proj(current_states) 2025-08-14T21:47:45.9012396Z 2025-08-14T21:47:45.9012971Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9013311Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9013663Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9014017Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9014404Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9015011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9015538Z return mod(**inputs) 2025-08-14T21:47:45.9016132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9016793Z outputs = self.model( 2025-08-14T21:47:45.9017463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9018189Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9018913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9019645Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9020219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9020856Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9021549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9022276Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9022947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9023648Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9024508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:45.9025170Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:45.9025417Z 2025-08-14T21:47:45.9025554Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9025957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9026312Z return mod(**inputs) 2025-08-14T21:47:45.9026709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9027130Z outputs = self.model( 2025-08-14T21:47:45.9027527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9027943Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9028360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9028811Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9029335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9029976Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9030728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9031224Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9031651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9032099Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9032582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:45.9033146Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:45.9033438Z 2025-08-14T21:47:45.9033611Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9034331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9034926Z return mod(**inputs) 2025-08-14T21:47:45.9035615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9036348Z outputs = self.model( 2025-08-14T21:47:45.9037030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9037458Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9038016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9038446Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9038994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9039721Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9040378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9041066Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9041784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:45.9042461Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:45.9042691Z 2025-08-14T21:47:45.9042871Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9043525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9044122Z return mod(**inputs) 2025-08-14T21:47:45.9044805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9045632Z outputs = self.model( 2025-08-14T21:47:45.9046480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9047191Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9047810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9048447Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9049032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9049618Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9050247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9050965Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9051283Z 2025-08-14T21:47:45.9051464Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9052044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9052587Z return mod(**inputs) 2025-08-14T21:47:45.9053233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9053929Z outputs = self.model( 2025-08-14T21:47:45.9054569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9055260Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9055911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9056542Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9057106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9057711Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9058398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9059131Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9059416Z 2025-08-14T21:47:45.9059582Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9060181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9060709Z return mod(**inputs) 2025-08-14T21:47:45.9061285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9061912Z outputs = self.model( 2025-08-14T21:47:45.9062542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9063206Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9063885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9064514Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9065087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9065715Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9066417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:47:45.9067169Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:45.9067399Z 2025-08-14T21:47:45.9067545Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9068131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9068692Z return mod(**inputs) 2025-08-14T21:47:45.9069354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9070062Z outputs = self.model( 2025-08-14T21:47:45.9070714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9071380Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9072045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9072733Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9073349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9074027Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9074763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9075449Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9076150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:45.9076914Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:45.9077268Z 2025-08-14T21:47:45.9077433Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9078048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9078583Z return mod(**inputs) 2025-08-14T21:47:45.9079242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9079916Z outputs = self.model( 2025-08-14T21:47:45.9080555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9081232Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9081870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9082523Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9083129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9083758Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9084467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9085213Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9086048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:45.9086796Z key_states = self.k_proj(current_states) 2025-08-14T21:47:45.9087038Z 2025-08-14T21:47:45.9087201Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9087849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9088386Z return mod(**inputs) 2025-08-14T21:47:45.9089007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9089688Z outputs = self.model( 2025-08-14T21:47:45.9090333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9091036Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9091716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9092387Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9092986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9093549Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9094066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9094478Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9094880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:45.9095270Z value_states = self.v_proj(current_states) 2025-08-14T21:47:45.9095419Z 2025-08-14T21:47:45.9095503Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9095723Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9095964Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9096261Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9096620Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9097289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9097892Z return mod(**inputs) 2025-08-14T21:47:45.9098568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9099258Z outputs = self.model( 2025-08-14T21:47:45.9099915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9100620Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9101304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9101979Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9102558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9103182Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9103855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9104587Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9105300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9106003Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9106750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:45.9107613Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:45.9107946Z 2025-08-14T21:47:45.9108104Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9108726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9109286Z return mod(**inputs) 2025-08-14T21:47:45.9109892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9110591Z outputs = self.model( 2025-08-14T21:47:45.9111205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9111761Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9112279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9112883Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9113407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9113948Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9114533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9115117Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9115772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9116380Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9117010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:45.9117663Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:45.9117885Z 2025-08-14T21:47:45.9118029Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9118525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9119027Z return mod(**inputs) 2025-08-14T21:47:45.9119684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9120349Z outputs = self.model( 2025-08-14T21:47:45.9120967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9121625Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9122274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9122932Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9123537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9124175Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9124877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9125745Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9126487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:45.9127170Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:45.9127382Z 2025-08-14T21:47:45.9127523Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9128052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9128528Z return mod(**inputs) 2025-08-14T21:47:45.9129040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9129597Z outputs = self.model( 2025-08-14T21:47:45.9130134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9130737Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9131310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9131932Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9132459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9133022Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9133600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9134265Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9134526Z 2025-08-14T21:47:45.9134677Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9135218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9135714Z return mod(**inputs) 2025-08-14T21:47:45.9136288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9136888Z outputs = self.model( 2025-08-14T21:47:45.9137514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9138292Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9138888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9139477Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9140007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9140553Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9141175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9141855Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9142210Z 2025-08-14T21:47:45.9142366Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9142941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9143449Z return mod(**inputs) 2025-08-14T21:47:45.9144022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9144644Z outputs = self.model( 2025-08-14T21:47:45.9145211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9145826Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9146443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9147067Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9147609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9148160Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9148720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:47:45.9149309Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:45.9149505Z 2025-08-14T21:47:45.9149655Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9150175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9150649Z return mod(**inputs) 2025-08-14T21:47:45.9151183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9151755Z outputs = self.model( 2025-08-14T21:47:45.9152342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9152981Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9153690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9154303Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9154862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9155437Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9156063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-08-14T21:47:45.9156701Z hidden_states = residual + hidden_states 2025-08-14T21:47:45.9156923Z 2025-08-14T21:47:45.9157080Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9157670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9158190Z return mod(**inputs) 2025-08-14T21:47:45.9158784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9159511Z outputs = self.model( 2025-08-14T21:47:45.9160118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9160774Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9161447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9162100Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9162667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9163280Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9163995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9164682Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9165439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:45.9166216Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:45.9166568Z 2025-08-14T21:47:45.9166722Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9167278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9167765Z return mod(**inputs) 2025-08-14T21:47:45.9168323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9168913Z outputs = self.model( 2025-08-14T21:47:45.9169451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9170040Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9170613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9171192Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9171704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9172252Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9172867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9173524Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9174185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:45.9174824Z key_states = self.k_proj(current_states) 2025-08-14T21:47:45.9175043Z 2025-08-14T21:47:45.9175200Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9175880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9176411Z return mod(**inputs) 2025-08-14T21:47:45.9177023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9177677Z outputs = self.model( 2025-08-14T21:47:45.9178249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9178835Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9179415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9180001Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9180518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9181061Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9181713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9182302Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9182905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:45.9183509Z value_states = self.v_proj(current_states) 2025-08-14T21:47:45.9183714Z 2025-08-14T21:47:45.9183839Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9184162Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9184493Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9184816Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9185191Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9185837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9186371Z return mod(**inputs) 2025-08-14T21:47:45.9186988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9187654Z outputs = self.model( 2025-08-14T21:47:45.9188275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9188897Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9189501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9190112Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9190655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9191235Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9191887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9192583Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9193292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9194023Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9194827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:45.9195629Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:45.9195932Z 2025-08-14T21:47:45.9196110Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9196735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9197295Z return mod(**inputs) 2025-08-14T21:47:45.9197957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9198715Z outputs = self.model( 2025-08-14T21:47:45.9199351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9200029Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9200719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9201402Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9202017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9202663Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9203428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9204206Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9205068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9205964Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9206843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:45.9207735Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:45.9208023Z 2025-08-14T21:47:45.9208204Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9208830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9209397Z return mod(**inputs) 2025-08-14T21:47:45.9210093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9210803Z outputs = self.model( 2025-08-14T21:47:45.9211474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9212167Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9212862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9213558Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9214172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9214801Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9215505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9216230Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9216946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:45.9217651Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:45.9217892Z 2025-08-14T21:47:45.9218077Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9218709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9219271Z return mod(**inputs) 2025-08-14T21:47:45.9219904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9220576Z outputs = self.model( 2025-08-14T21:47:45.9221223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9221924Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9222607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9223399Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9224025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9224679Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9225371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9226116Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9226430Z 2025-08-14T21:47:45.9226613Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9227242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9227804Z return mod(**inputs) 2025-08-14T21:47:45.9228431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9229100Z outputs = self.model( 2025-08-14T21:47:45.9253879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9254640Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9255328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9256027Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9256645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9257281Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9257964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9258723Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9259088Z 2025-08-14T21:47:45.9259274Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9259916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9260604Z return mod(**inputs) 2025-08-14T21:47:45.9261153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9261726Z outputs = self.model( 2025-08-14T21:47:45.9262318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9262986Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9263645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9264312Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9264915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9265538Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9266226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:47:45.9266878Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:45.9267112Z 2025-08-14T21:47:45.9267276Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9267879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9268400Z return mod(**inputs) 2025-08-14T21:47:45.9269026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9269683Z outputs = self.model( 2025-08-14T21:47:45.9270313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9270970Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9272562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9273295Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9273966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9274594Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9275270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9275963Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9276669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:45.9277474Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:45.9277825Z 2025-08-14T21:47:45.9278009Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9278734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9279281Z return mod(**inputs) 2025-08-14T21:47:45.9279913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9280580Z outputs = self.model( 2025-08-14T21:47:45.9281217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9281904Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9282600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9283289Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9283951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9284605Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9285419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9286185Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9286935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:45.9287656Z key_states = self.k_proj(current_states) 2025-08-14T21:47:45.9287894Z 2025-08-14T21:47:45.9288074Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9288721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9289305Z return mod(**inputs) 2025-08-14T21:47:45.9289960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9290649Z outputs = self.model( 2025-08-14T21:47:45.9291312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9292024Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9292718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9293421Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9294052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9294708Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9295388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9296113Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9296842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:45.9297586Z value_states = self.v_proj(current_states) 2025-08-14T21:47:45.9297847Z 2025-08-14T21:47:45.9297978Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9298333Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9298683Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9299013Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9299407Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9300029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9300577Z return mod(**inputs) 2025-08-14T21:47:45.9301224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9301900Z outputs = self.model( 2025-08-14T21:47:45.9302558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9303316Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9303999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9304689Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9305296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9305944Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9306640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9307332Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9308037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9308737Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9309518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:45.9310354Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:45.9310663Z 2025-08-14T21:47:45.9310833Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9311462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9312028Z return mod(**inputs) 2025-08-14T21:47:45.9312663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9313351Z outputs = self.model( 2025-08-14T21:47:45.9313994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9314642Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9315271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9315909Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9316480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9317071Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9317705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9318375Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9319049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9319730Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9320494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:45.9321326Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:45.9321602Z 2025-08-14T21:47:45.9321783Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9322396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9322948Z return mod(**inputs) 2025-08-14T21:47:45.9323570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9324224Z outputs = self.model( 2025-08-14T21:47:45.9324841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9325632Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9326376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9327137Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9327765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9328380Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9329046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9329740Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9330432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:45.9331106Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:45.9331337Z 2025-08-14T21:47:45.9331504Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9332169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9332729Z return mod(**inputs) 2025-08-14T21:47:45.9333373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9334046Z outputs = self.model( 2025-08-14T21:47:45.9334695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9335379Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9336032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9336681Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9337270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9338129Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9338792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9339534Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9339833Z 2025-08-14T21:47:45.9339993Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9340611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9341144Z return mod(**inputs) 2025-08-14T21:47:45.9341759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9342416Z outputs = self.model( 2025-08-14T21:47:45.9343038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9343707Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9344377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9345156Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9345744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9346371Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9347040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9347791Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9348089Z 2025-08-14T21:47:45.9348262Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9348883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9349443Z return mod(**inputs) 2025-08-14T21:47:45.9350060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9350733Z outputs = self.model( 2025-08-14T21:47:45.9351476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9352176Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9352840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9353569Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9354176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9354807Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9355495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:47:45.9356195Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:45.9356468Z 2025-08-14T21:47:45.9356654Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9357284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9357842Z return mod(**inputs) 2025-08-14T21:47:45.9358458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9359134Z outputs = self.model( 2025-08-14T21:47:45.9359761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9360451Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9361136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9361803Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9362412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9363049Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9363736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-08-14T21:47:45.9364440Z hidden_states = residual + hidden_states 2025-08-14T21:47:45.9364690Z 2025-08-14T21:47:45.9364864Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9365622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9366239Z return mod(**inputs) 2025-08-14T21:47:45.9366896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9367611Z outputs = self.model( 2025-08-14T21:47:45.9368280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9368968Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9369734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9370444Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9371058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9371704Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9372414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9373154Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9373878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:45.9374710Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:45.9375087Z 2025-08-14T21:47:45.9375274Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9376007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9376577Z return mod(**inputs) 2025-08-14T21:47:45.9377211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9377870Z outputs = self.model( 2025-08-14T21:47:45.9378491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9379168Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9379835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9380494Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9381110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9381738Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9382421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9383102Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9383821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:45.9384519Z key_states = self.k_proj(current_states) 2025-08-14T21:47:45.9384746Z 2025-08-14T21:47:45.9384925Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9385544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9386093Z return mod(**inputs) 2025-08-14T21:47:45.9386725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9387384Z outputs = self.model( 2025-08-14T21:47:45.9388008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9388686Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9389344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9390005Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9390596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9391213Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9391891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9392582Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9393322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:45.9394060Z value_states = self.v_proj(current_states) 2025-08-14T21:47:45.9394293Z 2025-08-14T21:47:45.9394438Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9394797Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9395146Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9395493Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9395892Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9396538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9397113Z return mod(**inputs) 2025-08-14T21:47:45.9397757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9398458Z outputs = self.model( 2025-08-14T21:47:45.9399120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9399876Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9400590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9401286Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9401907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9402518Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9403213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9403929Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9404667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9405488Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9406317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:45.9407199Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:45.9407559Z 2025-08-14T21:47:45.9407736Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9408349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9408905Z return mod(**inputs) 2025-08-14T21:47:45.9409535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9410189Z outputs = self.model( 2025-08-14T21:47:45.9410813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9411485Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9412145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9412797Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9413381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9414002Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9414664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9415343Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9416031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9416735Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9417497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:45.9418348Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:45.9418629Z 2025-08-14T21:47:45.9418791Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9419391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9419926Z return mod(**inputs) 2025-08-14T21:47:45.9420548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9421201Z outputs = self.model( 2025-08-14T21:47:45.9421808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9422453Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9423123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9423821Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9424493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9425122Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9425783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9426451Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9427120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:45.9427783Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:45.9427998Z 2025-08-14T21:47:45.9428167Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9428797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9429352Z return mod(**inputs) 2025-08-14T21:47:45.9429981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9430658Z outputs = self.model( 2025-08-14T21:47:45.9431276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9431953Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9432614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9433303Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9433895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9434520Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9435211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9435967Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9436271Z 2025-08-14T21:47:45.9436450Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9437074Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9437786Z return mod(**inputs) 2025-08-14T21:47:45.9438422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9439100Z outputs = self.model( 2025-08-14T21:47:45.9439756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9440445Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9441133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9441909Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9442528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9443170Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9443880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9444655Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9444962Z 2025-08-14T21:47:45.9445149Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9445849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9446460Z return mod(**inputs) 2025-08-14T21:47:45.9447125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9447789Z outputs = self.model( 2025-08-14T21:47:45.9448540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9449222Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9449904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9450576Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9451188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9451812Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9452489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:47:45.9453185Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:45.9453425Z 2025-08-14T21:47:45.9453635Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9454253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9454803Z return mod(**inputs) 2025-08-14T21:47:45.9455424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9456083Z outputs = self.model( 2025-08-14T21:47:45.9456706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9457377Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9458035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9458705Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9459291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9459908Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9460582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9461277Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9461951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:45.9462763Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:45.9463114Z 2025-08-14T21:47:45.9463292Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9463903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9464446Z return mod(**inputs) 2025-08-14T21:47:45.9465070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9465781Z outputs = self.model( 2025-08-14T21:47:45.9466398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9467057Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9467722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9468394Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9468979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9469570Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9470221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9470914Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9471611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:45.9472377Z key_states = self.k_proj(current_states) 2025-08-14T21:47:45.9472629Z 2025-08-14T21:47:45.9472794Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9473411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9473961Z return mod(**inputs) 2025-08-14T21:47:45.9474555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9475207Z outputs = self.model( 2025-08-14T21:47:45.9475809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9476442Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9477137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9477789Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9478350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9478933Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9479578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9480272Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9480951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:45.9481641Z value_states = self.v_proj(current_states) 2025-08-14T21:47:45.9481881Z 2025-08-14T21:47:45.9482003Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9482344Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9482678Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9483012Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9483394Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9484001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9484550Z return mod(**inputs) 2025-08-14T21:47:45.9485175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9486007Z outputs = self.model( 2025-08-14T21:47:45.9486689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9487392Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9488067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9488741Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9489351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9490021Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9490702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9491374Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9492072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9492765Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9493528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:45.9494374Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:45.9494696Z 2025-08-14T21:47:45.9494869Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9495524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9496097Z return mod(**inputs) 2025-08-14T21:47:45.9496724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9497379Z outputs = self.model( 2025-08-14T21:47:45.9498011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9498661Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9499308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9499971Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9500583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9501198Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9501883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9502565Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9503254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9503968Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9504749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:45.9505546Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:45.9505817Z 2025-08-14T21:47:45.9505995Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9506599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9507139Z return mod(**inputs) 2025-08-14T21:47:45.9507748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9508391Z outputs = self.model( 2025-08-14T21:47:45.9508999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9509652Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9510283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9510933Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9511508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9512120Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9512783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9513575Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9514298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:45.9514969Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:45.9515210Z 2025-08-14T21:47:45.9515378Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9515999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9516532Z return mod(**inputs) 2025-08-14T21:47:45.9517135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9517785Z outputs = self.model( 2025-08-14T21:47:45.9518393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9519098Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9519780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9520449Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9521056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9521663Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9522337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9523079Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9523382Z 2025-08-14T21:47:45.9523560Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9524216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9524785Z return mod(**inputs) 2025-08-14T21:47:45.9525504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9526184Z outputs = self.model( 2025-08-14T21:47:45.9526831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9527502Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9528169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9528824Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9529416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9530020Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9530688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9531441Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9531738Z 2025-08-14T21:47:45.9531906Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9532519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9533060Z return mod(**inputs) 2025-08-14T21:47:45.9533715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9534411Z outputs = self.model( 2025-08-14T21:47:45.9535071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9535756Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9536437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9537190Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9537949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9538584Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9539259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:47:45.9539955Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:45.9540183Z 2025-08-14T21:47:45.9540360Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9540984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9541543Z return mod(**inputs) 2025-08-14T21:47:45.9542184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9542844Z outputs = self.model( 2025-08-14T21:47:45.9543622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9544304Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9544971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9545658Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9546266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9546906Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9547588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-08-14T21:47:45.9548287Z hidden_states = residual + hidden_states 2025-08-14T21:47:45.9548520Z 2025-08-14T21:47:45.9548756Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9549388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9549957Z return mod(**inputs) 2025-08-14T21:47:45.9550604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9551277Z outputs = self.model( 2025-08-14T21:47:45.9551911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9552603Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9553287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9553971Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9554559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9555170Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9555832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9556494Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9557164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:45.9557938Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:45.9558283Z 2025-08-14T21:47:45.9558453Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9559036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9559572Z return mod(**inputs) 2025-08-14T21:47:45.9560187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9560889Z outputs = self.model( 2025-08-14T21:47:45.9561514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9562188Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9562854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9563516Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9564119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9564755Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9565534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9566316Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9567086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:45.9567893Z key_states = self.k_proj(current_states) 2025-08-14T21:47:45.9568148Z 2025-08-14T21:47:45.9568333Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9569040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9569661Z return mod(**inputs) 2025-08-14T21:47:45.9570354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9571096Z outputs = self.model( 2025-08-14T21:47:45.9571795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9572550Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9573344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9574105Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9574785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9575424Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9576091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9576787Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9577498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:45.9578207Z value_states = self.v_proj(current_states) 2025-08-14T21:47:45.9578452Z 2025-08-14T21:47:45.9578593Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9578958Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9579318Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9579650Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9580042Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9580669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9581227Z return mod(**inputs) 2025-08-14T21:47:45.9581865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9582539Z outputs = self.model( 2025-08-14T21:47:45.9583191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9583895Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9584593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9585265Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9585879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9586541Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9587221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9587959Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9588661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9589386Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9590173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:45.9590991Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:45.9591293Z 2025-08-14T21:47:45.9591462Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9592132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9592737Z return mod(**inputs) 2025-08-14T21:47:45.9593413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9594103Z outputs = self.model( 2025-08-14T21:47:45.9594735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9595428Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9596108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9596804Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9597460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9598108Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9598809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9599550Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9600284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9601023Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9601833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:45.9602657Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:45.9602942Z 2025-08-14T21:47:45.9603128Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9603773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9604353Z return mod(**inputs) 2025-08-14T21:47:45.9604993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9605797Z outputs = self.model( 2025-08-14T21:47:45.9606475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9607179Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9607862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9608509Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9609092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9609695Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9610367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9611138Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9611842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:45.9612530Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:45.9612758Z 2025-08-14T21:47:45.9612941Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9613554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9614116Z return mod(**inputs) 2025-08-14T21:47:45.9614749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9615408Z outputs = self.model( 2025-08-14T21:47:45.9616050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9616728Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9617450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9618117Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9618725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9619353Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9620005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9620731Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9621025Z 2025-08-14T21:47:45.9621183Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9621818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9622359Z return mod(**inputs) 2025-08-14T21:47:45.9622973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9623639Z outputs = self.model( 2025-08-14T21:47:45.9624270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9624931Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9625591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9626264Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9626851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9627459Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9628138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9628884Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9629170Z 2025-08-14T21:47:45.9629336Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9629943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9630491Z return mod(**inputs) 2025-08-14T21:47:45.9631108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9631750Z outputs = self.model( 2025-08-14T21:47:45.9632377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9633085Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9633772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9634485Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9635095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9635707Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9636379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:47:45.9637064Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:45.9637294Z 2025-08-14T21:47:45.9637478Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9638375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9638907Z return mod(**inputs) 2025-08-14T21:47:45.9639538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9640216Z outputs = self.model( 2025-08-14T21:47:45.9640928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9641660Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9642327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9643014Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9643636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9644288Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9644984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9645802Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9646648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:45.9647493Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:45.9647843Z 2025-08-14T21:47:45.9648017Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9648612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9649164Z return mod(**inputs) 2025-08-14T21:47:45.9649799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9650470Z outputs = self.model( 2025-08-14T21:47:45.9651099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9651766Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9652418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9653080Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9653677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9654302Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9654968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9655668Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9656351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:45.9657018Z key_states = self.k_proj(current_states) 2025-08-14T21:47:45.9657244Z 2025-08-14T21:47:45.9657413Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9658021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9658630Z return mod(**inputs) 2025-08-14T21:47:45.9659267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9659917Z outputs = self.model( 2025-08-14T21:47:45.9660549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9661223Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9661879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9662532Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9663104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9663715Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9664359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9665095Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9665784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:45.9666486Z value_states = self.v_proj(current_states) 2025-08-14T21:47:45.9666726Z 2025-08-14T21:47:45.9666859Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9667219Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9667566Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9667892Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9668279Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9668907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9669476Z return mod(**inputs) 2025-08-14T21:47:45.9670157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9670839Z outputs = self.model( 2025-08-14T21:47:45.9671492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9672171Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9672862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9673573Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9674176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9674791Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9675477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9676195Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9676910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9677644Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9678440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:45.9679298Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:45.9679618Z 2025-08-14T21:47:45.9679796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9680442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9681024Z return mod(**inputs) 2025-08-14T21:47:45.9681666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9682342Z outputs = self.model( 2025-08-14T21:47:45.9683036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9683745Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9684430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9685154Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9685900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9686589Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9687312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9688049Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9688788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9689573Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9690417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:45.9691266Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:45.9691556Z 2025-08-14T21:47:45.9691746Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9692401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9692978Z return mod(**inputs) 2025-08-14T21:47:45.9693646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9694346Z outputs = self.model( 2025-08-14T21:47:45.9695040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9695757Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9696448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9697102Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9697695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9698317Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9698993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9699689Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9700384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:45.9701065Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:45.9701297Z 2025-08-14T21:47:45.9701477Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9702085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9702645Z return mod(**inputs) 2025-08-14T21:47:45.9703284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9703956Z outputs = self.model( 2025-08-14T21:47:45.9704593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9705290Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9705958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9706612Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9707203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9707867Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9708541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9709282Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9709585Z 2025-08-14T21:47:45.9709754Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9710382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9710921Z return mod(**inputs) 2025-08-14T21:47:45.9711534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9712209Z outputs = self.model( 2025-08-14T21:47:45.9712822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9713519Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9714215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9714885Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9715445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9716040Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9716688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9717405Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9717687Z 2025-08-14T21:47:45.9717846Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9718481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9719025Z return mod(**inputs) 2025-08-14T21:47:45.9719642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9720289Z outputs = self.model( 2025-08-14T21:47:45.9720912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9721605Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9722241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9722914Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9723506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9724080Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9724728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:47:45.9725520Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:45.9725753Z 2025-08-14T21:47:45.9725945Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9726610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9727206Z return mod(**inputs) 2025-08-14T21:47:45.9727846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9728544Z outputs = self.model( 2025-08-14T21:47:45.9729242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9729990Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9730737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9731536Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9732200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9732896Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9733643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-08-14T21:47:45.9734349Z hidden_states = residual + hidden_states 2025-08-14T21:47:45.9734601Z 2025-08-14T21:47:45.9734779Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9735425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9736002Z return mod(**inputs) 2025-08-14T21:47:45.9736667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9737383Z outputs = self.model( 2025-08-14T21:47:45.9738284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9739026Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9739720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9740423Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9741057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9741682Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9742374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9743094Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9743831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:45.9744632Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:45.9744991Z 2025-08-14T21:47:45.9745160Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9745781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9746343Z return mod(**inputs) 2025-08-14T21:47:45.9747011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9747678Z outputs = self.model( 2025-08-14T21:47:45.9748307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9748963Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9749629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9750306Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9750905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9751523Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9752193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9752901Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9753603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:45.9754284Z key_states = self.k_proj(current_states) 2025-08-14T21:47:45.9754518Z 2025-08-14T21:47:45.9754686Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9755311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9755898Z return mod(**inputs) 2025-08-14T21:47:45.9756531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9757182Z outputs = self.model( 2025-08-14T21:47:45.9757803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9758456Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9759099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9759748Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9760311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9760926Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9761604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9762359Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9763045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:45.9763725Z value_states = self.v_proj(current_states) 2025-08-14T21:47:45.9763966Z 2025-08-14T21:47:45.9764102Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9764440Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9764791Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9765138Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9765624Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9766275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9766862Z return mod(**inputs) 2025-08-14T21:47:45.9767527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9768184Z outputs = self.model( 2025-08-14T21:47:45.9768815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9769479Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9770141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9770796Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9771384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9771984Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9772634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9773330Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9774037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9774766Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9775555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:45.9776381Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:45.9776684Z 2025-08-14T21:47:45.9776863Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9777482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9778031Z return mod(**inputs) 2025-08-14T21:47:45.9778680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9779344Z outputs = self.model( 2025-08-14T21:47:45.9780020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9780704Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9781378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9782051Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9782657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9783297Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9783983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9784666Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9785355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9786092Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9786927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:45.9787749Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:45.9788040Z 2025-08-14T21:47:45.9788213Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9788846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9789410Z return mod(**inputs) 2025-08-14T21:47:45.9790027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9790703Z outputs = self.model( 2025-08-14T21:47:45.9791354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9792022Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9792680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9793342Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9793937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9794543Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9795210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9795894Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9796597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:45.9797281Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:45.9797515Z 2025-08-14T21:47:45.9797682Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9798297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9798840Z return mod(**inputs) 2025-08-14T21:47:45.9799473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9800132Z outputs = self.model( 2025-08-14T21:47:45.9800752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9801414Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9802067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9802717Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9803311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9803973Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9804676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9805548Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9805869Z 2025-08-14T21:47:45.9806052Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9806715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9807276Z return mod(**inputs) 2025-08-14T21:47:45.9807919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9808579Z outputs = self.model( 2025-08-14T21:47:45.9809233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9809929Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9810662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9811356Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9811969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9812596Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9813278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9814067Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9814365Z 2025-08-14T21:47:45.9814547Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9815207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9815775Z return mod(**inputs) 2025-08-14T21:47:45.9816415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9817098Z outputs = self.model( 2025-08-14T21:47:45.9817728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9818405Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9819082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9819767Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9820359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9820997Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9821695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:47:45.9822390Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:45.9822634Z 2025-08-14T21:47:45.9822806Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9823434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9824016Z return mod(**inputs) 2025-08-14T21:47:45.9824607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9825261Z outputs = self.model( 2025-08-14T21:47:45.9825876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9826545Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9827200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9827906Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9828498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9829102Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9829779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9830499Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9831201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:45.9832001Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:45.9832362Z 2025-08-14T21:47:45.9832525Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9833169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9833736Z return mod(**inputs) 2025-08-14T21:47:45.9834475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9835172Z outputs = self.model( 2025-08-14T21:47:45.9835795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9836447Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9837095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9837923Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9838516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9839110Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9839859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9840588Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9841280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:45.9841983Z key_states = self.k_proj(current_states) 2025-08-14T21:47:45.9842227Z 2025-08-14T21:47:45.9842400Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9843046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9843150Z return mod(**inputs) 2025-08-14T21:47:45.9843644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9843766Z outputs = self.model( 2025-08-14T21:47:45.9844264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9844401Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9844901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9845017Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9845534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9845667Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9846137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9846300Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9846800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:45.9846944Z value_states = self.v_proj(current_states) 2025-08-14T21:47:45.9847016Z 2025-08-14T21:47:45.9847151Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9847282Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9847409Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9847526Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9847692Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9848058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9848161Z return mod(**inputs) 2025-08-14T21:47:45.9848631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9848738Z outputs = self.model( 2025-08-14T21:47:45.9849203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9849331Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9849796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9849992Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9850400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9850524Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9850995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9851142Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9851598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9851769Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9852328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:45.9852563Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:45.9852573Z 2025-08-14T21:47:45.9852752Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9853114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9853226Z return mod(**inputs) 2025-08-14T21:47:45.9853702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9853808Z outputs = self.model( 2025-08-14T21:47:45.9854278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9854391Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9854863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9854978Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9855384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9855526Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9855975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9856121Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9856573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9856728Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9857253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:45.9857432Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:45.9857471Z 2025-08-14T21:47:45.9857638Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9857996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9858096Z return mod(**inputs) 2025-08-14T21:47:45.9858553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9858659Z outputs = self.model( 2025-08-14T21:47:45.9859106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9859227Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9859676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9859794Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9860185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9860379Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9860812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9860948Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9861391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:45.9861525Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:45.9861530Z 2025-08-14T21:47:45.9861693Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9862042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9862141Z return mod(**inputs) 2025-08-14T21:47:45.9862611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9862721Z outputs = self.model( 2025-08-14T21:47:45.9863184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9863309Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9863763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9863872Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9864257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9864370Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9864804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9865003Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9865011Z 2025-08-14T21:47:45.9865177Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9865547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9865649Z return mod(**inputs) 2025-08-14T21:47:45.9866101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9866216Z outputs = self.model( 2025-08-14T21:47:45.9866670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9866779Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9867221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9867329Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9867717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9867863Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9868305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9868504Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9868510Z 2025-08-14T21:47:45.9868675Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9869030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9869121Z return mod(**inputs) 2025-08-14T21:47:45.9869574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9869682Z outputs = self.model( 2025-08-14T21:47:45.9870128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9870269Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9870751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9870859Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9871252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9871374Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9871821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:47:45.9871953Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:45.9871960Z 2025-08-14T21:47:45.9872125Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9872503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9872606Z return mod(**inputs) 2025-08-14T21:47:45.9873086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9873200Z outputs = self.model( 2025-08-14T21:47:45.9873675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9873791Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9874277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9874390Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9874824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9874945Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9875390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-08-14T21:47:45.9875527Z hidden_states = residual + hidden_states 2025-08-14T21:47:45.9875534Z 2025-08-14T21:47:45.9875698Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9876048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9876144Z return mod(**inputs) 2025-08-14T21:47:45.9876595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9876702Z outputs = self.model( 2025-08-14T21:47:45.9877148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9877259Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9877708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9877839Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9878234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9878345Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9878783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9878938Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9879384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:45.9879637Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:45.9879651Z 2025-08-14T21:47:45.9879812Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9880160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9880266Z return mod(**inputs) 2025-08-14T21:47:45.9880766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9880873Z outputs = self.model( 2025-08-14T21:47:45.9881346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9881460Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9881930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9882043Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9882451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9882582Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9883069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9883227Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9883704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:45.9883837Z key_states = self.k_proj(current_states) 2025-08-14T21:47:45.9883843Z 2025-08-14T21:47:45.9884035Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9884385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9884472Z return mod(**inputs) 2025-08-14T21:47:45.9884944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9885041Z outputs = self.model( 2025-08-14T21:47:45.9885600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9885730Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9886225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9886358Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9886756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9886880Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9887354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9887513Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9887967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:45.9888099Z value_states = self.v_proj(current_states) 2025-08-14T21:47:45.9888106Z 2025-08-14T21:47:45.9888267Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9888403Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9888519Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9888630Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9888799Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9889134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9889238Z return mod(**inputs) 2025-08-14T21:47:45.9889691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9889794Z outputs = self.model( 2025-08-14T21:47:45.9890253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9890366Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9890811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9891020Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9891411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9891535Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9891975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9892116Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9892570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9892728Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9893274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:45.9893498Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:45.9893507Z 2025-08-14T21:47:45.9893675Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9894024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9894123Z return mod(**inputs) 2025-08-14T21:47:45.9894573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9894672Z outputs = self.model( 2025-08-14T21:47:45.9895104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9895225Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9895668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9895773Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9896165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9896279Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9896734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9896872Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9897317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9897475Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9897983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:45.9898154Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:45.9898167Z 2025-08-14T21:47:45.9898373Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9898718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9898826Z return mod(**inputs) 2025-08-14T21:47:45.9899274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9899377Z outputs = self.model( 2025-08-14T21:47:45.9899831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9899946Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9900421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9900531Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9900918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9901071Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9901538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9901678Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9902126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:45.9902252Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:45.9902258Z 2025-08-14T21:47:45.9902433Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9902790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9902893Z return mod(**inputs) 2025-08-14T21:47:45.9903393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9903510Z outputs = self.model( 2025-08-14T21:47:45.9903971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9904085Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9904541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9904661Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9905060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9905183Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9905639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9905843Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9905850Z 2025-08-14T21:47:45.9906015Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9906354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9906449Z return mod(**inputs) 2025-08-14T21:47:45.9906922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9907024Z outputs = self.model( 2025-08-14T21:47:45.9907491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9907606Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9908064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9908183Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9908578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9908740Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9909204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9909399Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9909407Z 2025-08-14T21:47:45.9909587Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9909939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9910041Z return mod(**inputs) 2025-08-14T21:47:45.9910515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9910618Z outputs = self.model( 2025-08-14T21:47:45.9911089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9911234Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9911719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9911843Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9912237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9912361Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9912827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:47:45.9912954Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:45.9912960Z 2025-08-14T21:47:45.9913138Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9913517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9913619Z return mod(**inputs) 2025-08-14T21:47:45.9914106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9914211Z outputs = self.model( 2025-08-14T21:47:45.9914669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9914786Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9915245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9915361Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9915755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9915877Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9916339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9916490Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9916953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:45.9917211Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:45.9917219Z 2025-08-14T21:47:45.9917383Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9917752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9917855Z return mod(**inputs) 2025-08-14T21:47:45.9918322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9918434Z outputs = self.model( 2025-08-14T21:47:45.9918899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9919048Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9919501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9919611Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9920013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9920132Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9920582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9920722Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9921166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:45.9921300Z key_states = self.k_proj(current_states) 2025-08-14T21:47:45.9921308Z 2025-08-14T21:47:45.9921473Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9921865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9921974Z return mod(**inputs) 2025-08-14T21:47:45.9922432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9922549Z outputs = self.model( 2025-08-14T21:47:45.9923028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9923145Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9923625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9923739Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9924185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9924316Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9924780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9924931Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9925500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:45.9925643Z value_states = self.v_proj(current_states) 2025-08-14T21:47:45.9925650Z 2025-08-14T21:47:45.9925787Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9925913Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9926044Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9926164Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9926345Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9926713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9926828Z return mod(**inputs) 2025-08-14T21:47:45.9927342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9927457Z outputs = self.model( 2025-08-14T21:47:45.9927939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9928057Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9928526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9928640Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9929050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9929174Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9929652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9929806Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9930246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9930412Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9930932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:45.9931143Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:45.9931150Z 2025-08-14T21:47:45.9931324Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9931671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9931778Z return mod(**inputs) 2025-08-14T21:47:45.9932278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9932377Z outputs = self.model( 2025-08-14T21:47:45.9932833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9932946Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9933401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9933521Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9933935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9934075Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9934567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9934712Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9935179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9935333Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9935857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:45.9936031Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:45.9936037Z 2025-08-14T21:47:45.9936200Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9936550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9936647Z return mod(**inputs) 2025-08-14T21:47:45.9937106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9937220Z outputs = self.model( 2025-08-14T21:47:45.9937831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9937961Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9938409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9938520Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9938918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9939035Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9939488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 378, in forward 2025-08-14T21:47:45.9939634Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:47:45.9940089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:45.9940315Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:45.9940322Z 2025-08-14T21:47:45.9940491Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9940844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9940956Z return mod(**inputs) 2025-08-14T21:47:45.9941428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9941544Z outputs = self.model( 2025-08-14T21:47:45.9942010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9942121Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9942588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9942774Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9943182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9943302Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9943758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9943956Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9943963Z 2025-08-14T21:47:45.9944130Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9944480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9944583Z return mod(**inputs) 2025-08-14T21:47:45.9945076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9945191Z outputs = self.model( 2025-08-14T21:47:45.9945655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9945768Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9946229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9946336Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9946729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9946858Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9947310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 389, in forward 2025-08-14T21:47:45.9947516Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:45.9947524Z 2025-08-14T21:47:45.9947690Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9948044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9948144Z return mod(**inputs) 2025-08-14T21:47:45.9948605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9948715Z outputs = self.model( 2025-08-14T21:47:45.9949172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9949284Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9949755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9949872Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9950264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9950427Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9950876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 391, in forward 2025-08-14T21:47:45.9951008Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:45.9951015Z 2025-08-14T21:47:45.9951180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9951537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9951654Z return mod(**inputs) 2025-08-14T21:47:45.9952103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9952210Z outputs = self.model( 2025-08-14T21:47:45.9952655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1261, in forward 2025-08-14T21:47:45.9952768Z encoder_outputs = self.encoder( 2025-08-14T21:47:45.9953265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 878, in forward 2025-08-14T21:47:45.9953377Z layer_outputs = encoder_layer( 2025-08-14T21:47:45.9953765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9953894Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9954339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 393, in forward 2025-08-14T21:47:45.9954470Z hidden_states = residual + hidden_states 2025-08-14T21:47:45.9954476Z 2025-08-14T21:47:45.9954631Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9955014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9955121Z return mod(**inputs) 2025-08-14T21:47:45.9955574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9955682Z outputs = self.model( 2025-08-14T21:47:45.9956121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:45.9956233Z decoder_outputs = self.decoder( 2025-08-14T21:47:45.9956684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1095, in forward 2025-08-14T21:47:45.9956951Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-08-14T21:47:45.9957342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-08-14T21:47:45.9957454Z return func(*args, **kwargs) 2025-08-14T21:47:45.9957889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-08-14T21:47:45.9958257Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-08-14T21:47:45.9958807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-08-14T21:47:45.9959123Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-14T21:47:45.9959130Z 2025-08-14T21:47:45.9959302Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9959648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9959756Z return mod(**inputs) 2025-08-14T21:47:45.9960201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9960306Z outputs = self.model( 2025-08-14T21:47:45.9960795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:45.9960908Z decoder_outputs = self.decoder( 2025-08-14T21:47:45.9961363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1095, in forward 2025-08-14T21:47:45.9961638Z positions = self.embed_positions(input_ids, inputs_embeds, past_key_values_length) 2025-08-14T21:47:45.9962040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context 2025-08-14T21:47:45.9962155Z return func(*args, **kwargs) 2025-08-14T21:47:45.9962608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 148, in forward 2025-08-14T21:47:45.9962976Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length).to( 2025-08-14T21:47:45.9963580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 81, in create_position_ids_from_input_ids 2025-08-14T21:47:45.9963924Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-14T21:47:45.9963931Z 2025-08-14T21:47:45.9964109Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9964467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9964568Z return mod(**inputs) 2025-08-14T21:47:45.9965038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9965142Z outputs = self.model( 2025-08-14T21:47:45.9965785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:45.9965915Z decoder_outputs = self.decoder( 2025-08-14T21:47:45.9966414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:45.9966551Z layer_outputs = decoder_layer( 2025-08-14T21:47:45.9966966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9967087Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9967544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:45.9967708Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:45.9968168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:45.9968413Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:45.9968420Z 2025-08-14T21:47:45.9968591Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9968957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9969057Z return mod(**inputs) 2025-08-14T21:47:45.9969537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9969641Z outputs = self.model( 2025-08-14T21:47:45.9970096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:45.9970218Z decoder_outputs = self.decoder( 2025-08-14T21:47:45.9970677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:45.9970790Z layer_outputs = decoder_layer( 2025-08-14T21:47:45.9971201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9971354Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9971831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:45.9971988Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:45.9972443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:45.9972576Z key_states = self.k_proj(current_states) 2025-08-14T21:47:45.9972583Z 2025-08-14T21:47:45.9972755Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9973116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9973217Z return mod(**inputs) 2025-08-14T21:47:45.9973684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9973801Z outputs = self.model( 2025-08-14T21:47:45.9974329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:45.9974444Z decoder_outputs = self.decoder( 2025-08-14T21:47:45.9974903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:45.9975016Z layer_outputs = decoder_layer( 2025-08-14T21:47:45.9975416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9975543Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9975996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:45.9976186Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:45.9976646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:45.9976793Z value_states = self.v_proj(current_states) 2025-08-14T21:47:45.9976800Z 2025-08-14T21:47:45.9976926Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9977043Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9977173Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9977290Z cudagraph partition due to non gpu ops 2025-08-14T21:47:45.9977462Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9977831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9977933Z return mod(**inputs) 2025-08-14T21:47:45.9978409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9978516Z outputs = self.model( 2025-08-14T21:47:45.9978971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:45.9979097Z decoder_outputs = self.decoder( 2025-08-14T21:47:45.9979551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:45.9979661Z layer_outputs = decoder_layer( 2025-08-14T21:47:45.9980069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9980192Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9980656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:45.9980815Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:45.9981283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9981475Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9981991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:45.9982214Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:45.9982220Z 2025-08-14T21:47:45.9982390Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9982714Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9982819Z return mod(**inputs) 2025-08-14T21:47:45.9983267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9983374Z outputs = self.model( 2025-08-14T21:47:45.9983839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:45.9983950Z decoder_outputs = self.decoder( 2025-08-14T21:47:45.9984458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:45.9984571Z layer_outputs = decoder_layer( 2025-08-14T21:47:45.9984977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9985111Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9985577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:45.9985731Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:45.9986198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:45.9986376Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:45.9986906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:45.9987088Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:45.9987094Z 2025-08-14T21:47:45.9987273Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9987626Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9987727Z return mod(**inputs) 2025-08-14T21:47:45.9988181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9988278Z outputs = self.model( 2025-08-14T21:47:45.9988729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:45.9988847Z decoder_outputs = self.decoder( 2025-08-14T21:47:45.9989300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:45.9989414Z layer_outputs = decoder_layer( 2025-08-14T21:47:45.9989805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9989924Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9990383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:45.9990534Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:45.9990973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:45.9991107Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:45.9991112Z 2025-08-14T21:47:45.9991274Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9991625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9991750Z return mod(**inputs) 2025-08-14T21:47:45.9992209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9992321Z outputs = self.model( 2025-08-14T21:47:45.9992783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:45.9992898Z decoder_outputs = self.decoder( 2025-08-14T21:47:45.9993376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:45.9993486Z layer_outputs = decoder_layer( 2025-08-14T21:47:45.9993879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9994000Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9994464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:45.9994727Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:45.9995182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:45.9995437Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:45.9995443Z 2025-08-14T21:47:45.9995608Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:45.9995948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:45.9996055Z return mod(**inputs) 2025-08-14T21:47:45.9996510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:45.9996651Z outputs = self.model( 2025-08-14T21:47:45.9997112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:45.9997229Z decoder_outputs = self.decoder( 2025-08-14T21:47:45.9997672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:45.9997781Z layer_outputs = decoder_layer( 2025-08-14T21:47:45.9998162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:45.9998289Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:45.9998732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:45.9998914Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:45.9999373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:45.9999499Z key_states = self.k_proj(current_states) 2025-08-14T21:47:45.9999508Z 2025-08-14T21:47:45.9999687Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0000038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0000149Z return mod(**inputs) 2025-08-14T21:47:46.0000615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0000715Z outputs = self.model( 2025-08-14T21:47:46.0001166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0001274Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0001721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0001839Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0002252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0002381Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0002848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0003029Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0003494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:46.0003634Z value_states = self.v_proj(current_states) 2025-08-14T21:47:46.0003640Z 2025-08-14T21:47:46.0003771Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0003907Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0004032Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0004152Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0004327Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0004742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0004855Z return mod(**inputs) 2025-08-14T21:47:46.0005424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0005546Z outputs = self.model( 2025-08-14T21:47:46.0006056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0006181Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0006682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0006803Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0007261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0007397Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0007867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0008043Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0008513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0008672Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0009220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:46.0009446Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:46.0009454Z 2025-08-14T21:47:46.0009632Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0010000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0010110Z return mod(**inputs) 2025-08-14T21:47:46.0010601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0010708Z outputs = self.model( 2025-08-14T21:47:46.0011182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0011307Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0011781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0011899Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0012322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0012452Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0012962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0013142Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0013604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0013773Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0014328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:46.0014494Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:46.0014501Z 2025-08-14T21:47:46.0014664Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0015009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0015118Z return mod(**inputs) 2025-08-14T21:47:46.0015623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0015753Z outputs = self.model( 2025-08-14T21:47:46.0016221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0016336Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0016807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0016927Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0017316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0017442Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0017905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0018086Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0018534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:46.0018662Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:46.0018668Z 2025-08-14T21:47:46.0018847Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0019197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0019299Z return mod(**inputs) 2025-08-14T21:47:46.0019771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0019872Z outputs = self.model( 2025-08-14T21:47:46.0020339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0020456Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0020918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0021041Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0021453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0021588Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0022057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0022263Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0022270Z 2025-08-14T21:47:46.0022451Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0022810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0022921Z return mod(**inputs) 2025-08-14T21:47:46.0023417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0023524Z outputs = self.model( 2025-08-14T21:47:46.0023996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0024109Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0024564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0024680Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0025071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0025197Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0025637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0025846Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0025876Z 2025-08-14T21:47:46.0026049Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0026393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0026487Z return mod(**inputs) 2025-08-14T21:47:46.0026860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0026962Z outputs = self.model( 2025-08-14T21:47:46.0027425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0027540Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0028019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0028140Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0028517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0028637Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0029095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:47:46.0029222Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:46.0029228Z 2025-08-14T21:47:46.0029397Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0029743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0029843Z return mod(**inputs) 2025-08-14T21:47:46.0030321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0030427Z outputs = self.model( 2025-08-14T21:47:46.0030903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0031018Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0031482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0031615Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0032014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0032137Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0032600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0032763Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0033244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:46.0033547Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:46.0033555Z 2025-08-14T21:47:46.0033733Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0034113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0034215Z return mod(**inputs) 2025-08-14T21:47:46.0034682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0034783Z outputs = self.model( 2025-08-14T21:47:46.0035247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0035367Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0035832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0035948Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0036394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0036517Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0036976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0037136Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0037583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:46.0037892Z key_states = self.k_proj(current_states) 2025-08-14T21:47:46.0037900Z 2025-08-14T21:47:46.0038070Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0038508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0038615Z return mod(**inputs) 2025-08-14T21:47:46.0039070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0039187Z outputs = self.model( 2025-08-14T21:47:46.0039637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0039752Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0040216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0040334Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0040762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0040889Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0041365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0041545Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0042009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:46.0042151Z value_states = self.v_proj(current_states) 2025-08-14T21:47:46.0042157Z 2025-08-14T21:47:46.0042283Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0043010Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0043154Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0043285Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0043473Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0043860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0043966Z return mod(**inputs) 2025-08-14T21:47:46.0044461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0044632Z outputs = self.model( 2025-08-14T21:47:46.0045109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0045238Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0045828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0045954Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0046401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0046530Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0047017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0047172Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0047664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0047859Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0048386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:46.0048612Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:46.0048619Z 2025-08-14T21:47:46.0048789Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0049138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0049245Z return mod(**inputs) 2025-08-14T21:47:46.0049725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0049824Z outputs = self.model( 2025-08-14T21:47:46.0050271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0050382Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0050822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0050927Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0051303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0051427Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0051865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0052027Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0052467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0052625Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0053153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:46.0053327Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:46.0053334Z 2025-08-14T21:47:46.0053508Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0053849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0053947Z return mod(**inputs) 2025-08-14T21:47:46.0054410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0054510Z outputs = self.model( 2025-08-14T21:47:46.0054947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0055108Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0055543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0055656Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0056034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0056153Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0056600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0056750Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0057181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:46.0057315Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:46.0057323Z 2025-08-14T21:47:46.0057483Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0057896Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0057994Z return mod(**inputs) 2025-08-14T21:47:46.0058430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0058535Z outputs = self.model( 2025-08-14T21:47:46.0058983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0059105Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0059549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0059656Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0060071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0060196Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0060649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 482, in forward 2025-08-14T21:47:46.0060781Z hidden_states = residual + hidden_states 2025-08-14T21:47:46.0060786Z 2025-08-14T21:47:46.0060950Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0061311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0061411Z return mod(**inputs) 2025-08-14T21:47:46.0061868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0061980Z outputs = self.model( 2025-08-14T21:47:46.0062436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0062550Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0063016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0063126Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0063528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0063654Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0064123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0064319Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0064758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:46.0065017Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:46.0065047Z 2025-08-14T21:47:46.0065209Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0065551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0065655Z return mod(**inputs) 2025-08-14T21:47:46.0066106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0066200Z outputs = self.model( 2025-08-14T21:47:46.0066639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0066749Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0067193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0067302Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0067688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0067867Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0068315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0068491Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0068937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:46.0069064Z key_states = self.k_proj(current_states) 2025-08-14T21:47:46.0069070Z 2025-08-14T21:47:46.0069248Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0069602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0069702Z return mod(**inputs) 2025-08-14T21:47:46.0070183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0070296Z outputs = self.model( 2025-08-14T21:47:46.0070792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0070905Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0071356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0071479Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0071880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0072018Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0072465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0072649Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0073136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:46.0073276Z value_states = self.v_proj(current_states) 2025-08-14T21:47:46.0073283Z 2025-08-14T21:47:46.0073414Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0073554Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0073678Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0073810Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0073980Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0074345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0074447Z return mod(**inputs) 2025-08-14T21:47:46.0074929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0075036Z outputs = self.model( 2025-08-14T21:47:46.0075529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0075645Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0076101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0076202Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0076601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0076734Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0077191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0077369Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0077819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0078023Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0078543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:46.0078753Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:46.0078760Z 2025-08-14T21:47:46.0078926Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0079273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0079370Z return mod(**inputs) 2025-08-14T21:47:46.0079830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0079930Z outputs = self.model( 2025-08-14T21:47:46.0080396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0080517Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0080963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0081084Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0081463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0081583Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0082040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0082201Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0082684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0082856Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0083407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:46.0083601Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:46.0083608Z 2025-08-14T21:47:46.0083784Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0084151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0084262Z return mod(**inputs) 2025-08-14T21:47:46.0084720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0084838Z outputs = self.model( 2025-08-14T21:47:46.0085438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0085573Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0086086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0086206Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0086617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0086751Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0087202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0087383Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0087850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:46.0087980Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:46.0087986Z 2025-08-14T21:47:46.0088174Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0088543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0088702Z return mod(**inputs) 2025-08-14T21:47:46.0089168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0089272Z outputs = self.model( 2025-08-14T21:47:46.0089757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0089873Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0090346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0090471Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0090922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0091052Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0091525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0091728Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0091735Z 2025-08-14T21:47:46.0091916Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0092279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0092385Z return mod(**inputs) 2025-08-14T21:47:46.0092872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0092979Z outputs = self.model( 2025-08-14T21:47:46.0093465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0093583Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0094056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0094184Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0094594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0094727Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0095199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0095397Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0095403Z 2025-08-14T21:47:46.0095580Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0095939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0096047Z return mod(**inputs) 2025-08-14T21:47:46.0096524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0096684Z outputs = self.model( 2025-08-14T21:47:46.0097161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0097276Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0097733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0097848Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0098224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0098350Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0098796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:47:46.0098922Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:46.0098954Z 2025-08-14T21:47:46.0099149Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0099491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0099587Z return mod(**inputs) 2025-08-14T21:47:46.0100039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0100135Z outputs = self.model( 2025-08-14T21:47:46.0100589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0100700Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0101148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0101291Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0101678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0101801Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0102250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0102410Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0102876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:46.0103128Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:46.0103133Z 2025-08-14T21:47:46.0103301Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0103658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0103751Z return mod(**inputs) 2025-08-14T21:47:46.0104222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0104323Z outputs = self.model( 2025-08-14T21:47:46.0104779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0104902Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0105350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0105458Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0105861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0105985Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0106467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0106654Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0107120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:46.0107245Z key_states = self.k_proj(current_states) 2025-08-14T21:47:46.0107251Z 2025-08-14T21:47:46.0107419Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0107779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0107878Z return mod(**inputs) 2025-08-14T21:47:46.0108347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0108457Z outputs = self.model( 2025-08-14T21:47:46.0108913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0109026Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0109521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0109653Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0110052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0110173Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0110632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0110799Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0111253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:46.0111394Z value_states = self.v_proj(current_states) 2025-08-14T21:47:46.0111427Z 2025-08-14T21:47:46.0111557Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0111684Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0111820Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0111944Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0112115Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0112478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0112583Z return mod(**inputs) 2025-08-14T21:47:46.0113061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0113170Z outputs = self.model( 2025-08-14T21:47:46.0113640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0113766Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0114234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0114363Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0114769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0114898Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0115351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0115509Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0115982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0116148Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0116700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:46.0116935Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:46.0116969Z 2025-08-14T21:47:46.0117146Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0117509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0117622Z return mod(**inputs) 2025-08-14T21:47:46.0118093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0118202Z outputs = self.model( 2025-08-14T21:47:46.0118684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0118802Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0119273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0119392Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0119816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0119975Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0120442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0120611Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0121080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0121240Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0121792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:46.0121971Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:46.0122012Z 2025-08-14T21:47:46.0122186Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0122566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0122669Z return mod(**inputs) 2025-08-14T21:47:46.0123164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0123278Z outputs = self.model( 2025-08-14T21:47:46.0123785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0123921Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0124422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0124543Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0124951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0125074Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0125674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0125852Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0126355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:46.0126500Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:46.0126508Z 2025-08-14T21:47:46.0126693Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0127077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0127179Z return mod(**inputs) 2025-08-14T21:47:46.0127638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0127787Z outputs = self.model( 2025-08-14T21:47:46.0128271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0128385Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0128847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0128960Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0129374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0129490Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0129926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0130102Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0130551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:46.0130875Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:46.0130881Z 2025-08-14T21:47:46.0131042Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0131383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0131490Z return mod(**inputs) 2025-08-14T21:47:46.0131948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0132057Z outputs = self.model( 2025-08-14T21:47:46.0132545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0132661Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0133168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0133292Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0133698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0133833Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0134291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0134471Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0134914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:46.0135037Z key_states = self.k_proj(current_states) 2025-08-14T21:47:46.0135043Z 2025-08-14T21:47:46.0135215Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0135560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0135661Z return mod(**inputs) 2025-08-14T21:47:46.0136123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0136226Z outputs = self.model( 2025-08-14T21:47:46.0136681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0136789Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0137225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0137340Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0137873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0140996Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0141263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0141372Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0141619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:46.0141723Z value_states = self.v_proj(current_states) 2025-08-14T21:47:46.0141727Z 2025-08-14T21:47:46.0141809Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0141890Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0141963Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0142033Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0142140Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0142331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0142431Z return mod(**inputs) 2025-08-14T21:47:46.0142714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0142807Z outputs = self.model( 2025-08-14T21:47:46.0143046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0143124Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0143363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0143431Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0143649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0143725Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0143997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0144102Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0144341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0144440Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0144713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:46.0144843Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:46.0144847Z 2025-08-14T21:47:46.0144946Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0145133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0145203Z return mod(**inputs) 2025-08-14T21:47:46.0145447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0145518Z outputs = self.model( 2025-08-14T21:47:46.0145758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0145830Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0146073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0146142Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0146350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0146430Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0146669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0146779Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0147085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0147181Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0147458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:46.0147559Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:46.0147562Z 2025-08-14T21:47:46.0147667Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0147856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0147919Z return mod(**inputs) 2025-08-14T21:47:46.0148171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0148238Z outputs = self.model( 2025-08-14T21:47:46.0148466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0148574Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0148808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0148883Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0149090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0149163Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0149402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0149500Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0149751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:46.0149839Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:46.0149844Z 2025-08-14T21:47:46.0149942Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0150136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0150198Z return mod(**inputs) 2025-08-14T21:47:46.0150436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0150508Z outputs = self.model( 2025-08-14T21:47:46.0150743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0150819Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0151053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0151125Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0151339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0151414Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0151648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 499, in forward 2025-08-14T21:47:46.0151731Z hidden_states = residual + hidden_states 2025-08-14T21:47:46.0151734Z 2025-08-14T21:47:46.0151829Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0152023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0152086Z return mod(**inputs) 2025-08-14T21:47:46.0152327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0152399Z outputs = self.model( 2025-08-14T21:47:46.0152677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0152759Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0153009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0153080Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0153315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0153388Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0153625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0153747Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0153750Z 2025-08-14T21:47:46.0153845Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0154038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0154117Z return mod(**inputs) 2025-08-14T21:47:46.0154375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0154446Z outputs = self.model( 2025-08-14T21:47:46.0154689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0154764Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0155007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0155076Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0155292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0155381Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0155620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0155741Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0155745Z 2025-08-14T21:47:46.0155841Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0156035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0156096Z return mod(**inputs) 2025-08-14T21:47:46.0156341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0156409Z outputs = self.model( 2025-08-14T21:47:46.0156639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0156707Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0156951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0157020Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0157231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0157304Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0157543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:47:46.0157626Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:46.0157630Z 2025-08-14T21:47:46.0157723Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0157916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0157978Z return mod(**inputs) 2025-08-14T21:47:46.0158214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0158306Z outputs = self.model( 2025-08-14T21:47:46.0158546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0158614Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0158861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0158928Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0159143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0159218Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0159457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0159562Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0159838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:46.0160015Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:46.0160019Z 2025-08-14T21:47:46.0160118Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0160307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0160378Z return mod(**inputs) 2025-08-14T21:47:46.0160623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0160689Z outputs = self.model( 2025-08-14T21:47:46.0160937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0161025Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0161291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0161365Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0161577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0161659Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0161902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0162008Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0162253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:46.0162334Z key_states = self.k_proj(current_states) 2025-08-14T21:47:46.0162337Z 2025-08-14T21:47:46.0162450Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0162653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0162725Z return mod(**inputs) 2025-08-14T21:47:46.0162992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0163063Z outputs = self.model( 2025-08-14T21:47:46.0163328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0163404Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0163659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0163742Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0163966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0164062Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0164321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0164417Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0164672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:46.0164758Z value_states = self.v_proj(current_states) 2025-08-14T21:47:46.0164762Z 2025-08-14T21:47:46.0164841Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0164931Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0165008Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0165091Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0165192Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0165497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0165586Z return mod(**inputs) 2025-08-14T21:47:46.0165900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0165975Z outputs = self.model( 2025-08-14T21:47:46.0166261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0166336Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0166596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0166669Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0166888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0166976Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0167241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0167341Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0167609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0167703Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0167993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:46.0168122Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:46.0168127Z 2025-08-14T21:47:46.0168228Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0168428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0168492Z return mod(**inputs) 2025-08-14T21:47:46.0168746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0168814Z outputs = self.model( 2025-08-14T21:47:46.0169056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0169135Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0169379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0169451Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0169673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0169748Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0169999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0170117Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0170362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0170464Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0170748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:46.0170861Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:46.0170864Z 2025-08-14T21:47:46.0170963Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0171157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0171226Z return mod(**inputs) 2025-08-14T21:47:46.0171473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0171541Z outputs = self.model( 2025-08-14T21:47:46.0171806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0171893Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0172146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0172216Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0172426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0172508Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0172752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0172851Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0173110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:46.0173192Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:46.0173195Z 2025-08-14T21:47:46.0173302Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0173496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0173560Z return mod(**inputs) 2025-08-14T21:47:46.0173811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0173875Z outputs = self.model( 2025-08-14T21:47:46.0174125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0174195Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0174438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0174518Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0174737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0174819Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0175064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0175168Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0175420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:46.0175567Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:46.0175570Z 2025-08-14T21:47:46.0175668Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0175868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0175950Z return mod(**inputs) 2025-08-14T21:47:46.0176208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0176274Z outputs = self.model( 2025-08-14T21:47:46.0176518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0176596Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0176849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0176924Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0177132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0177206Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0177457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0177588Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0177826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:46.0177912Z key_states = self.k_proj(current_states) 2025-08-14T21:47:46.0177915Z 2025-08-14T21:47:46.0178011Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0178206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0178268Z return mod(**inputs) 2025-08-14T21:47:46.0178508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0178579Z outputs = self.model( 2025-08-14T21:47:46.0178828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0178901Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0179150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0179218Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0179434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0179508Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0179747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0179854Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0180093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:46.0180184Z value_states = self.v_proj(current_states) 2025-08-14T21:47:46.0180187Z 2025-08-14T21:47:46.0180264Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0180341Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0180422Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0180493Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0180589Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0180783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0180845Z return mod(**inputs) 2025-08-14T21:47:46.0181086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0181150Z outputs = self.model( 2025-08-14T21:47:46.0181388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0181494Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0181729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0181800Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0182017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0182091Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0182349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0182453Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0182702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0182817Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0183098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:46.0183264Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:46.0183268Z 2025-08-14T21:47:46.0183368Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0183562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0183636Z return mod(**inputs) 2025-08-14T21:47:46.0183885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0183954Z outputs = self.model( 2025-08-14T21:47:46.0184210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0184282Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0184551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0184627Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0184855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0184935Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0185173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0185278Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0185517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0185608Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0185887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:46.0185991Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:46.0185994Z 2025-08-14T21:47:46.0186090Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0186283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0186346Z return mod(**inputs) 2025-08-14T21:47:46.0186589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0186653Z outputs = self.model( 2025-08-14T21:47:46.0186889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0186965Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0187201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0187275Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0187535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0187614Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0187857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0187955Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0188192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:46.0188275Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:46.0188279Z 2025-08-14T21:47:46.0188372Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0188562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0188623Z return mod(**inputs) 2025-08-14T21:47:46.0188865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0188954Z outputs = self.model( 2025-08-14T21:47:46.0189206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0189284Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0189519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0189589Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0189804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0189877Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0190111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0190253Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0190257Z 2025-08-14T21:47:46.0190354Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0190551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0190614Z return mod(**inputs) 2025-08-14T21:47:46.0190849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0190923Z outputs = self.model( 2025-08-14T21:47:46.0191159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0191227Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0191471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0191541Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0191756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0191832Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0192068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0192194Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0192198Z 2025-08-14T21:47:46.0192298Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0192502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0192567Z return mod(**inputs) 2025-08-14T21:47:46.0192831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0192906Z outputs = self.model( 2025-08-14T21:47:46.0193170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0193241Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0193490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0193560Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0193779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0193852Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0194104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:47:46.0194190Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:46.0194193Z 2025-08-14T21:47:46.0194288Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0194483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0194544Z return mod(**inputs) 2025-08-14T21:47:46.0194814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0194890Z outputs = self.model( 2025-08-14T21:47:46.0195126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0195194Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0195443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0195511Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0195724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0195797Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0196048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-08-14T21:47:46.0196136Z hidden_states = residual + hidden_states 2025-08-14T21:47:46.0196140Z 2025-08-14T21:47:46.0196236Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0196430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0196492Z return mod(**inputs) 2025-08-14T21:47:46.0196729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0196801Z outputs = self.model( 2025-08-14T21:47:46.0197035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0197105Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0197351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0197421Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0197637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0197709Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0197954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0198052Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0198281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:46.0198438Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:46.0198442Z 2025-08-14T21:47:46.0198536Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0198742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0198813Z return mod(**inputs) 2025-08-14T21:47:46.0199053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0199125Z outputs = self.model( 2025-08-14T21:47:46.0199361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0199429Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0199671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0199740Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0199948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0200030Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0200266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0200400Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0200637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:46.0200714Z key_states = self.k_proj(current_states) 2025-08-14T21:47:46.0200718Z 2025-08-14T21:47:46.0200824Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0201013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0201083Z return mod(**inputs) 2025-08-14T21:47:46.0201322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0201386Z outputs = self.model( 2025-08-14T21:47:46.0201661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0201735Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0201980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0202056Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0202270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0202356Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0202634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0202736Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0203023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:46.0203115Z value_states = self.v_proj(current_states) 2025-08-14T21:47:46.0203121Z 2025-08-14T21:47:46.0203213Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0203300Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0203381Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0203469Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0203574Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0203784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0203861Z return mod(**inputs) 2025-08-14T21:47:46.0204131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0204208Z outputs = self.model( 2025-08-14T21:47:46.0204479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0204581Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0204859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0204944Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0205164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0205252Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0205591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0205701Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0205951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0206048Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0206359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:46.0206525Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:46.0206529Z 2025-08-14T21:47:46.0206642Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0206839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0206906Z return mod(**inputs) 2025-08-14T21:47:46.0207164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0207233Z outputs = self.model( 2025-08-14T21:47:46.0207481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0207565Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0207837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0207916Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0208127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0208201Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0208445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0208537Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0208787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0208878Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0209156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:46.0209268Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:46.0209274Z 2025-08-14T21:47:46.0209374Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0209564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0209636Z return mod(**inputs) 2025-08-14T21:47:46.0209879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0209953Z outputs = self.model( 2025-08-14T21:47:46.0210196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0210268Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0210515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0210607Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0210825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0210913Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0211162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0211262Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0211510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:46.0211595Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:46.0211598Z 2025-08-14T21:47:46.0211708Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0211903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0211974Z return mod(**inputs) 2025-08-14T21:47:46.0212227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0212323Z outputs = self.model( 2025-08-14T21:47:46.0212579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0212650Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0212896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0212975Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0213189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0213270Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0213528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0213634Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0213887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:46.0214030Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:46.0214034Z 2025-08-14T21:47:46.0214139Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0214329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0214393Z return mod(**inputs) 2025-08-14T21:47:46.0214641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0214706Z outputs = self.model( 2025-08-14T21:47:46.0214952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0215029Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0215273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0215350Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0215563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0215639Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0215890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0215992Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0216243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:46.0216321Z key_states = self.k_proj(current_states) 2025-08-14T21:47:46.0216345Z 2025-08-14T21:47:46.0216445Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0216653Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0216717Z return mod(**inputs) 2025-08-14T21:47:46.0216958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0217032Z outputs = self.model( 2025-08-14T21:47:46.0217277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0217354Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0217596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0217665Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0217883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0217957Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0218244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0218359Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0218597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:46.0218686Z value_states = self.v_proj(current_states) 2025-08-14T21:47:46.0218690Z 2025-08-14T21:47:46.0218764Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0218838Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0218917Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0218989Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0219091Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0219295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0219363Z return mod(**inputs) 2025-08-14T21:47:46.0219609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0219672Z outputs = self.model( 2025-08-14T21:47:46.0219905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0219981Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0220218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0220293Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0220500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0220573Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0220816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0220918Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0221162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0221261Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0221538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:46.0221666Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:46.0221670Z 2025-08-14T21:47:46.0221764Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0221956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0222045Z return mod(**inputs) 2025-08-14T21:47:46.0222288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0222362Z outputs = self.model( 2025-08-14T21:47:46.0222602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0222673Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0222924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0222992Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0223202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0223283Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0223524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0223633Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0223915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0224007Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0224284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:46.0224384Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:46.0224388Z 2025-08-14T21:47:46.0224493Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0224680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0224742Z return mod(**inputs) 2025-08-14T21:47:46.0225003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0225069Z outputs = self.model( 2025-08-14T21:47:46.0225318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0225388Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0225625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0225700Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0225908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0225983Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0226227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0226328Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0226573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:46.0226651Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:46.0226655Z 2025-08-14T21:47:46.0226749Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0226946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0227010Z return mod(**inputs) 2025-08-14T21:47:46.0227252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0227323Z outputs = self.model( 2025-08-14T21:47:46.0227559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0227634Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0227891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0227962Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0228176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0228249Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0228490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0228604Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0228608Z 2025-08-14T21:47:46.0228706Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0228904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0228970Z return mod(**inputs) 2025-08-14T21:47:46.0229210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0229285Z outputs = self.model( 2025-08-14T21:47:46.0229549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0229628Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0229865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0229933Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0230149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0230222Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0230463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0230592Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0230597Z 2025-08-14T21:47:46.0230696Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0230900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0230964Z return mod(**inputs) 2025-08-14T21:47:46.0231207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0231280Z outputs = self.model( 2025-08-14T21:47:46.0231521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0231599Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0231842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0231910Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0232135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0232212Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0232472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:47:46.0232549Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:46.0232553Z 2025-08-14T21:47:46.0232647Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0232841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0232905Z return mod(**inputs) 2025-08-14T21:47:46.0233155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0233228Z outputs = self.model( 2025-08-14T21:47:46.0233479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0233582Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0233834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0233905Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0234128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0234204Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0234453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0234556Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0234813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:46.0234964Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:46.0234968Z 2025-08-14T21:47:46.0235065Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0235286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0235360Z return mod(**inputs) 2025-08-14T21:47:46.0235607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0235679Z outputs = self.model( 2025-08-14T21:47:46.0235926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0235996Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0236256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0236324Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0236547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0236631Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0236875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0236975Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0237217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:46.0237294Z key_states = self.k_proj(current_states) 2025-08-14T21:47:46.0237297Z 2025-08-14T21:47:46.0237402Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0237906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0237989Z return mod(**inputs) 2025-08-14T21:47:46.0238242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0238312Z outputs = self.model( 2025-08-14T21:47:46.0238568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0238640Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0238886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0238965Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0239182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0239266Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0239513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0239685Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0239941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:46.0240025Z value_states = self.v_proj(current_states) 2025-08-14T21:47:46.0240029Z 2025-08-14T21:47:46.0240113Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0240191Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0240266Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0240348Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0240444Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0240639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0240712Z return mod(**inputs) 2025-08-14T21:47:46.0240958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0241035Z outputs = self.model( 2025-08-14T21:47:46.0241280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0241399Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0241651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0241723Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0241932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0242015Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0242258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0242356Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0242630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0242727Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0243029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:46.0243158Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:46.0243162Z 2025-08-14T21:47:46.0243270Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0243467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0243531Z return mod(**inputs) 2025-08-14T21:47:46.0243790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0243857Z outputs = self.model( 2025-08-14T21:47:46.0244110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0244191Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0244443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0244521Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0244740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0244816Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0245074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0245169Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0245475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0245619Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0245932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:46.0246054Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:46.0246058Z 2025-08-14T21:47:46.0246166Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0246373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0246459Z return mod(**inputs) 2025-08-14T21:47:46.0246707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0246778Z outputs = self.model( 2025-08-14T21:47:46.0247024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0247094Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0247350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0247460Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0247673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0247758Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0247997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0248100Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0248341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:46.0248420Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:46.0248424Z 2025-08-14T21:47:46.0248544Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0248744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0248813Z return mod(**inputs) 2025-08-14T21:47:46.0249068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0249134Z outputs = self.model( 2025-08-14T21:47:46.0249371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0249437Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0249666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0249740Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0249941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0250022Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0250253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 482, in forward 2025-08-14T21:47:46.0250327Z hidden_states = residual + hidden_states 2025-08-14T21:47:46.0250330Z 2025-08-14T21:47:46.0250430Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0250610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0250677Z return mod(**inputs) 2025-08-14T21:47:46.0250910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0250974Z outputs = self.model( 2025-08-14T21:47:46.0251209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0251276Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0251525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0251601Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0251803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0251881Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0252113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0252213Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0252453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:46.0252591Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:46.0252595Z 2025-08-14T21:47:46.0252702Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0252889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0252983Z return mod(**inputs) 2025-08-14T21:47:46.0253232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0253296Z outputs = self.model( 2025-08-14T21:47:46.0253533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0253609Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0253847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0253921Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0254128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0254219Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0254465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0254567Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0254812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:46.0254886Z key_states = self.k_proj(current_states) 2025-08-14T21:47:46.0254890Z 2025-08-14T21:47:46.0254984Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0255178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0255240Z return mod(**inputs) 2025-08-14T21:47:46.0255476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0255554Z outputs = self.model( 2025-08-14T21:47:46.0255790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0255870Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0256113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0256182Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0256394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0256465Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0256699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0256804Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0257042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:46.0257151Z value_states = self.v_proj(current_states) 2025-08-14T21:47:46.0257156Z 2025-08-14T21:47:46.0257231Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0257305Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0257385Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0257457Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0257561Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0257747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0257808Z return mod(**inputs) 2025-08-14T21:47:46.0258052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0258116Z outputs = self.model( 2025-08-14T21:47:46.0258354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0258432Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0258701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0258778Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0258985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0259059Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0259304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0259406Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0259643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0259763Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0260040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:46.0260172Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:46.0260176Z 2025-08-14T21:47:46.0260269Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0260453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0260534Z return mod(**inputs) 2025-08-14T21:47:46.0260762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0260831Z outputs = self.model( 2025-08-14T21:47:46.0261061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0261130Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0261364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0261435Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0261634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0261713Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0261944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0262046Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0262280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0262368Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0262647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:46.0262766Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:46.0262771Z 2025-08-14T21:47:46.0262874Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0263059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0263121Z return mod(**inputs) 2025-08-14T21:47:46.0263362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0263424Z outputs = self.model( 2025-08-14T21:47:46.0263659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0263733Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0263977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0264053Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0264286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0264357Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0264594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0264691Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0264929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:46.0265002Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:46.0265006Z 2025-08-14T21:47:46.0265096Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0265298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0265361Z return mod(**inputs) 2025-08-14T21:47:46.0265592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0265663Z outputs = self.model( 2025-08-14T21:47:46.0265892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0265967Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0266195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0266262Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0266472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0266542Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0266786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0266901Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0266905Z 2025-08-14T21:47:46.0267001Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0267195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0267257Z return mod(**inputs) 2025-08-14T21:47:46.0267493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0267563Z outputs = self.model( 2025-08-14T21:47:46.0267827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0267904Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0268142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0268237Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0268457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0268532Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0268772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0268891Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0268895Z 2025-08-14T21:47:46.0268990Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0269183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0269245Z return mod(**inputs) 2025-08-14T21:47:46.0269487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0269561Z outputs = self.model( 2025-08-14T21:47:46.0269816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0269906Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0270191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0270261Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0270479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0270554Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0270794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:47:46.0270879Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:46.0270883Z 2025-08-14T21:47:46.0270996Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0271197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0271263Z return mod(**inputs) 2025-08-14T21:47:46.0271509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0271582Z outputs = self.model( 2025-08-14T21:47:46.0271824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0271899Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0272141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0272211Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0272430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0272508Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0272759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0272865Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0273115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:46.0273279Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:46.0273283Z 2025-08-14T21:47:46.0273380Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0273569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0273643Z return mod(**inputs) 2025-08-14T21:47:46.0273889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0273979Z outputs = self.model( 2025-08-14T21:47:46.0274227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0274297Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0274558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0274626Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0274833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0274914Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0275151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0275249Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0275489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:46.0275595Z key_states = self.k_proj(current_states) 2025-08-14T21:47:46.0275598Z 2025-08-14T21:47:46.0275702Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0275889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0275959Z return mod(**inputs) 2025-08-14T21:47:46.0276196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0276259Z outputs = self.model( 2025-08-14T21:47:46.0276503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0276572Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0276845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0276925Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0277135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0277214Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0277450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0277538Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0277783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:46.0277862Z value_states = self.v_proj(current_states) 2025-08-14T21:47:46.0277866Z 2025-08-14T21:47:46.0277949Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0278023Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0278098Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0278177Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0278275Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0278463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0278532Z return mod(**inputs) 2025-08-14T21:47:46.0278769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0278834Z outputs = self.model( 2025-08-14T21:47:46.0279081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0279150Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0279403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0279492Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0279707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0279793Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0280037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0280138Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0280382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0280474Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0280762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:46.0280889Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:46.0280894Z 2025-08-14T21:47:46.0281001Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0281206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0281286Z return mod(**inputs) 2025-08-14T21:47:46.0281544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0281609Z outputs = self.model( 2025-08-14T21:47:46.0281858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0281939Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0282213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0282290Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0282526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0282607Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0282867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0282968Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0283242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0283350Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0283654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:46.0283773Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:46.0283777Z 2025-08-14T21:47:46.0283882Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0284092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0284171Z return mod(**inputs) 2025-08-14T21:47:46.0284441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0284519Z outputs = self.model( 2025-08-14T21:47:46.0284784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0284861Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0285130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0285205Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0285515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0285613Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0285903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0286016Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0286283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:46.0286369Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:46.0286373Z 2025-08-14T21:47:46.0286488Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0286697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0286773Z return mod(**inputs) 2025-08-14T21:47:46.0287041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0287113Z outputs = self.model( 2025-08-14T21:47:46.0287386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0287481Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0288265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0288360Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0288591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0288680Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0288974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0289088Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0289363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:46.0289548Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:46.0289553Z 2025-08-14T21:47:46.0289671Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0289880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0289948Z return mod(**inputs) 2025-08-14T21:47:46.0290221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0290294Z outputs = self.model( 2025-08-14T21:47:46.0290556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0290640Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0290904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0290990Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0291220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0291305Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0291576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0291688Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0291960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:46.0292043Z key_states = self.k_proj(current_states) 2025-08-14T21:47:46.0292047Z 2025-08-14T21:47:46.0292152Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0292366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0292434Z return mod(**inputs) 2025-08-14T21:47:46.0292716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0292801Z outputs = self.model( 2025-08-14T21:47:46.0293062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0293145Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0293407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0293482Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0293720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0293801Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0294064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0294186Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0294493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:46.0294585Z value_states = self.v_proj(current_states) 2025-08-14T21:47:46.0294589Z 2025-08-14T21:47:46.0294669Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0294748Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0294832Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0294906Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0295017Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0295225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0295288Z return mod(**inputs) 2025-08-14T21:47:46.0295550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0295618Z outputs = self.model( 2025-08-14T21:47:46.0295869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0295946Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0296189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0296265Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0296477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0296551Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0296804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0296907Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0297158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0297262Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0297544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:46.0297679Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:46.0297682Z 2025-08-14T21:47:46.0297784Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0297978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0298049Z return mod(**inputs) 2025-08-14T21:47:46.0298293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0298366Z outputs = self.model( 2025-08-14T21:47:46.0298629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0298701Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0298953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0299022Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0299232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0299315Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0299556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0299663Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0299907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0300001Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0300325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:46.0300428Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:46.0300431Z 2025-08-14T21:47:46.0300536Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0300729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0300792Z return mod(**inputs) 2025-08-14T21:47:46.0301043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0301109Z outputs = self.model( 2025-08-14T21:47:46.0301369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0301449Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0301695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0301772Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0301985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0302060Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0302310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0302410Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0302662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:46.0302740Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:46.0302745Z 2025-08-14T21:47:46.0302846Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0303046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0303110Z return mod(**inputs) 2025-08-14T21:47:46.0303358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0303430Z outputs = self.model( 2025-08-14T21:47:46.0303675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0303753Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0304045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0304115Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0304336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0304431Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0304683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 499, in forward 2025-08-14T21:47:46.0304760Z hidden_states = residual + hidden_states 2025-08-14T21:47:46.0304764Z 2025-08-14T21:47:46.0304860Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0305056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0305119Z return mod(**inputs) 2025-08-14T21:47:46.0305361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0305434Z outputs = self.model( 2025-08-14T21:47:46.0305674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0305752Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0306009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0306091Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0306310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0306386Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0306629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0306751Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0306755Z 2025-08-14T21:47:46.0306853Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0307050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0307130Z return mod(**inputs) 2025-08-14T21:47:46.0307375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0307451Z outputs = self.model( 2025-08-14T21:47:46.0307693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0307769Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0308010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0308078Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0308296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0308371Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0308612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0308739Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0308744Z 2025-08-14T21:47:46.0308838Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0309025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0309085Z return mod(**inputs) 2025-08-14T21:47:46.0309312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0309383Z outputs = self.model( 2025-08-14T21:47:46.0309614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0309687Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0309915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0310000Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0310215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0310288Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0310518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:47:46.0310600Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:46.0310603Z 2025-08-14T21:47:46.0310696Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0310886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0310945Z return mod(**inputs) 2025-08-14T21:47:46.0311179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0311250Z outputs = self.model( 2025-08-14T21:47:46.0311485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0311602Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0311841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0311909Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0312125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0312199Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0312440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0312544Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0312811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:46.0312980Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:46.0312986Z 2025-08-14T21:47:46.0313093Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0313304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0313381Z return mod(**inputs) 2025-08-14T21:47:46.0313652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0313727Z outputs = self.model( 2025-08-14T21:47:46.0313984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0314064Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0314329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0314399Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0314610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0314690Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0314935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0315033Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0315269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:46.0315344Z key_states = self.k_proj(current_states) 2025-08-14T21:47:46.0315347Z 2025-08-14T21:47:46.0315450Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0315637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0315720Z return mod(**inputs) 2025-08-14T21:47:46.0315962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0316028Z outputs = self.model( 2025-08-14T21:47:46.0316275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0316344Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0316590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0316664Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0316865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0316942Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0317171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0317261Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0317542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:46.0317623Z value_states = self.v_proj(current_states) 2025-08-14T21:47:46.0317626Z 2025-08-14T21:47:46.0317701Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0317781Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0317853Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0317931Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0318024Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0318207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0318275Z return mod(**inputs) 2025-08-14T21:47:46.0318519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0318584Z outputs = self.model( 2025-08-14T21:47:46.0318825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0318892Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0319128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0319195Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0319397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0319477Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0319708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0319806Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0320039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0320132Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0320407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:46.0320528Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:46.0320531Z 2025-08-14T21:47:46.0320623Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0320810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0320871Z return mod(**inputs) 2025-08-14T21:47:46.0321110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0321189Z outputs = self.model( 2025-08-14T21:47:46.0321427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0321505Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0321741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0321812Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0322018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0322092Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0322340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0322430Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0322672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0322773Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0323081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:46.0323192Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:46.0323195Z 2025-08-14T21:47:46.0323291Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0323478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0323547Z return mod(**inputs) 2025-08-14T21:47:46.0323783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0323856Z outputs = self.model( 2025-08-14T21:47:46.0324114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0324189Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0324440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0324510Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0324723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0324807Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0325049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0325150Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0325463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:46.0325548Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:46.0325556Z 2025-08-14T21:47:46.0325666Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0325868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0325943Z return mod(**inputs) 2025-08-14T21:47:46.0326196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0326266Z outputs = self.model( 2025-08-14T21:47:46.0326525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0326600Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0326854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0326935Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0327163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0327271Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0327517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0327621Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0327875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:46.0328020Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:46.0328024Z 2025-08-14T21:47:46.0328130Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0328322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0328385Z return mod(**inputs) 2025-08-14T21:47:46.0328639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0328722Z outputs = self.model( 2025-08-14T21:47:46.0328985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0329064Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0329309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0329383Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0329597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0329673Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0329923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0330043Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0330291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:46.0330378Z key_states = self.k_proj(current_states) 2025-08-14T21:47:46.0330382Z 2025-08-14T21:47:46.0330479Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0330675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0330739Z return mod(**inputs) 2025-08-14T21:47:46.0330984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0331058Z outputs = self.model( 2025-08-14T21:47:46.0331305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0331383Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0331632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0331704Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0331926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0332001Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0332241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0332353Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0332595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:46.0332683Z value_states = self.v_proj(current_states) 2025-08-14T21:47:46.0332686Z 2025-08-14T21:47:46.0332764Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0332863Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0332947Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0333022Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0333121Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0333318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0333381Z return mod(**inputs) 2025-08-14T21:47:46.0333634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0333699Z outputs = self.model( 2025-08-14T21:47:46.0333945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0334026Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0334271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0334347Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0334606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0334681Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0334933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0335033Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0335277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0335377Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0335657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:46.0335802Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:46.0335807Z 2025-08-14T21:47:46.0335905Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0336098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0336168Z return mod(**inputs) 2025-08-14T21:47:46.0336411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0336482Z outputs = self.model( 2025-08-14T21:47:46.0336723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0336793Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0337040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0337110Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0337326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0337411Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0337835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0337961Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0338197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0338289Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0338567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:46.0338670Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:46.0338674Z 2025-08-14T21:47:46.0338780Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0339012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0339077Z return mod(**inputs) 2025-08-14T21:47:46.0339325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0339390Z outputs = self.model( 2025-08-14T21:47:46.0339629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0339708Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0339949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0340027Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0340236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0340313Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0340592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0340711Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0340946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:46.0341020Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:46.0341024Z 2025-08-14T21:47:46.0341118Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0341307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0341368Z return mod(**inputs) 2025-08-14T21:47:46.0341597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0341691Z outputs = self.model( 2025-08-14T21:47:46.0341924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0342001Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0342235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0342303Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0342517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0342592Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0342827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0342949Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0342952Z 2025-08-14T21:47:46.0343051Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0343243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0343309Z return mod(**inputs) 2025-08-14T21:47:46.0343546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0343617Z outputs = self.model( 2025-08-14T21:47:46.0343852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0343926Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0344172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0344237Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0344443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0344532Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0344766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0344881Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0344885Z 2025-08-14T21:47:46.0344978Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0345170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0345231Z return mod(**inputs) 2025-08-14T21:47:46.0345469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0345541Z outputs = self.model( 2025-08-14T21:47:46.0345782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0345859Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0346093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0346192Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0346409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0346493Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0346725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:47:46.0346808Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:46.0346811Z 2025-08-14T21:47:46.0346904Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0347100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0347161Z return mod(**inputs) 2025-08-14T21:47:46.0347425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0347500Z outputs = self.model( 2025-08-14T21:47:46.0347740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0347814Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0348051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0348119Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0348332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0348404Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0348639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-08-14T21:47:46.0348732Z hidden_states = residual + hidden_states 2025-08-14T21:47:46.0348735Z 2025-08-14T21:47:46.0348827Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0349017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0349078Z return mod(**inputs) 2025-08-14T21:47:46.0349307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0349375Z outputs = self.model( 2025-08-14T21:47:46.0349604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0349670Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0349906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0349973Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0350208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0350283Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0350520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0350619Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0350857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:46.0351002Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:46.0351005Z 2025-08-14T21:47:46.0351100Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0351285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0351354Z return mod(**inputs) 2025-08-14T21:47:46.0351596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0351691Z outputs = self.model( 2025-08-14T21:47:46.0351935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0352001Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0352241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0352310Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0352520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0352603Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0352859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0352967Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0353213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:46.0353291Z key_states = self.k_proj(current_states) 2025-08-14T21:47:46.0353294Z 2025-08-14T21:47:46.0353399Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0353589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0353653Z return mod(**inputs) 2025-08-14T21:47:46.0353907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0353972Z outputs = self.model( 2025-08-14T21:47:46.0354224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0354293Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0354522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0354599Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0354802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0354881Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0355115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0355209Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0355458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:46.0355538Z value_states = self.v_proj(current_states) 2025-08-14T21:47:46.0355542Z 2025-08-14T21:47:46.0355646Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0355731Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0355807Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0355890Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0355989Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0356180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0356250Z return mod(**inputs) 2025-08-14T21:47:46.0356495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0356559Z outputs = self.model( 2025-08-14T21:47:46.0356809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0356878Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0357129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0357201Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0357451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0357534Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0357779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0357872Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0358126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0358218Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0358506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:46.0358650Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:46.0358653Z 2025-08-14T21:47:46.0358756Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0358957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0359020Z return mod(**inputs) 2025-08-14T21:47:46.0359272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0359339Z outputs = self.model( 2025-08-14T21:47:46.0359583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0359663Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0359907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0359978Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0360199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0360276Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0360527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0360618Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0360860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0360960Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0361239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:46.0361348Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:46.0361352Z 2025-08-14T21:47:46.0361472Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0361665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0361738Z return mod(**inputs) 2025-08-14T21:47:46.0361983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0362050Z outputs = self.model( 2025-08-14T21:47:46.0362299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0362370Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0362620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0362690Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0362905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0362988Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0363265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0363366Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0363607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:46.0363684Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:46.0363687Z 2025-08-14T21:47:46.0363792Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0363987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0364052Z return mod(**inputs) 2025-08-14T21:47:46.0364307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0364390Z outputs = self.model( 2025-08-14T21:47:46.0364649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0364722Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0364969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0365048Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0365273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0365426Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0365707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0365834Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0366114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:46.0366273Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:46.0366278Z 2025-08-14T21:47:46.0366388Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0366603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0366666Z return mod(**inputs) 2025-08-14T21:47:46.0366924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0366990Z outputs = self.model( 2025-08-14T21:47:46.0367237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0367318Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0367565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0367663Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0367884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0367959Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0368214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0368318Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0368566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:46.0368651Z key_states = self.k_proj(current_states) 2025-08-14T21:47:46.0368654Z 2025-08-14T21:47:46.0368752Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0369022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0369116Z return mod(**inputs) 2025-08-14T21:47:46.0369529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0369629Z outputs = self.model( 2025-08-14T21:47:46.0369983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0370067Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0370309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0370381Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0370600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0370676Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0370961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0371085Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0371353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:46.0371465Z value_states = self.v_proj(current_states) 2025-08-14T21:47:46.0371473Z 2025-08-14T21:47:46.0371595Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0371682Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0371771Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0371849Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0371953Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0372172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0372242Z return mod(**inputs) 2025-08-14T21:47:46.0372524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0372600Z outputs = self.model( 2025-08-14T21:47:46.0372877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0372962Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0373225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0373299Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0373534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0373614Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0373888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0374022Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0374290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0374400Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0374703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:46.0374841Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:46.0374845Z 2025-08-14T21:47:46.0374951Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0375158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0375234Z return mod(**inputs) 2025-08-14T21:47:46.0375503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0375576Z outputs = self.model( 2025-08-14T21:47:46.0375870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0375966Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0376240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0376315Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0376545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0376637Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0376893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0377002Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0377285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0377380Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0377660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:46.0377760Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:46.0377763Z 2025-08-14T21:47:46.0377868Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0378054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0378115Z return mod(**inputs) 2025-08-14T21:47:46.0378358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0378421Z outputs = self.model( 2025-08-14T21:47:46.0378671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0378753Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0379003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0379081Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0379299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0379377Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0379635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0379739Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0379987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:46.0380091Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:46.0380095Z 2025-08-14T21:47:46.0380196Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0380404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0380470Z return mod(**inputs) 2025-08-14T21:47:46.0380722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0380795Z outputs = self.model( 2025-08-14T21:47:46.0381047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0381127Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0381377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0381451Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0381753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0381894Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0382152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0382320Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0382324Z 2025-08-14T21:47:46.0382428Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0382635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0382701Z return mod(**inputs) 2025-08-14T21:47:46.0382966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0383045Z outputs = self.model( 2025-08-14T21:47:46.0383350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0383437Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0383720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0383796Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0384039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0384122Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0384410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0384545Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0384549Z 2025-08-14T21:47:46.0384658Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0384887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0384958Z return mod(**inputs) 2025-08-14T21:47:46.0385238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0385317Z outputs = self.model( 2025-08-14T21:47:46.0385591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0385675Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0385949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0386024Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0386261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0386337Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0386602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:47:46.0386691Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:46.0386695Z 2025-08-14T21:47:46.0386792Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0386986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0387048Z return mod(**inputs) 2025-08-14T21:47:46.0387300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0387378Z outputs = self.model( 2025-08-14T21:47:46.0387651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0387728Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0388021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0388113Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0388366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0388449Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0388729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0388840Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0389108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:46.0389271Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:46.0389275Z 2025-08-14T21:47:46.0389380Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0389606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0389685Z return mod(**inputs) 2025-08-14T21:47:46.0389970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0390041Z outputs = self.model( 2025-08-14T21:47:46.0390315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0390391Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0390666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0390743Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0390977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0391071Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0391337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0391451Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0391719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:46.0391804Z key_states = self.k_proj(current_states) 2025-08-14T21:47:46.0391808Z 2025-08-14T21:47:46.0391921Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0392130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0392200Z return mod(**inputs) 2025-08-14T21:47:46.0392475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0392545Z outputs = self.model( 2025-08-14T21:47:46.0392839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0392917Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0393188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0393270Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0393503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0393587Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0393855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0393954Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0394231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:46.0394323Z value_states = self.v_proj(current_states) 2025-08-14T21:47:46.0394342Z 2025-08-14T21:47:46.0394449Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0394541Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0394623Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0394708Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0394814Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0395020Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0395092Z return mod(**inputs) 2025-08-14T21:47:46.0395362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0395433Z outputs = self.model( 2025-08-14T21:47:46.0395719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0395796Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0396059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0396130Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0396357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0396440Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0396686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0396779Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0397034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0397127Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0397419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:46.0397548Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:46.0397552Z 2025-08-14T21:47:46.0397651Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0397851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0397914Z return mod(**inputs) 2025-08-14T21:47:46.0398169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0398237Z outputs = self.model( 2025-08-14T21:47:46.0398480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0398556Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0398818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0398890Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0399110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0399187Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0399446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0399541Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0399795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0399902Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0400196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:46.0400317Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:46.0400335Z 2025-08-14T21:47:46.0400460Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0400658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0400732Z return mod(**inputs) 2025-08-14T21:47:46.0400982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0401049Z outputs = self.model( 2025-08-14T21:47:46.0401305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0401376Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0401631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0401721Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0401943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0402030Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0402276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0402378Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0402626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:46.0402706Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:46.0402709Z 2025-08-14T21:47:46.0402816Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0403014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0403084Z return mod(**inputs) 2025-08-14T21:47:46.0403337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0403411Z outputs = self.model( 2025-08-14T21:47:46.0403683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0403761Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0404021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0404104Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0404333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0404421Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0404688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 482, in forward 2025-08-14T21:47:46.0404785Z hidden_states = residual + hidden_states 2025-08-14T21:47:46.0404790Z 2025-08-14T21:47:46.0404900Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0405099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0405169Z return mod(**inputs) 2025-08-14T21:47:46.0405614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0405693Z outputs = self.model( 2025-08-14T21:47:46.0405970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0406046Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0406310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0406401Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0406644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0406758Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0407016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0407121Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0407373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:46.0407517Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:46.0407521Z 2025-08-14T21:47:46.0407619Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0407817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0407897Z return mod(**inputs) 2025-08-14T21:47:46.0408154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0408219Z outputs = self.model( 2025-08-14T21:47:46.0408461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0408538Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0408781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0408858Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0409068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0409141Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0409389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0409490Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0409734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:46.0409818Z key_states = self.k_proj(current_states) 2025-08-14T21:47:46.0409821Z 2025-08-14T21:47:46.0409918Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0410114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0410177Z return mod(**inputs) 2025-08-14T21:47:46.0410418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0410489Z outputs = self.model( 2025-08-14T21:47:46.0410730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0410817Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0411072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0411141Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0411359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0411435Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0411678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0411795Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0412033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:46.0412119Z value_states = self.v_proj(current_states) 2025-08-14T21:47:46.0412125Z 2025-08-14T21:47:46.0412201Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0412291Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0412391Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0412464Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0412559Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0412754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0412816Z return mod(**inputs) 2025-08-14T21:47:46.0413061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0413125Z outputs = self.model( 2025-08-14T21:47:46.0413365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0413442Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0413698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0413773Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0413998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0414071Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0414320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0414423Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0414664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0414765Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0415048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:46.0415180Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:46.0415185Z 2025-08-14T21:47:46.0415282Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0415469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0415537Z return mod(**inputs) 2025-08-14T21:47:46.0415771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0415835Z outputs = self.model( 2025-08-14T21:47:46.0416079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0416148Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0416390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0416475Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0416686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0416768Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0417006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0417113Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0417350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0417441Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0417721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:46.0417823Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:46.0417828Z 2025-08-14T21:47:46.0417923Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0418146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0418211Z return mod(**inputs) 2025-08-14T21:47:46.0418460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0418524Z outputs = self.model( 2025-08-14T21:47:46.0418762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0418839Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0419076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0419152Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0419375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0419451Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0419697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0419798Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0420037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:46.0420120Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:46.0420124Z 2025-08-14T21:47:46.0420218Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0420410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0420474Z return mod(**inputs) 2025-08-14T21:47:46.0420713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0420785Z outputs = self.model( 2025-08-14T21:47:46.0421026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0421102Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0421339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0421405Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0421619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0421690Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0421924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0422046Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0422064Z 2025-08-14T21:47:46.0422161Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0422357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0422421Z return mod(**inputs) 2025-08-14T21:47:46.0422664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0422736Z outputs = self.model( 2025-08-14T21:47:46.0422979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0423049Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0423297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0423367Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0423589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0423692Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0423947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0424070Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0424073Z 2025-08-14T21:47:46.0424172Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0424368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0424432Z return mod(**inputs) 2025-08-14T21:47:46.0424678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0424750Z outputs = self.model( 2025-08-14T21:47:46.0425016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0425089Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0425333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0425402Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0425611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0425684Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0425918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:47:46.0426002Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:46.0426005Z 2025-08-14T21:47:46.0426099Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0426294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0426358Z return mod(**inputs) 2025-08-14T21:47:46.0426595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0426666Z outputs = self.model( 2025-08-14T21:47:46.0426901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0426969Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0427212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0427279Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0427491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0427564Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0427804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0427932Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0428179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:46.0428330Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:46.0428334Z 2025-08-14T21:47:46.0428433Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0428623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0428695Z return mod(**inputs) 2025-08-14T21:47:46.0428939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0429005Z outputs = self.model( 2025-08-14T21:47:46.0429267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0429352Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0429615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0429687Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0429901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0429984Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0430229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0430340Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0430583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:46.0430678Z key_states = self.k_proj(current_states) 2025-08-14T21:47:46.0430681Z 2025-08-14T21:47:46.0430788Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0430980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0431043Z return mod(**inputs) 2025-08-14T21:47:46.0431295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0431359Z outputs = self.model( 2025-08-14T21:47:46.0431606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0431675Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0431917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0431995Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0432208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0432286Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0432545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0432647Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0432918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:46.0433004Z value_states = self.v_proj(current_states) 2025-08-14T21:47:46.0433008Z 2025-08-14T21:47:46.0433091Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0433182Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0433262Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0433348Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0433474Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0433681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0433762Z return mod(**inputs) 2025-08-14T21:47:46.0434029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0434100Z outputs = self.model( 2025-08-14T21:47:46.0434374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0434449Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0434717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0434790Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0435032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0435117Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0435382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0435511Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0435769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0435862Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0436159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:46.0436289Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:46.0436292Z 2025-08-14T21:47:46.0436393Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0436611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0436681Z return mod(**inputs) 2025-08-14T21:47:46.0436947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0437016Z outputs = self.model( 2025-08-14T21:47:46.0437268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0437346Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0437798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0437913Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0438207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0438286Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0438552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0438653Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0438918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0439025Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0439326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:46.0439449Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:46.0439454Z 2025-08-14T21:47:46.0439559Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0439764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0439843Z return mod(**inputs) 2025-08-14T21:47:46.0440177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0440252Z outputs = self.model( 2025-08-14T21:47:46.0440526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0440602Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0440876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0440954Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0441188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0441286Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0441537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0441646Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0441930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:46.0442042Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:46.0442046Z 2025-08-14T21:47:46.0442160Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0442369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0442438Z return mod(**inputs) 2025-08-14T21:47:46.0442713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0442785Z outputs = self.model( 2025-08-14T21:47:46.0443055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0443154Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0443425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0443511Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0443748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0443837Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0444106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0444218Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0444499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:46.0444647Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:46.0444651Z 2025-08-14T21:47:46.0444757Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0444961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0445030Z return mod(**inputs) 2025-08-14T21:47:46.0445305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0445455Z outputs = self.model( 2025-08-14T21:47:46.0445720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0445804Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0446070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0446152Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0446391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0446514Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0446773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0446876Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0447121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:46.0447209Z key_states = self.k_proj(current_states) 2025-08-14T21:47:46.0447213Z 2025-08-14T21:47:46.0447313Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0447516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0447582Z return mod(**inputs) 2025-08-14T21:47:46.0447830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0447907Z outputs = self.model( 2025-08-14T21:47:46.0448171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0448268Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0448515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0448584Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0448805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0448880Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0449128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0449238Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0449497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:46.0449590Z value_states = self.v_proj(current_states) 2025-08-14T21:47:46.0449595Z 2025-08-14T21:47:46.0449671Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0449747Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0449831Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0449904Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0450002Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0450202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0450265Z return mod(**inputs) 2025-08-14T21:47:46.0450516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0450582Z outputs = self.model( 2025-08-14T21:47:46.0450823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0450905Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0451151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0451221Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0451441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0451515Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0464563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0464681Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0464944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0465136Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0465426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:46.0465568Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:46.0465575Z 2025-08-14T21:47:46.0465684Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0465890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0465961Z return mod(**inputs) 2025-08-14T21:47:46.0466209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0466291Z outputs = self.model( 2025-08-14T21:47:46.0466535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0466615Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0466890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0466987Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0467212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0467294Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0467532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0467645Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0467884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0467988Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0468286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:46.0468400Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:46.0468405Z 2025-08-14T21:47:46.0468517Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0468711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0468778Z return mod(**inputs) 2025-08-14T21:47:46.0469025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0469092Z outputs = self.model( 2025-08-14T21:47:46.0469338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0469413Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0469654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0469737Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0469952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0470036Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0470275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0470374Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0470619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:46.0470697Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:46.0470701Z 2025-08-14T21:47:46.0470801Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0470999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0471082Z return mod(**inputs) 2025-08-14T21:47:46.0471338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0471405Z outputs = self.model( 2025-08-14T21:47:46.0471640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0471718Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0471955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0472034Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0472243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0472319Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0472573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 499, in forward 2025-08-14T21:47:46.0472669Z hidden_states = residual + hidden_states 2025-08-14T21:47:46.0472686Z 2025-08-14T21:47:46.0472788Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0472987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0473052Z return mod(**inputs) 2025-08-14T21:47:46.0473301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0473377Z outputs = self.model( 2025-08-14T21:47:46.0473614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0473685Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0473953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0474020Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0474223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0474297Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0474523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0474638Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0474643Z 2025-08-14T21:47:46.0474733Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0474915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0474985Z return mod(**inputs) 2025-08-14T21:47:46.0475216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0475290Z outputs = self.model( 2025-08-14T21:47:46.0475521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0475590Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0475831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0475900Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0476109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0476190Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0476427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0476547Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0476567Z 2025-08-14T21:47:46.0476666Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0476855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0476928Z return mod(**inputs) 2025-08-14T21:47:46.0477165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0477240Z outputs = self.model( 2025-08-14T21:47:46.0477479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0477558Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0477798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0477865Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0478068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0478151Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0478440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:47:46.0478524Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:46.0478528Z 2025-08-14T21:47:46.0478623Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0478809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0478879Z return mod(**inputs) 2025-08-14T21:47:46.0479118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0479188Z outputs = self.model( 2025-08-14T21:47:46.0479424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0479509Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0479756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0479824Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0480027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0480107Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0480341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0480443Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0480678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:46.0480821Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:46.0480827Z 2025-08-14T21:47:46.0480931Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0481122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0481191Z return mod(**inputs) 2025-08-14T21:47:46.0481429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0481493Z outputs = self.model( 2025-08-14T21:47:46.0481744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0481815Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0482063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0482143Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0482372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0482476Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0482743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0482848Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0483121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:46.0483203Z key_states = self.k_proj(current_states) 2025-08-14T21:47:46.0483207Z 2025-08-14T21:47:46.0483320Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0483529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0483593Z return mod(**inputs) 2025-08-14T21:47:46.0483848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0483917Z outputs = self.model( 2025-08-14T21:47:46.0484207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0484287Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0484537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0484615Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0484832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0484907Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0485163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0485260Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0485717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:46.0485829Z value_states = self.v_proj(current_states) 2025-08-14T21:47:46.0485833Z 2025-08-14T21:47:46.0485924Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0486021Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0486104Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0486188Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0486305Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0486530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0486595Z return mod(**inputs) 2025-08-14T21:47:46.0486850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0486917Z outputs = self.model( 2025-08-14T21:47:46.0487179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0487251Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0487488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0487566Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0487796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0487891Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0488161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0488266Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0488546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0488672Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0488989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:46.0489140Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:46.0489144Z 2025-08-14T21:47:46.0489253Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0489472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0489545Z return mod(**inputs) 2025-08-14T21:47:46.0489817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0489899Z outputs = self.model( 2025-08-14T21:47:46.0490170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0490259Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0490545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0490640Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0490890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0490974Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0491256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0491368Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0491650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0491762Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0492101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:46.0492224Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:46.0492228Z 2025-08-14T21:47:46.0492347Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0492562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0492641Z return mod(**inputs) 2025-08-14T21:47:46.0492913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0492987Z outputs = self.model( 2025-08-14T21:47:46.0493269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0493349Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0493674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0493753Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0493959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0494040Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0494277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 473, in forward 2025-08-14T21:47:46.0494366Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:47:46.0494612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:46.0494688Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:46.0494691Z 2025-08-14T21:47:46.0494795Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0495002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0495065Z return mod(**inputs) 2025-08-14T21:47:46.0495314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0495378Z outputs = self.model( 2025-08-14T21:47:46.0495614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0495691Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0495928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0496003Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0496210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0496283Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0496530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0496661Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0496899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 281, in forward 2025-08-14T21:47:46.0497050Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:47:46.0497054Z 2025-08-14T21:47:46.0497149Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0497341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0497404Z return mod(**inputs) 2025-08-14T21:47:46.0497642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0497714Z outputs = self.model( 2025-08-14T21:47:46.0497968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0498047Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0498285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0498352Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0498564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0498638Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0498874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0498983Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0499218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 300, in forward 2025-08-14T21:47:46.0499306Z key_states = self.k_proj(current_states) 2025-08-14T21:47:46.0499310Z 2025-08-14T21:47:46.0499406Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0499595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0499663Z return mod(**inputs) 2025-08-14T21:47:46.0499900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0499970Z outputs = self.model( 2025-08-14T21:47:46.0500205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0500275Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0500520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0500605Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0500813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0500895Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0501134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0501238Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0501476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 301, in forward 2025-08-14T21:47:46.0501556Z value_states = self.v_proj(current_states) 2025-08-14T21:47:46.0501560Z 2025-08-14T21:47:46.0501643Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0501718Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0501798Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0501872Z cudagraph partition due to non gpu ops 2025-08-14T21:47:46.0501968Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0502191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0502255Z return mod(**inputs) 2025-08-14T21:47:46.0502498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0502572Z outputs = self.model( 2025-08-14T21:47:46.0502813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0502892Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0503141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0503209Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0503435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0503512Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0503751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0503858Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0504095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0504194Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0504465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:47:46.0504590Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:47:46.0504594Z 2025-08-14T21:47:46.0504695Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0504884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0504956Z return mod(**inputs) 2025-08-14T21:47:46.0505196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0505261Z outputs = self.model( 2025-08-14T21:47:46.0505509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0505578Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0505816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0505893Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0506099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0506201Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0506438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0506540Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0506785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 319, in forward 2025-08-14T21:47:46.0506876Z attn_output, attn_weights = attention_interface( 2025-08-14T21:47:46.0507160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:47:46.0507260Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:47:46.0507263Z 2025-08-14T21:47:46.0507358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0507552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0507616Z return mod(**inputs) 2025-08-14T21:47:46.0507881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0507968Z outputs = self.model( 2025-08-14T21:47:46.0508206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0508282Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0508520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0508588Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0508801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0508874Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0509132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 490, in forward 2025-08-14T21:47:46.0509234Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:47:46.0509470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 333, in forward 2025-08-14T21:47:46.0509555Z attn_output = self.out_proj(attn_output) 2025-08-14T21:47:46.0509559Z 2025-08-14T21:47:46.0509655Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0509842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0509911Z return mod(**inputs) 2025-08-14T21:47:46.0510151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0510221Z outputs = self.model( 2025-08-14T21:47:46.0510456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0510527Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0510771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0510841Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0511045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0511126Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0511360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0511480Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0511484Z 2025-08-14T21:47:46.0511580Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0511766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0511859Z return mod(**inputs) 2025-08-14T21:47:46.0512100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0512171Z outputs = self.model( 2025-08-14T21:47:46.0512411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0512479Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0512722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0512789Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0512997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0513077Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0513311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 504, in forward 2025-08-14T21:47:46.0513430Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:47:46.0513448Z 2025-08-14T21:47:46.0513560Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0513747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0513818Z return mod(**inputs) 2025-08-14T21:47:46.0514052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0514122Z outputs = self.model( 2025-08-14T21:47:46.0514354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0514421Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0514674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0514744Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0514958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0515039Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0515277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 506, in forward 2025-08-14T21:47:46.0515361Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:47:46.0515364Z 2025-08-14T21:47:46.0515459Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0515643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0515710Z return mod(**inputs) 2025-08-14T21:47:46.0515949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1404, in forward 2025-08-14T21:47:46.0516021Z outputs = self.model( 2025-08-14T21:47:46.0516259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1279, in forward 2025-08-14T21:47:46.0516329Z decoder_outputs = self.decoder( 2025-08-14T21:47:46.0516578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1127, in forward 2025-08-14T21:47:46.0516645Z layer_outputs = decoder_layer( 2025-08-14T21:47:46.0516852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:47:46.0516933Z return super().__call__(*args, **kwargs) 2025-08-14T21:47:46.0517181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 508, in forward 2025-08-14T21:47:46.0517263Z hidden_states = residual + hidden_states 2025-08-14T21:47:46.0517266Z 2025-08-14T21:47:46.0517366Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0517571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0517643Z return mod(**inputs) 2025-08-14T21:47:46.0517885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1422, in forward 2025-08-14T21:47:46.0517962Z lm_logits = self.lm_head(outputs[0]) 2025-08-14T21:47:46.0517974Z 2025-08-14T21:47:46.0518071Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:47:46.0518259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:47:46.0518331Z return mod(**inputs) 2025-08-14T21:47:46.0518575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/m2m_100/modeling_m2m_100.py", line 1429, in forward 2025-08-14T21:47:46.0518737Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:47:46.0518742Z 2025-08-14T21:47:57.8006318Z Compilation time (from dynamo_timed): 26.835130424 2025-08-14T21:47:57.8105371Z pass 2025-08-14T21:47:57.8113279Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:47:57.8118413Z TIMING: _recursive_pre_grad_passes:0.01403 _recursive_joint_graph_passes:1.14818 _recursive_post_grad_passes:0.16937 async_compile.wait:0.79149 code_gen:11.43334 inductor_compile:14.38776 backend_compile:21.37095 gc:0.00026 entire_frame_compile:26.83513 total_wall_time:26.83513 2025-08-14T21:47:57.8119615Z STATS: call_* op count: 1014 | FakeTensorMode.__torch_dispatch__:33764 | FakeTensor.__torch_dispatch__:11261 | ProxyTorchDispatchMode.__torch_dispatch__:12417 2025-08-14T21:47:57.8120246Z Dynamo produced 1 graphs covering 1014 ops with 0 graph breaks (0 unique) 2025-08-14T21:48:03.5388081Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:48:03.5388996Z from pkg_resources import resource_filename 2025-08-14T21:48:04.1382983Z 2025-08-14T21:48:07.0369982Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:48:07.0370463Z loading model: 0it [00:02, ?it/s] 2025-08-14T21:48:07.0394358Z cpu eval MBartForCausalLM 2025-08-14T21:48:08.6762100Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:48:09.3004522Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:48:09.9247353Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:48:17.3427388Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3428008Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3428345Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3428589Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3428918Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3429247Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3430036Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3430387Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3430605Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3430806Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3431022Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3431221Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3431449Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3431826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3432159Z return mod(**inputs) 2025-08-14T21:48:17.3432570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3433338Z outputs = self.model.decoder( 2025-08-14T21:48:17.3433756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3434166Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3434576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3434952Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3435344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3435761Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3436167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:17.3436636Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:17.3436839Z 2025-08-14T21:48:17.3436951Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3437450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3438037Z return mod(**inputs) 2025-08-14T21:48:17.3438427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3438826Z outputs = self.model.decoder( 2025-08-14T21:48:17.3439207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3439598Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3439953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3440339Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3440819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3441275Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3441736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:17.3442168Z key_states = self.k_proj(current_states) 2025-08-14T21:48:17.3442313Z 2025-08-14T21:48:17.3442425Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3442814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3443168Z return mod(**inputs) 2025-08-14T21:48:17.3443553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3443984Z outputs = self.model.decoder( 2025-08-14T21:48:17.3444406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3444815Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3445177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3445754Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3446171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3446612Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3447027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:17.3447430Z value_states = self.v_proj(current_states) 2025-08-14T21:48:17.3447572Z 2025-08-14T21:48:17.3447660Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3447871Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3448128Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3448334Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3448561Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3449035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3449364Z return mod(**inputs) 2025-08-14T21:48:17.3449733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3450117Z outputs = self.model.decoder( 2025-08-14T21:48:17.3450495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3450883Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3451237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3451627Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3452039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3452547Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3452970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3453401Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3453868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:17.3454375Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:17.3454569Z 2025-08-14T21:48:17.3454678Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3455055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3455393Z return mod(**inputs) 2025-08-14T21:48:17.3455790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3456211Z outputs = self.model.decoder( 2025-08-14T21:48:17.3456613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3457046Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3457418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3457804Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3458216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3458651Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3459081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3459520Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3459992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:17.3460476Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:17.3460652Z 2025-08-14T21:48:17.3460761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3461171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3461513Z return mod(**inputs) 2025-08-14T21:48:17.3461901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3462327Z outputs = self.model.decoder( 2025-08-14T21:48:17.3462728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3463209Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3463575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3463957Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3464356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3464787Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3465220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:17.3465608Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:17.3465741Z 2025-08-14T21:48:17.3465843Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3466190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3466522Z return mod(**inputs) 2025-08-14T21:48:17.3466895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3467345Z outputs = self.model.decoder( 2025-08-14T21:48:17.3467756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3468166Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3468522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3468901Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3469303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3469749Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3469936Z 2025-08-14T21:48:17.3470062Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3470437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3470767Z return mod(**inputs) 2025-08-14T21:48:17.3471142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3471552Z outputs = self.model.decoder( 2025-08-14T21:48:17.3471982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3472393Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3472751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3473131Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3473539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3473991Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3474408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:17.3474768Z return self.act(input) 2025-08-14T21:48:17.3474879Z 2025-08-14T21:48:17.3474988Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3475337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3475662Z return mod(**inputs) 2025-08-14T21:48:17.3476048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3476455Z outputs = self.model.decoder( 2025-08-14T21:48:17.3476878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3477304Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3477701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3478077Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3478488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:48:17.3478907Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:17.3479054Z 2025-08-14T21:48:17.3479171Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3479549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3479874Z return mod(**inputs) 2025-08-14T21:48:17.3480257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3480673Z outputs = self.model.decoder( 2025-08-14T21:48:17.3481080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3481496Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3481927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3482307Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3482729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3483176Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3483607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:17.3484105Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:17.3484332Z 2025-08-14T21:48:17.3484446Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3484844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3485183Z return mod(**inputs) 2025-08-14T21:48:17.3485682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3486102Z outputs = self.model.decoder( 2025-08-14T21:48:17.3486517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3486967Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3487334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3487734Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3488139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3488576Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3489023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:17.3489449Z key_states = self.k_proj(current_states) 2025-08-14T21:48:17.3489585Z 2025-08-14T21:48:17.3489688Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3490046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3490371Z return mod(**inputs) 2025-08-14T21:48:17.3490729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3491117Z outputs = self.model.decoder( 2025-08-14T21:48:17.3491494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3491882Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3492222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3492620Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3493010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3493423Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3493822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:17.3494224Z value_states = self.v_proj(current_states) 2025-08-14T21:48:17.3494367Z 2025-08-14T21:48:17.3494461Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3494673Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3494883Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3495091Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3495324Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3495675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3496013Z return mod(**inputs) 2025-08-14T21:48:17.3496387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3496768Z outputs = self.model.decoder( 2025-08-14T21:48:17.3497154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3497542Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3497896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3498258Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3498657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3499096Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3499512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3499953Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3500425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:17.3500928Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:17.3501113Z 2025-08-14T21:48:17.3501216Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3501575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3501903Z return mod(**inputs) 2025-08-14T21:48:17.3502273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3502670Z outputs = self.model.decoder( 2025-08-14T21:48:17.3503051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3503440Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3503781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3504268Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3504661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3505098Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3505528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3505939Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3506418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:17.3506937Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:17.3507110Z 2025-08-14T21:48:17.3507219Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3507596Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3507944Z return mod(**inputs) 2025-08-14T21:48:17.3508379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3508806Z outputs = self.model.decoder( 2025-08-14T21:48:17.3509219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3509745Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3510116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3510501Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3511512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3511944Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3512394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:17.3512836Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:17.3512985Z 2025-08-14T21:48:17.3513106Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3513488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3513843Z return mod(**inputs) 2025-08-14T21:48:17.3514230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3514663Z outputs = self.model.decoder( 2025-08-14T21:48:17.3515077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3515494Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3515862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3516249Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3516671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3517107Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3517282Z 2025-08-14T21:48:17.3517390Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3517751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3518081Z return mod(**inputs) 2025-08-14T21:48:17.3518449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3518861Z outputs = self.model.decoder( 2025-08-14T21:48:17.3519260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3519675Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3520042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3520415Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3520829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3521292Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3521703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:17.3522084Z return self.act(input) 2025-08-14T21:48:17.3522206Z 2025-08-14T21:48:17.3522317Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3522694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3523027Z return mod(**inputs) 2025-08-14T21:48:17.3523406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3523815Z outputs = self.model.decoder( 2025-08-14T21:48:17.3524339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3524768Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3525134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3525595Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3526012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:48:17.3526474Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:17.3526630Z 2025-08-14T21:48:17.3526741Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3527126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3527467Z return mod(**inputs) 2025-08-14T21:48:17.3527857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3528280Z outputs = self.model.decoder( 2025-08-14T21:48:17.3528684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3529109Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3529506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3529869Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3530251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-14T21:48:17.3530648Z hidden_states = residual + hidden_states 2025-08-14T21:48:17.3530790Z 2025-08-14T21:48:17.3530903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3531263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3531579Z return mod(**inputs) 2025-08-14T21:48:17.3531938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3532325Z outputs = self.model.decoder( 2025-08-14T21:48:17.3532700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3533094Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3533449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3533810Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3534191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3534603Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3535024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:17.3535514Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:17.3535728Z 2025-08-14T21:48:17.3535837Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3536228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3536577Z return mod(**inputs) 2025-08-14T21:48:17.3536948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3537360Z outputs = self.model.decoder( 2025-08-14T21:48:17.3537958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3538392Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3538761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3539152Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3539573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3540005Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3540455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:17.3540944Z key_states = self.k_proj(current_states) 2025-08-14T21:48:17.3541117Z 2025-08-14T21:48:17.3541237Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3541611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3541953Z return mod(**inputs) 2025-08-14T21:48:17.3542336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3542838Z outputs = self.model.decoder( 2025-08-14T21:48:17.3543240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3543662Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3544056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3544433Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3544847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3545287Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3545717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:17.3546093Z value_states = self.v_proj(current_states) 2025-08-14T21:48:17.3546233Z 2025-08-14T21:48:17.3546311Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3546517Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3546751Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3546952Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3547173Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3547521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3547839Z return mod(**inputs) 2025-08-14T21:48:17.3548205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3548598Z outputs = self.model.decoder( 2025-08-14T21:48:17.3548998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3549416Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3549789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3550166Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3550569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3550985Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3551422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3551826Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3552255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:17.3552733Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:17.3552904Z 2025-08-14T21:48:17.3553012Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3553354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3553671Z return mod(**inputs) 2025-08-14T21:48:17.3554025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3554420Z outputs = self.model.decoder( 2025-08-14T21:48:17.3554788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3555207Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3555553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3555895Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3556274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3556674Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3557069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3557461Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3557902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:17.3558348Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:17.3558508Z 2025-08-14T21:48:17.3558620Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3558966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3559287Z return mod(**inputs) 2025-08-14T21:48:17.3559649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3560039Z outputs = self.model.decoder( 2025-08-14T21:48:17.3560429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3560822Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3561172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3561538Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3561934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3562353Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3562769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:17.3563170Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:17.3563316Z 2025-08-14T21:48:17.3563421Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3563787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3564114Z return mod(**inputs) 2025-08-14T21:48:17.3564492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3564905Z outputs = self.model.decoder( 2025-08-14T21:48:17.3565335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3565750Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3566101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3566472Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3566857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3567293Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3567477Z 2025-08-14T21:48:17.3567583Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3567943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3568268Z return mod(**inputs) 2025-08-14T21:48:17.3568647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3569062Z outputs = self.model.decoder( 2025-08-14T21:48:17.3569463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3569848Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3570192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3570558Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3570939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3571374Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3571758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:17.3572118Z return self.act(input) 2025-08-14T21:48:17.3572232Z 2025-08-14T21:48:17.3572338Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3572703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3573028Z return mod(**inputs) 2025-08-14T21:48:17.3573392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3573767Z outputs = self.model.decoder( 2025-08-14T21:48:17.3574134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3574504Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3574834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3575184Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3575590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:48:17.3575976Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:17.3576110Z 2025-08-14T21:48:17.3576209Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3576554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3576868Z return mod(**inputs) 2025-08-14T21:48:17.3577233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3577619Z outputs = self.model.decoder( 2025-08-14T21:48:17.3578013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3578463Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3578796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3579170Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3579559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3579958Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3580357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:17.3580818Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:17.3581016Z 2025-08-14T21:48:17.3581124Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3581464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3581781Z return mod(**inputs) 2025-08-14T21:48:17.3582136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3582516Z outputs = self.model.decoder( 2025-08-14T21:48:17.3582918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3583298Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3583643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3583989Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3584367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3584768Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3585166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:17.3585552Z key_states = self.k_proj(current_states) 2025-08-14T21:48:17.3585711Z 2025-08-14T21:48:17.3585812Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3586165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3586482Z return mod(**inputs) 2025-08-14T21:48:17.3586828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3587206Z outputs = self.model.decoder( 2025-08-14T21:48:17.3587576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3587937Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3588267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3588611Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3588986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3589381Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3589781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:17.3590171Z value_states = self.v_proj(current_states) 2025-08-14T21:48:17.3590317Z 2025-08-14T21:48:17.3590395Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3590599Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3590796Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3590995Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3591207Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3591548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3591857Z return mod(**inputs) 2025-08-14T21:48:17.3592193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3592592Z outputs = self.model.decoder( 2025-08-14T21:48:17.3592955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3593330Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3593655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3593995Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3594367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3594752Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3595141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3595538Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3595975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:17.3596442Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:17.3596624Z 2025-08-14T21:48:17.3596722Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3597062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3597367Z return mod(**inputs) 2025-08-14T21:48:17.3597705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3598075Z outputs = self.model.decoder( 2025-08-14T21:48:17.3598434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3598810Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3599142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3599497Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3599881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3600266Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3600660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3601051Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3601474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:17.3601901Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:17.3602060Z 2025-08-14T21:48:17.3602164Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3602513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3602823Z return mod(**inputs) 2025-08-14T21:48:17.3603174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3603548Z outputs = self.model.decoder( 2025-08-14T21:48:17.3603916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3604289Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3604625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3604984Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3605472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3605957Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3606396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:17.3606825Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:17.3606971Z 2025-08-14T21:48:17.3607093Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3607447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3607769Z return mod(**inputs) 2025-08-14T21:48:17.3608125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3608499Z outputs = self.model.decoder( 2025-08-14T21:48:17.3608870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3609247Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3609575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3609960Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3610336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3610752Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3610921Z 2025-08-14T21:48:17.3611020Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3611367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3611679Z return mod(**inputs) 2025-08-14T21:48:17.3612024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3612401Z outputs = self.model.decoder( 2025-08-14T21:48:17.3612783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3613162Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3613487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3613838Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3614214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3614633Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3615002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:17.3615331Z return self.act(input) 2025-08-14T21:48:17.3615437Z 2025-08-14T21:48:17.3615545Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3615891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3616203Z return mod(**inputs) 2025-08-14T21:48:17.3616558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3616932Z outputs = self.model.decoder( 2025-08-14T21:48:17.3617291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3617667Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3618006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3618349Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3618726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:48:17.3619108Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:17.3619264Z 2025-08-14T21:48:17.3619371Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3619716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3620039Z return mod(**inputs) 2025-08-14T21:48:17.3620388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3620762Z outputs = self.model.decoder( 2025-08-14T21:48:17.3621123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3621504Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3621828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3622159Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3622525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-14T21:48:17.3622893Z hidden_states = residual + hidden_states 2025-08-14T21:48:17.3623041Z 2025-08-14T21:48:17.3623160Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3623493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3623800Z return mod(**inputs) 2025-08-14T21:48:17.3624137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3624497Z outputs = self.model.decoder( 2025-08-14T21:48:17.3624858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3625226Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3625565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3625900Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3626267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3626655Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3627045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:17.3627472Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:17.3627668Z 2025-08-14T21:48:17.3627765Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3628100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3628398Z return mod(**inputs) 2025-08-14T21:48:17.3628746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3629116Z outputs = self.model.decoder( 2025-08-14T21:48:17.3629474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3629836Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3630165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3630506Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3630862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3631251Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3631635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:17.3632007Z key_states = self.k_proj(current_states) 2025-08-14T21:48:17.3632154Z 2025-08-14T21:48:17.3632255Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3632602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3632913Z return mod(**inputs) 2025-08-14T21:48:17.3633258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3633618Z outputs = self.model.decoder( 2025-08-14T21:48:17.3633978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3634346Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3634665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3635011Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3635381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3635770Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3636181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:17.3636570Z value_states = self.v_proj(current_states) 2025-08-14T21:48:17.3636701Z 2025-08-14T21:48:17.3636786Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3636989Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3637182Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3637377Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3637599Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3638072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3638398Z return mod(**inputs) 2025-08-14T21:48:17.3638814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3639190Z outputs = self.model.decoder( 2025-08-14T21:48:17.3639575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3639976Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3640311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3640648Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3641021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3641422Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3641819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3642211Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3642646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:17.3643114Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:17.3643291Z 2025-08-14T21:48:17.3643389Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3643739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3644056Z return mod(**inputs) 2025-08-14T21:48:17.3644413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3644786Z outputs = self.model.decoder( 2025-08-14T21:48:17.3645163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3645605Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3645987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3646359Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3646736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3647133Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3647526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3647929Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3648363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:17.3648805Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:17.3648962Z 2025-08-14T21:48:17.3649066Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3649418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3649765Z return mod(**inputs) 2025-08-14T21:48:17.3650143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3650529Z outputs = self.model.decoder( 2025-08-14T21:48:17.3650901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3651279Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3651609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3651959Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3652339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3652759Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3653153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:17.3653540Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:17.3653671Z 2025-08-14T21:48:17.3653953Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3654292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3654610Z return mod(**inputs) 2025-08-14T21:48:17.3654969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3655341Z outputs = self.model.decoder( 2025-08-14T21:48:17.3655698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3656065Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3656399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3656744Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3657104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3657509Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3657671Z 2025-08-14T21:48:17.3657774Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3658106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3658419Z return mod(**inputs) 2025-08-14T21:48:17.3658769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3659145Z outputs = self.model.decoder( 2025-08-14T21:48:17.3659535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3659919Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3660264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3660625Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3660997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3661412Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3661789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:17.3662112Z return self.act(input) 2025-08-14T21:48:17.3662225Z 2025-08-14T21:48:17.3662325Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3662675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3662992Z return mod(**inputs) 2025-08-14T21:48:17.3663366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3663739Z outputs = self.model.decoder( 2025-08-14T21:48:17.3664098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3664460Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3664786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3665126Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3665497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:48:17.3665865Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:17.3666003Z 2025-08-14T21:48:17.3666123Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3666470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3666773Z return mod(**inputs) 2025-08-14T21:48:17.3667116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3667481Z outputs = self.model.decoder( 2025-08-14T21:48:17.3667840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3668201Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3668524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3668864Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3669236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3669617Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3670006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:17.3670442Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:17.3670633Z 2025-08-14T21:48:17.3670731Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3671073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3671380Z return mod(**inputs) 2025-08-14T21:48:17.3671724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3672083Z outputs = self.model.decoder( 2025-08-14T21:48:17.3672443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3672870Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3673193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3673537Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3673915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3674294Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3674663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:17.3675028Z key_states = self.k_proj(current_states) 2025-08-14T21:48:17.3675157Z 2025-08-14T21:48:17.3675253Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3675591Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3675890Z return mod(**inputs) 2025-08-14T21:48:17.3676243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3676615Z outputs = self.model.decoder( 2025-08-14T21:48:17.3676954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3677312Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3677630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3677963Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3678319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3678700Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3679097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:17.3679476Z value_states = self.v_proj(current_states) 2025-08-14T21:48:17.3679608Z 2025-08-14T21:48:17.3679686Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3679890Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3680092Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3680284Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3680502Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3680838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3681140Z return mod(**inputs) 2025-08-14T21:48:17.3681486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3681859Z outputs = self.model.decoder( 2025-08-14T21:48:17.3682210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3682566Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3682888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3683220Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3683576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3683962Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3684348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3684740Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3685156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:17.3685733Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:17.3685925Z 2025-08-14T21:48:17.3686033Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3686400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3686720Z return mod(**inputs) 2025-08-14T21:48:17.3687091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3687487Z outputs = self.model.decoder( 2025-08-14T21:48:17.3687842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3688214Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3688557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3688912Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3689290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3689752Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3690137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3690520Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3690928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:17.3691353Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:17.3691505Z 2025-08-14T21:48:17.3691609Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3691939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3692243Z return mod(**inputs) 2025-08-14T21:48:17.3692605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3692976Z outputs = self.model.decoder( 2025-08-14T21:48:17.3693333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3693707Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3694038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3694378Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3694738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3695127Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3695514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:17.3695882Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:17.3696020Z 2025-08-14T21:48:17.3696118Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3696458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3696766Z return mod(**inputs) 2025-08-14T21:48:17.3697099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3697467Z outputs = self.model.decoder( 2025-08-14T21:48:17.3697825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3698183Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3698511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3698853Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3699236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3699637Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3699808Z 2025-08-14T21:48:17.3699902Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3700239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3700541Z return mod(**inputs) 2025-08-14T21:48:17.3700871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3701233Z outputs = self.model.decoder( 2025-08-14T21:48:17.3701587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3701952Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3702279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3702645Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3703026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3703435Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3703801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:17.3704129Z return self.act(input) 2025-08-14T21:48:17.3704233Z 2025-08-14T21:48:17.3704329Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3704667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3704998Z return mod(**inputs) 2025-08-14T21:48:17.3705361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3705722Z outputs = self.model.decoder( 2025-08-14T21:48:17.3706082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3706450Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3706771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3707103Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3707468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:48:17.3707839Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:17.3707969Z 2025-08-14T21:48:17.3708066Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3708406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3708721Z return mod(**inputs) 2025-08-14T21:48:17.3709078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3709445Z outputs = self.model.decoder( 2025-08-14T21:48:17.3709820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3710185Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3710502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3710842Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3711205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-14T21:48:17.3711574Z hidden_states = residual + hidden_states 2025-08-14T21:48:17.3711701Z 2025-08-14T21:48:17.3711820Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3712163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3712472Z return mod(**inputs) 2025-08-14T21:48:17.3712814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3713175Z outputs = self.model.decoder( 2025-08-14T21:48:17.3713533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3713903Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3714222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3714562Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3714927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3715319Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3715728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:17.3716171Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:17.3716356Z 2025-08-14T21:48:17.3716460Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3716788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3717079Z return mod(**inputs) 2025-08-14T21:48:17.3717411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3717769Z outputs = self.model.decoder( 2025-08-14T21:48:17.3718113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3718497Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3718835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3719183Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3719544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3719931Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3720314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:17.3720677Z key_states = self.k_proj(current_states) 2025-08-14T21:48:17.3720812Z 2025-08-14T21:48:17.3720907Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3721249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3721565Z return mod(**inputs) 2025-08-14T21:48:17.3721911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3722291Z outputs = self.model.decoder( 2025-08-14T21:48:17.3722657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3723023Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3723355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3723701Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3724072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3724462Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3724859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:17.3725359Z value_states = self.v_proj(current_states) 2025-08-14T21:48:17.3725521Z 2025-08-14T21:48:17.3725620Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3725842Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3726065Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3726293Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3726522Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3726885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3727212Z return mod(**inputs) 2025-08-14T21:48:17.3727575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3727976Z outputs = self.model.decoder( 2025-08-14T21:48:17.3728360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3728755Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3729145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3729516Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3729896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3730300Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3730684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3731080Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3731521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:17.3732021Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:17.3732216Z 2025-08-14T21:48:17.3732318Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3732681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3733002Z return mod(**inputs) 2025-08-14T21:48:17.3733363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3733761Z outputs = self.model.decoder( 2025-08-14T21:48:17.3734144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3734542Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3734884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3735243Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3735640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3736054Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3736468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3736881Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3737322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:17.3737949Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:17.3738125Z 2025-08-14T21:48:17.3738228Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3738598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3738934Z return mod(**inputs) 2025-08-14T21:48:17.3739343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3739731Z outputs = self.model.decoder( 2025-08-14T21:48:17.3740114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3740495Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3740840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3741200Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3741589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3741998Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3742407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:17.3742809Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:17.3742943Z 2025-08-14T21:48:17.3743053Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3743472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3743798Z return mod(**inputs) 2025-08-14T21:48:17.3744154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3744509Z outputs = self.model.decoder( 2025-08-14T21:48:17.3744858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3745224Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3745548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3745882Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3746279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3746692Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3746854Z 2025-08-14T21:48:17.3746950Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3747297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3747598Z return mod(**inputs) 2025-08-14T21:48:17.3747932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3748283Z outputs = self.model.decoder( 2025-08-14T21:48:17.3748643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3749008Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3749324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3749666Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3750046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3750442Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3750788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:17.3751098Z return self.act(input) 2025-08-14T21:48:17.3751198Z 2025-08-14T21:48:17.3751299Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3751628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3751922Z return mod(**inputs) 2025-08-14T21:48:17.3752255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3752632Z outputs = self.model.decoder( 2025-08-14T21:48:17.3752978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3753343Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3753663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3754000Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3754355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:48:17.3754733Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:17.3754865Z 2025-08-14T21:48:17.3754970Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3755305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3755619Z return mod(**inputs) 2025-08-14T21:48:17.3755971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3756368Z outputs = self.model.decoder( 2025-08-14T21:48:17.3756715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3757070Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3757390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3757719Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3758070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3758457Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3758869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:17.3759315Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:17.3759521Z 2025-08-14T21:48:17.3759620Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3759973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3760293Z return mod(**inputs) 2025-08-14T21:48:17.3760640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3761015Z outputs = self.model.decoder( 2025-08-14T21:48:17.3761386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3761764Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3762095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3762450Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3762829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3763222Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3763617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:17.3763999Z key_states = self.k_proj(current_states) 2025-08-14T21:48:17.3764129Z 2025-08-14T21:48:17.3764239Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3764579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3764894Z return mod(**inputs) 2025-08-14T21:48:17.3765311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3765744Z outputs = self.model.decoder( 2025-08-14T21:48:17.3766120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3766512Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3766853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3767202Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3767582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3767988Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3768385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:17.3768765Z value_states = self.v_proj(current_states) 2025-08-14T21:48:17.3768907Z 2025-08-14T21:48:17.3768990Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3769203Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3769401Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3769647Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3769880Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3770223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3770545Z return mod(**inputs) 2025-08-14T21:48:17.3770902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3771280Z outputs = self.model.decoder( 2025-08-14T21:48:17.3771649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3772028Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3772383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3772731Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3773110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3773506Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3773897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3774289Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3774715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:17.3775179Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:17.3775355Z 2025-08-14T21:48:17.3775462Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3775806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3776119Z return mod(**inputs) 2025-08-14T21:48:17.3776479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3776846Z outputs = self.model.decoder( 2025-08-14T21:48:17.3777208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3777581Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3777901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3778235Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3778600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3778986Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3779395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3779789Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3780193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:17.3780615Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:17.3780760Z 2025-08-14T21:48:17.3780864Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3781185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3781484Z return mod(**inputs) 2025-08-14T21:48:17.3781820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3782170Z outputs = self.model.decoder( 2025-08-14T21:48:17.3782518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3782911Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3783241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3783573Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3783935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3784322Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3784703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:17.3785084Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:17.3785222Z 2025-08-14T21:48:17.3785324Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3785694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3785997Z return mod(**inputs) 2025-08-14T21:48:17.3786350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3786706Z outputs = self.model.decoder( 2025-08-14T21:48:17.3787051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3787408Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3787726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3788060Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3788418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3788828Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3788991Z 2025-08-14T21:48:17.3789095Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3789433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3789790Z return mod(**inputs) 2025-08-14T21:48:17.3790125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3790481Z outputs = self.model.decoder( 2025-08-14T21:48:17.3790822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3791178Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3791498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3791838Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3792235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3792647Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3793011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:17.3793330Z return self.act(input) 2025-08-14T21:48:17.3793442Z 2025-08-14T21:48:17.3793539Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3793879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3794189Z return mod(**inputs) 2025-08-14T21:48:17.3794532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3794892Z outputs = self.model.decoder( 2025-08-14T21:48:17.3795252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3795622Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3796031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3796380Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3796748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:48:17.3797114Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:17.3797253Z 2025-08-14T21:48:17.3797350Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3797698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3798005Z return mod(**inputs) 2025-08-14T21:48:17.3798341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3798729Z outputs = self.model.decoder( 2025-08-14T21:48:17.3799096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3799468Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3799800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3800149Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3800523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-14T21:48:17.3800896Z hidden_states = residual + hidden_states 2025-08-14T21:48:17.3801032Z 2025-08-14T21:48:17.3801132Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3801323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3801397Z return mod(**inputs) 2025-08-14T21:48:17.3801641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3801724Z outputs = self.model.decoder( 2025-08-14T21:48:17.3801965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3802038Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3802259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3802337Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3802585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3802681Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3802927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:17.3803101Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:17.3803107Z 2025-08-14T21:48:17.3803207Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3803398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3803470Z return mod(**inputs) 2025-08-14T21:48:17.3803718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3803798Z outputs = self.model.decoder( 2025-08-14T21:48:17.3804040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3804110Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3804331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3804411Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3804677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3804791Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3805034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:17.3805122Z key_states = self.k_proj(current_states) 2025-08-14T21:48:17.3805126Z 2025-08-14T21:48:17.3805227Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3805492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3805567Z return mod(**inputs) 2025-08-14T21:48:17.3805842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3805932Z outputs = self.model.decoder( 2025-08-14T21:48:17.3806242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3806334Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3806558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3806636Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3806879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3806988Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3807224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:17.3807313Z value_states = self.v_proj(current_states) 2025-08-14T21:48:17.3807317Z 2025-08-14T21:48:17.3807397Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3807473Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3807555Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3807628Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3807733Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3807919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3807983Z return mod(**inputs) 2025-08-14T21:48:17.3808229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3808299Z outputs = self.model.decoder( 2025-08-14T21:48:17.3808539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3808616Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3808825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3808928Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3809166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3809258Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3809504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3809597Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3809873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:17.3810005Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:17.3810008Z 2025-08-14T21:48:17.3810103Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3810299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3810363Z return mod(**inputs) 2025-08-14T21:48:17.3810634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3810713Z outputs = self.model.decoder( 2025-08-14T21:48:17.3810954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3811031Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3811236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3811311Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3811555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3811647Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3811901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3812004Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3812287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:17.3812394Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:17.3812398Z 2025-08-14T21:48:17.3812491Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3812673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3812743Z return mod(**inputs) 2025-08-14T21:48:17.3812979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3813055Z outputs = self.model.decoder( 2025-08-14T21:48:17.3813290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3813361Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3813576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3813648Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3813886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3813984Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3814221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:17.3814305Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:17.3814308Z 2025-08-14T21:48:17.3814404Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3814590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3814677Z return mod(**inputs) 2025-08-14T21:48:17.3814918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3814993Z outputs = self.model.decoder( 2025-08-14T21:48:17.3815242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3815310Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3815518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3815590Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3815819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3815937Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3815943Z 2025-08-14T21:48:17.3816036Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3816264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3816326Z return mod(**inputs) 2025-08-14T21:48:17.3816560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3816637Z outputs = self.model.decoder( 2025-08-14T21:48:17.3816869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3816935Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3817143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3817213Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3817471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3817582Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3817779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:17.3817852Z return self.act(input) 2025-08-14T21:48:17.3817855Z 2025-08-14T21:48:17.3817949Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3818138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3818200Z return mod(**inputs) 2025-08-14T21:48:17.3818431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3818507Z outputs = self.model.decoder( 2025-08-14T21:48:17.3818740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3818808Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3819019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3819091Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3819329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:48:17.3819404Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:17.3819408Z 2025-08-14T21:48:17.3819501Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3819694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3819756Z return mod(**inputs) 2025-08-14T21:48:17.3820003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3820072Z outputs = self.model.decoder( 2025-08-14T21:48:17.3820330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3820408Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3820615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3820689Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3820933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3821026Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3821269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:17.3821412Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:17.3821415Z 2025-08-14T21:48:17.3821523Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3821713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3821805Z return mod(**inputs) 2025-08-14T21:48:17.3822046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3822115Z outputs = self.model.decoder( 2025-08-14T21:48:17.3822346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3822420Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3822620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3822692Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3822930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3823039Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3823286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:17.3823361Z key_states = self.k_proj(current_states) 2025-08-14T21:48:17.3823364Z 2025-08-14T21:48:17.3823459Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3823648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3823708Z return mod(**inputs) 2025-08-14T21:48:17.3823940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3824015Z outputs = self.model.decoder( 2025-08-14T21:48:17.3824246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3824325Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3824526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3824600Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3824842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3824929Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3825172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:17.3825252Z value_states = self.v_proj(current_states) 2025-08-14T21:48:17.3825256Z 2025-08-14T21:48:17.3825331Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3825418Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3825490Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3825560Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3825684Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3825868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3825939Z return mod(**inputs) 2025-08-14T21:48:17.3826176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3826247Z outputs = self.model.decoder( 2025-08-14T21:48:17.3826490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3826562Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3826767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3826849Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3827086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3827185Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3827503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3827598Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3827882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:17.3828012Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:17.3828015Z 2025-08-14T21:48:17.3828119Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3828307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3828371Z return mod(**inputs) 2025-08-14T21:48:17.3828640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3828713Z outputs = self.model.decoder( 2025-08-14T21:48:17.3828954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3829032Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3829243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3829326Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3829567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3829662Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3829914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3830008Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3830296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:17.3830405Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:17.3830410Z 2025-08-14T21:48:17.3830509Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3830709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3830774Z return mod(**inputs) 2025-08-14T21:48:17.3831032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3831107Z outputs = self.model.decoder( 2025-08-14T21:48:17.3831344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3831419Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3831653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3831732Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3831984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3832076Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3832330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:17.3832405Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:17.3832408Z 2025-08-14T21:48:17.3832503Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3832693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3832754Z return mod(**inputs) 2025-08-14T21:48:17.3832990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3833070Z outputs = self.model.decoder( 2025-08-14T21:48:17.3833342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3833420Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3833630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3833703Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3833951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3834062Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3834066Z 2025-08-14T21:48:17.3834169Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3834408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3834474Z return mod(**inputs) 2025-08-14T21:48:17.3834728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3834800Z outputs = self.model.decoder( 2025-08-14T21:48:17.3835042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3835118Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3835330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3835414Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3835657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3835777Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3835985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:17.3836052Z return self.act(input) 2025-08-14T21:48:17.3836057Z 2025-08-14T21:48:17.3836154Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3836348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3836411Z return mod(**inputs) 2025-08-14T21:48:17.3836655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3836725Z outputs = self.model.decoder( 2025-08-14T21:48:17.3836962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3837042Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3837254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3837355Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3837739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:48:17.3837834Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:17.3837838Z 2025-08-14T21:48:17.3837946Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3838139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3838204Z return mod(**inputs) 2025-08-14T21:48:17.3838456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3838529Z outputs = self.model.decoder( 2025-08-14T21:48:17.3838784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3838861Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3839076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3839251Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3839521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-14T21:48:17.3839605Z hidden_states = residual + hidden_states 2025-08-14T21:48:17.3839616Z 2025-08-14T21:48:17.3839723Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3839935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3840014Z return mod(**inputs) 2025-08-14T21:48:17.3840287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3840365Z outputs = self.model.decoder( 2025-08-14T21:48:17.3840668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3840746Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3840980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3841058Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3841311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3841415Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3841667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:17.3841818Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:17.3841829Z 2025-08-14T21:48:17.3841930Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3842128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3842206Z return mod(**inputs) 2025-08-14T21:48:17.3842463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3842537Z outputs = self.model.decoder( 2025-08-14T21:48:17.3842802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3842877Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3843102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3843179Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3843424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3843562Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3843831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:17.3843918Z key_states = self.k_proj(current_states) 2025-08-14T21:48:17.3843930Z 2025-08-14T21:48:17.3844037Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3844247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3844322Z return mod(**inputs) 2025-08-14T21:48:17.3844590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3844666Z outputs = self.model.decoder( 2025-08-14T21:48:17.3844939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3845017Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3845312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3845444Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3845715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3845823Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3846083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:17.3846171Z value_states = self.v_proj(current_states) 2025-08-14T21:48:17.3846175Z 2025-08-14T21:48:17.3846268Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3846353Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3846441Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3846522Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3846646Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3846864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3846942Z return mod(**inputs) 2025-08-14T21:48:17.3847192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3847272Z outputs = self.model.decoder( 2025-08-14T21:48:17.3847516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3847592Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3847801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3847876Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3848149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3848255Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3848525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3848636Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3848943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:17.3849086Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:17.3849090Z 2025-08-14T21:48:17.3849194Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3849411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3849486Z return mod(**inputs) 2025-08-14T21:48:17.3849768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3849872Z outputs = self.model.decoder( 2025-08-14T21:48:17.3850150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3850227Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3850463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3850545Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3850809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3850917Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3851179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3851288Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3851595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:17.3851744Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:17.3851748Z 2025-08-14T21:48:17.3851864Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3852085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3852161Z return mod(**inputs) 2025-08-14T21:48:17.3852432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3852509Z outputs = self.model.decoder( 2025-08-14T21:48:17.3852781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3852857Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3853106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3853198Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3853469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3853578Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3853842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:17.3853928Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:17.3853931Z 2025-08-14T21:48:17.3854045Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3854256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3854334Z return mod(**inputs) 2025-08-14T21:48:17.3854602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3854681Z outputs = self.model.decoder( 2025-08-14T21:48:17.3854959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3855036Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3855264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3855354Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3855616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3855751Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3855755Z 2025-08-14T21:48:17.3855862Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3856068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3856166Z return mod(**inputs) 2025-08-14T21:48:17.3856440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3856530Z outputs = self.model.decoder( 2025-08-14T21:48:17.3856797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3856872Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3857111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3857197Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3857465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3857593Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3857819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:17.3857909Z return self.act(input) 2025-08-14T21:48:17.3857933Z 2025-08-14T21:48:17.3858046Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3858240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3858311Z return mod(**inputs) 2025-08-14T21:48:17.3858551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3858623Z outputs = self.model.decoder( 2025-08-14T21:48:17.3858871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3858943Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3859158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3859264Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3859500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:48:17.3859588Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:17.3859592Z 2025-08-14T21:48:17.3859686Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3859879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3859942Z return mod(**inputs) 2025-08-14T21:48:17.3860188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3860268Z outputs = self.model.decoder( 2025-08-14T21:48:17.3860511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3860581Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3860804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3860885Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3861151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3861253Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3861516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:17.3861681Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:17.3861685Z 2025-08-14T21:48:17.3861791Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3862003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3862070Z return mod(**inputs) 2025-08-14T21:48:17.3862360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3862451Z outputs = self.model.decoder( 2025-08-14T21:48:17.3862724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3862799Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3863038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3863123Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3863369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3863458Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3863697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:17.3863782Z key_states = self.k_proj(current_states) 2025-08-14T21:48:17.3863786Z 2025-08-14T21:48:17.3863911Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3864121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3864184Z return mod(**inputs) 2025-08-14T21:48:17.3864420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3864499Z outputs = self.model.decoder( 2025-08-14T21:48:17.3864736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3864808Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3865034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3865126Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3865394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3865499Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3865764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:17.3865863Z value_states = self.v_proj(current_states) 2025-08-14T21:48:17.3865867Z 2025-08-14T21:48:17.3865951Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3866044Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3866127Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3866210Z cudagraph partition due to non gpu ops 2025-08-14T21:48:17.3866324Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3866538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3866611Z return mod(**inputs) 2025-08-14T21:48:17.3866897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3866979Z outputs = self.model.decoder( 2025-08-14T21:48:17.3867258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3867345Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3867587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3867679Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3867952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3868062Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3868326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3868444Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3868741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:17.3868871Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:17.3868874Z 2025-08-14T21:48:17.3868974Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3869179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3869243Z return mod(**inputs) 2025-08-14T21:48:17.3869497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3869580Z outputs = self.model.decoder( 2025-08-14T21:48:17.3869830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3869912Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3870188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3870276Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3870564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3870668Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3870957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:17.3871066Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:17.3871385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:17.3871530Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:17.3871536Z 2025-08-14T21:48:17.3871647Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3871875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3871939Z return mod(**inputs) 2025-08-14T21:48:17.3872187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3872268Z outputs = self.model.decoder( 2025-08-14T21:48:17.3872519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3872591Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3872814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3872892Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3873152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:17.3873252Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:17.3873502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:17.3873594Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:17.3873597Z 2025-08-14T21:48:17.3873700Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3873910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3873986Z return mod(**inputs) 2025-08-14T21:48:17.3874250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3874335Z outputs = self.model.decoder( 2025-08-14T21:48:17.3874603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3874700Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3874944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3875025Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3875295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3875416Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3875420Z 2025-08-14T21:48:17.3875527Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3875748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3875812Z return mod(**inputs) 2025-08-14T21:48:17.3876065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3876148Z outputs = self.model.decoder( 2025-08-14T21:48:17.3876412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3876512Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3876735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3876811Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3877065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:17.3877180Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:17.3877394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:17.3877464Z return self.act(input) 2025-08-14T21:48:17.3877468Z 2025-08-14T21:48:17.3877584Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3877789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3877856Z return mod(**inputs) 2025-08-14T21:48:17.3878108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3878189Z outputs = self.model.decoder( 2025-08-14T21:48:17.3878434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3878513Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3878727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3878804Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3879062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:48:17.3879144Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:17.3879148Z 2025-08-14T21:48:17.3879249Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3879452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3879516Z return mod(**inputs) 2025-08-14T21:48:17.3879769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1864, in forward 2025-08-14T21:48:17.3879842Z outputs = self.model.decoder( 2025-08-14T21:48:17.3880094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:17.3880174Z layer_outputs = decoder_layer( 2025-08-14T21:48:17.3880392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:17.3880477Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:17.3880748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-14T21:48:17.3880830Z hidden_states = residual + hidden_states 2025-08-14T21:48:17.3880833Z 2025-08-14T21:48:17.3880940Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3881137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3881204Z return mod(**inputs) 2025-08-14T21:48:17.3881461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1880, in forward 2025-08-14T21:48:17.3881542Z logits = self.lm_head(outputs[0]) 2025-08-14T21:48:17.3881545Z 2025-08-14T21:48:17.3881651Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:17.3881845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:17.3881914Z return mod(**inputs) 2025-08-14T21:48:17.3882173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1886, in forward 2025-08-14T21:48:17.3882354Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:48:17.3882358Z 2025-08-14T21:48:26.9267777Z Compilation time (from dynamo_timed): 15.163657412 2025-08-14T21:48:26.9504228Z pass 2025-08-14T21:48:26.9505796Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:48:26.9506670Z TIMING: _recursive_pre_grad_passes:0.00735 _recursive_joint_graph_passes:0.6357 _recursive_post_grad_passes:0.08221 async_compile.wait:0.66462 code_gen:8.1945 inductor_compile:9.42747 backend_compile:12.58776 gc:0.00013 entire_frame_compile:15.16366 total_wall_time:15.16366 2025-08-14T21:48:26.9509918Z STATS: call_* op count: 373 | FakeTensorMode.__torch_dispatch__:13266 | FakeTensor.__torch_dispatch__:4931 | ProxyTorchDispatchMode.__torch_dispatch__:4844 2025-08-14T21:48:26.9510510Z Dynamo produced 1 graphs covering 373 ops with 0 graph breaks (0 unique) 2025-08-14T21:48:32.1773772Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:48:32.1774641Z from pkg_resources import resource_filename 2025-08-14T21:48:32.7682766Z 2025-08-14T21:48:37.7714304Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:48:37.7717873Z loading model: 0it [00:05, ?it/s] 2025-08-14T21:48:37.7738424Z cpu eval MBartForConditionalGeneration 2025-08-14T21:48:40.9722699Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:48:42.1791728Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:48:43.3796879Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:48:59.9359297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9359824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9360175Z return mod(**inputs) 2025-08-14T21:48:59.9360585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1436, in forward 2025-08-14T21:48:59.9361112Z decoder_input_ids = shift_tokens_right(labels, self.config.pad_token_id) 2025-08-14T21:48:59.9361667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 76, in shift_tokens_right 2025-08-14T21:48:59.9362242Z index_of_eos = (prev_output_tokens.ne(pad_token_id).sum(dim=1) - 1).unsqueeze(-1) 2025-08-14T21:48:59.9362480Z 2025-08-14T21:48:59.9362916Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9363159Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9363392Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9363617Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9363834Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9364066Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9364312Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9364539Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9364749Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9364969Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9365197Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9365593Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9365866Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9366275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9366636Z return mod(**inputs) 2025-08-14T21:48:59.9367017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9367556Z outputs = self.model( 2025-08-14T21:48:59.9367939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9368336Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9368728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9369127Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9369478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9369865Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9370346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9370765Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9371191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:59.9371676Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:59.9371955Z 2025-08-14T21:48:59.9372071Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9372463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9372800Z return mod(**inputs) 2025-08-14T21:48:59.9373208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9373632Z outputs = self.model( 2025-08-14T21:48:59.9374008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9374444Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9374852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9375259Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9375633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9376011Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9376422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9376845Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9377260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:59.9377676Z key_states = self.k_proj(current_states) 2025-08-14T21:48:59.9377819Z 2025-08-14T21:48:59.9377968Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9378358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9378727Z return mod(**inputs) 2025-08-14T21:48:59.9379132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9379541Z outputs = self.model( 2025-08-14T21:48:59.9379930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9380349Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9380757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9381209Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9381568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9381940Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9382377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9382825Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9383283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:59.9383722Z value_states = self.v_proj(current_states) 2025-08-14T21:48:59.9383878Z 2025-08-14T21:48:59.9383972Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9384194Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9384430Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9384636Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9385075Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9385462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9385859Z return mod(**inputs) 2025-08-14T21:48:59.9386231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9386631Z outputs = self.model( 2025-08-14T21:48:59.9387068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9387458Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9387846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9388239Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9388648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9389038Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9389427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9389838Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9390241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9390650Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9391093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:59.9391572Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:59.9391758Z 2025-08-14T21:48:59.9391868Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9392246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9392607Z return mod(**inputs) 2025-08-14T21:48:59.9392992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9393477Z outputs = self.model( 2025-08-14T21:48:59.9393867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9394285Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9394703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9395084Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9395439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9395807Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9396218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9396643Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9397078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9398354Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9398807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:59.9399276Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:59.9399456Z 2025-08-14T21:48:59.9399567Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9399949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9400299Z return mod(**inputs) 2025-08-14T21:48:59.9400708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9401126Z outputs = self.model( 2025-08-14T21:48:59.9401537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9401948Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9402360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9402779Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9403146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9403542Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9403959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9404391Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9404820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:59.9405326Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:59.9405484Z 2025-08-14T21:48:59.9405608Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9405995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9406337Z return mod(**inputs) 2025-08-14T21:48:59.9406725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9407137Z outputs = self.model( 2025-08-14T21:48:59.9407519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9407938Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9408335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9408742Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9409141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9409531Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9409945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9410395Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9410587Z 2025-08-14T21:48:59.9410697Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9411075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9411419Z return mod(**inputs) 2025-08-14T21:48:59.9411795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9412199Z outputs = self.model( 2025-08-14T21:48:59.9412584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9413064Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9413488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9413903Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9414269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9414640Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9415049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9415521Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9415911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:59.9416268Z return self.act(input) 2025-08-14T21:48:59.9416390Z 2025-08-14T21:48:59.9416493Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9416860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9417176Z return mod(**inputs) 2025-08-14T21:48:59.9417561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9417972Z outputs = self.model( 2025-08-14T21:48:59.9418365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9418784Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9419188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9419596Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9419963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9420343Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9420761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:48:59.9421184Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:59.9421330Z 2025-08-14T21:48:59.9421439Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9421821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9422205Z return mod(**inputs) 2025-08-14T21:48:59.9422603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9423011Z outputs = self.model( 2025-08-14T21:48:59.9423411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9423868Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9424266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9424684Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9425056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9425462Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9425862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9426299Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9426731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:59.9427230Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:59.9427449Z 2025-08-14T21:48:59.9427558Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9428001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9428365Z return mod(**inputs) 2025-08-14T21:48:59.9428743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9429152Z outputs = self.model( 2025-08-14T21:48:59.9429545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9429969Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9430364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9430778Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9431184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9431591Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9432006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9432444Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9432881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:59.9433313Z key_states = self.k_proj(current_states) 2025-08-14T21:48:59.9433470Z 2025-08-14T21:48:59.9433583Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9433970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9434345Z return mod(**inputs) 2025-08-14T21:48:59.9434719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9435128Z outputs = self.model( 2025-08-14T21:48:59.9435522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9435939Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9436334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9436743Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9437122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9437495Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9438227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9438670Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9439100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:59.9439594Z value_states = self.v_proj(current_states) 2025-08-14T21:48:59.9439756Z 2025-08-14T21:48:59.9439846Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9440084Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9440302Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9440528Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9440784Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9441181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9441556Z return mod(**inputs) 2025-08-14T21:48:59.9441980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9442400Z outputs = self.model( 2025-08-14T21:48:59.9442794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9443223Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9443725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9444148Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9444516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9444908Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9445401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9445848Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9446297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9446790Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9447277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:59.9447776Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:59.9447978Z 2025-08-14T21:48:59.9448088Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9448467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9448812Z return mod(**inputs) 2025-08-14T21:48:59.9449203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9449585Z outputs = self.model( 2025-08-14T21:48:59.9449945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9450331Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9450711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9451097Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9451466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9451821Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9452207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9452626Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9453042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9453462Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9453921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:59.9454427Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:59.9454598Z 2025-08-14T21:48:59.9454708Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9455086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9455412Z return mod(**inputs) 2025-08-14T21:48:59.9455797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9456204Z outputs = self.model( 2025-08-14T21:48:59.9456588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9457009Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9457399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9457816Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9458199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9458608Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9459019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9459456Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9459889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:59.9460350Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:59.9460497Z 2025-08-14T21:48:59.9460606Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9460978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9461330Z return mod(**inputs) 2025-08-14T21:48:59.9461705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9462111Z outputs = self.model( 2025-08-14T21:48:59.9462493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9462904Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9463297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9463705Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9464081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9464463Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9464890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9465320Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9465493Z 2025-08-14T21:48:59.9465606Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9465969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9466311Z return mod(**inputs) 2025-08-14T21:48:59.9466689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9467090Z outputs = self.model( 2025-08-14T21:48:59.9467469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9467878Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9468279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9468723Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9469082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9469460Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9469867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9470309Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9470712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:59.9471074Z return self.act(input) 2025-08-14T21:48:59.9471187Z 2025-08-14T21:48:59.9471301Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9471671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9472010Z return mod(**inputs) 2025-08-14T21:48:59.9472405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9472839Z outputs = self.model( 2025-08-14T21:48:59.9473257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9473683Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9474098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9474498Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9474918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9475308Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9475712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:48:59.9476153Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:59.9476308Z 2025-08-14T21:48:59.9476419Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9476798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9477133Z return mod(**inputs) 2025-08-14T21:48:59.9477512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9477934Z outputs = self.model( 2025-08-14T21:48:59.9478311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9478736Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9479138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9479552Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9479912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9480308Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9480734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-08-14T21:48:59.9481164Z hidden_states = residual + hidden_states 2025-08-14T21:48:59.9481313Z 2025-08-14T21:48:59.9481424Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9481813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9482266Z return mod(**inputs) 2025-08-14T21:48:59.9482680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9483096Z outputs = self.model( 2025-08-14T21:48:59.9483497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9483945Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9484344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9484759Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9485129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9485593Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9486014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9486455Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9486889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:59.9487385Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:59.9487622Z 2025-08-14T21:48:59.9487734Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9488169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9488522Z return mod(**inputs) 2025-08-14T21:48:59.9488906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9489342Z outputs = self.model( 2025-08-14T21:48:59.9489737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9490151Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9490563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9490984Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9491384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9491773Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9492194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9492633Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9493138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:59.9493602Z key_states = self.k_proj(current_states) 2025-08-14T21:48:59.9493764Z 2025-08-14T21:48:59.9493885Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9494250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9494571Z return mod(**inputs) 2025-08-14T21:48:59.9494940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9495330Z outputs = self.model( 2025-08-14T21:48:59.9495699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9496086Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9496469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9496863Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9497205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9497569Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9497965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9498372Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9498771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:59.9499240Z value_states = self.v_proj(current_states) 2025-08-14T21:48:59.9499386Z 2025-08-14T21:48:59.9499467Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9499683Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9499884Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9500089Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9500318Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9500669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9500994Z return mod(**inputs) 2025-08-14T21:48:59.9501354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9501730Z outputs = self.model( 2025-08-14T21:48:59.9502107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9502521Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9502962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9503366Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9503739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9504120Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9504543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9504945Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9505377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9505848Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9506334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:59.9506814Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:59.9507002Z 2025-08-14T21:48:59.9507107Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9507473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9507788Z return mod(**inputs) 2025-08-14T21:48:59.9508149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9508536Z outputs = self.model( 2025-08-14T21:48:59.9508898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9509279Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9509657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9510060Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9510397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9510756Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9511144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9511545Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9511933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9512360Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9512833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:59.9513308Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:59.9513475Z 2025-08-14T21:48:59.9513579Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9513939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9514264Z return mod(**inputs) 2025-08-14T21:48:59.9514616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9515002Z outputs = self.model( 2025-08-14T21:48:59.9515365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9515755Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9516128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9516515Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9516879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9517283Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9517668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9518067Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9518464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:59.9518871Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:59.9519022Z 2025-08-14T21:48:59.9519130Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9519512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9519896Z return mod(**inputs) 2025-08-14T21:48:59.9520253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9520636Z outputs = self.model( 2025-08-14T21:48:59.9520997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9521384Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9521764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9522152Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9522522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9522894Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9523303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9523760Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9523942Z 2025-08-14T21:48:59.9524051Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9524430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9524772Z return mod(**inputs) 2025-08-14T21:48:59.9525152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9525647Z outputs = self.model( 2025-08-14T21:48:59.9526052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9526490Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9526887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9527303Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9527651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9528018Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9528411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9528835Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9529211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:59.9529540Z return self.act(input) 2025-08-14T21:48:59.9529650Z 2025-08-14T21:48:59.9529759Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9530145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9530495Z return mod(**inputs) 2025-08-14T21:48:59.9530885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9531321Z outputs = self.model( 2025-08-14T21:48:59.9531717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9532109Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9532498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9532903Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9533267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9533633Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9534025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:48:59.9534423Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:59.9534559Z 2025-08-14T21:48:59.9534669Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9535017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9535334Z return mod(**inputs) 2025-08-14T21:48:59.9535691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9536072Z outputs = self.model( 2025-08-14T21:48:59.9536422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9536804Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9537177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9537548Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9538046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9538416Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9538816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9539217Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9539624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:59.9540096Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:59.9540302Z 2025-08-14T21:48:59.9540415Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9540771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9541097Z return mod(**inputs) 2025-08-14T21:48:59.9541463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9541900Z outputs = self.model( 2025-08-14T21:48:59.9542264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9542651Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9543033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9543413Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9543759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9544123Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9544503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9544909Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9545314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:59.9545766Z key_states = self.k_proj(current_states) 2025-08-14T21:48:59.9545902Z 2025-08-14T21:48:59.9546004Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9546369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9546691Z return mod(**inputs) 2025-08-14T21:48:59.9547049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9547437Z outputs = self.model( 2025-08-14T21:48:59.9547798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9548262Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9548667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9549055Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9549398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9549751Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9550125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9550527Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9550921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:59.9551312Z value_states = self.v_proj(current_states) 2025-08-14T21:48:59.9551459Z 2025-08-14T21:48:59.9551551Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9551756Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9551959Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9552150Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9552374Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9552720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9553040Z return mod(**inputs) 2025-08-14T21:48:59.9553452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9553820Z outputs = self.model( 2025-08-14T21:48:59.9554173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9554543Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9554904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9555278Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9555630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9555983Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9556359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9556748Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9557134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9557537Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9557971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:59.9558435Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:59.9558613Z 2025-08-14T21:48:59.9558715Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9559061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9559412Z return mod(**inputs) 2025-08-14T21:48:59.9559759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9560130Z outputs = self.model( 2025-08-14T21:48:59.9560481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9560858Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9561217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9561595Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9561931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9562291Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9562669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9563059Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9563442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9563832Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9564268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:59.9564743Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:59.9564911Z 2025-08-14T21:48:59.9565025Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9565486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9565834Z return mod(**inputs) 2025-08-14T21:48:59.9566215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9566619Z outputs = self.model( 2025-08-14T21:48:59.9566979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9567373Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9567742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9568114Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9568455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9568818Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9569208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9569627Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9570016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:59.9570398Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:59.9570529Z 2025-08-14T21:48:59.9570627Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9570983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9571303Z return mod(**inputs) 2025-08-14T21:48:59.9571661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9572032Z outputs = self.model( 2025-08-14T21:48:59.9572394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9572782Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9573176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9573596Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9573942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9574301Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9574684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9575117Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9575288Z 2025-08-14T21:48:59.9575396Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9575746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9576095Z return mod(**inputs) 2025-08-14T21:48:59.9576458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9576839Z outputs = self.model( 2025-08-14T21:48:59.9577197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9577584Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9577961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9578344Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9578681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9579038Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9579439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9579864Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9580250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:59.9580587Z return self.act(input) 2025-08-14T21:48:59.9580696Z 2025-08-14T21:48:59.9580805Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9581154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9581476Z return mod(**inputs) 2025-08-14T21:48:59.9581832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9582206Z outputs = self.model( 2025-08-14T21:48:59.9582575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9582984Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9583361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9583746Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9584083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9584425Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9584793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:48:59.9585175Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:59.9585311Z 2025-08-14T21:48:59.9585409Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9585787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9586095Z return mod(**inputs) 2025-08-14T21:48:59.9586450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9586841Z outputs = self.model( 2025-08-14T21:48:59.9587202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9587578Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9587955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9588338Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9588677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9589035Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9589421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-08-14T21:48:59.9589833Z hidden_states = residual + hidden_states 2025-08-14T21:48:59.9589963Z 2025-08-14T21:48:59.9590061Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9590410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9590726Z return mod(**inputs) 2025-08-14T21:48:59.9591067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9591440Z outputs = self.model( 2025-08-14T21:48:59.9591791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9592166Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9592533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9592918Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9593269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9593621Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9593996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9594391Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9594776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:59.9595219Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:59.9595422Z 2025-08-14T21:48:59.9595521Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9595865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9596179Z return mod(**inputs) 2025-08-14T21:48:59.9596522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9596924Z outputs = self.model( 2025-08-14T21:48:59.9597276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9597651Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9598151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9598533Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9598866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9599208Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9599585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9599977Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9600365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:59.9600792Z key_states = self.k_proj(current_states) 2025-08-14T21:48:59.9600933Z 2025-08-14T21:48:59.9601033Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9601384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9601694Z return mod(**inputs) 2025-08-14T21:48:59.9602055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9602438Z outputs = self.model( 2025-08-14T21:48:59.9602819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9603231Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9603661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9604050Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9604384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9604740Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9605138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9605646Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9606071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:59.9606500Z value_states = self.v_proj(current_states) 2025-08-14T21:48:59.9606649Z 2025-08-14T21:48:59.9606730Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9606945Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9607152Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9607363Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9607603Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9607959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9608289Z return mod(**inputs) 2025-08-14T21:48:59.9608656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9609035Z outputs = self.model( 2025-08-14T21:48:59.9609405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9609799Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9610183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9610572Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9610949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9611314Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9611700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9612095Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9612493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9612907Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9613343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:59.9613823Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:59.9614011Z 2025-08-14T21:48:59.9614118Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9614494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9614849Z return mod(**inputs) 2025-08-14T21:48:59.9615219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9615596Z outputs = self.model( 2025-08-14T21:48:59.9615950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9616318Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9616683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9617056Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9617383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9617755Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9618120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9618500Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9618867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9619255Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9619672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:59.9620101Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:59.9620250Z 2025-08-14T21:48:59.9620347Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9620682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9620988Z return mod(**inputs) 2025-08-14T21:48:59.9621322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9621683Z outputs = self.model( 2025-08-14T21:48:59.9622024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9622389Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9622741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9623102Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9623427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9623755Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9624117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9624519Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9624898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:59.9625267Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:59.9625407Z 2025-08-14T21:48:59.9625506Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9625859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9626176Z return mod(**inputs) 2025-08-14T21:48:59.9626522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9626895Z outputs = self.model( 2025-08-14T21:48:59.9627247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9627609Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9628010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9628390Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9628726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9629060Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9629433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9629848Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9630013Z 2025-08-14T21:48:59.9630111Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9630456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9630785Z return mod(**inputs) 2025-08-14T21:48:59.9631136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9631500Z outputs = self.model( 2025-08-14T21:48:59.9631856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9632244Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9632617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9633006Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9633355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9633717Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9634088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9634512Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9634899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:59.9635222Z return self.act(input) 2025-08-14T21:48:59.9635328Z 2025-08-14T21:48:59.9635431Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9635781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9636094Z return mod(**inputs) 2025-08-14T21:48:59.9636441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9636812Z outputs = self.model( 2025-08-14T21:48:59.9637166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9637566Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9638077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9638466Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9638807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9639163Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9639562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:48:59.9639961Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:59.9640096Z 2025-08-14T21:48:59.9640206Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9640565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9640887Z return mod(**inputs) 2025-08-14T21:48:59.9641244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9641668Z outputs = self.model( 2025-08-14T21:48:59.9642043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9642437Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9642816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9643198Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9643562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9643953Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9644337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9644780Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9645205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:59.9645770Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:59.9645986Z 2025-08-14T21:48:59.9646104Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9646487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9646807Z return mod(**inputs) 2025-08-14T21:48:59.9647183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9647563Z outputs = self.model( 2025-08-14T21:48:59.9647920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9648299Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9648669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9649039Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9649375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9649726Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9650099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9650494Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9650885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:59.9651267Z key_states = self.k_proj(current_states) 2025-08-14T21:48:59.9651395Z 2025-08-14T21:48:59.9651496Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9651918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9652230Z return mod(**inputs) 2025-08-14T21:48:59.9652565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9652953Z outputs = self.model( 2025-08-14T21:48:59.9653311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9653697Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9654062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9654443Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9654789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9655157Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9655517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9655940Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9656320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:59.9656691Z value_states = self.v_proj(current_states) 2025-08-14T21:48:59.9656827Z 2025-08-14T21:48:59.9656908Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9657114Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9657315Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9657509Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9657732Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9658081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9658413Z return mod(**inputs) 2025-08-14T21:48:59.9658770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9659156Z outputs = self.model( 2025-08-14T21:48:59.9659519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9659899Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9660279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9660667Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9661006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9661414Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9661781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9662173Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9662558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9662957Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9663385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:59.9663847Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:59.9664026Z 2025-08-14T21:48:59.9664129Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9664497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9664815Z return mod(**inputs) 2025-08-14T21:48:59.9665160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9665555Z outputs = self.model( 2025-08-14T21:48:59.9665912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9666276Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9666623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9666987Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9667310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9667636Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9667999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9668378Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9668751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9669157Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9688888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:59.9689652Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:59.9689817Z 2025-08-14T21:48:59.9689928Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9690291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9690610Z return mod(**inputs) 2025-08-14T21:48:59.9690968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9691347Z outputs = self.model( 2025-08-14T21:48:59.9691767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9692160Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9692537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9692916Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9693266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9693633Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9694016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9694435Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9694831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:59.9695215Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:59.9695358Z 2025-08-14T21:48:59.9695463Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9695816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9696136Z return mod(**inputs) 2025-08-14T21:48:59.9696496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9696875Z outputs = self.model( 2025-08-14T21:48:59.9697235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9697617Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9698009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9698381Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9698721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9699098Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9699479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9699897Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9700080Z 2025-08-14T21:48:59.9700185Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9700547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9700856Z return mod(**inputs) 2025-08-14T21:48:59.9701195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9701560Z outputs = self.model( 2025-08-14T21:48:59.9701904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9702280Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9702718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9703106Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9703455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9703815Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9704193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9704615Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9704993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:59.9705322Z return self.act(input) 2025-08-14T21:48:59.9705436Z 2025-08-14T21:48:59.9705553Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9705905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9706216Z return mod(**inputs) 2025-08-14T21:48:59.9706569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9706941Z outputs = self.model( 2025-08-14T21:48:59.9707292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9707662Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9708029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9708402Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9708730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9709078Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9709457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:48:59.9709837Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:59.9709971Z 2025-08-14T21:48:59.9710070Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9710415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9710733Z return mod(**inputs) 2025-08-14T21:48:59.9711090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9711492Z outputs = self.model( 2025-08-14T21:48:59.9711866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9712298Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9712692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9713102Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9713447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9713793Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9714163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-08-14T21:48:59.9714548Z hidden_states = residual + hidden_states 2025-08-14T21:48:59.9714678Z 2025-08-14T21:48:59.9714783Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9715121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9715435Z return mod(**inputs) 2025-08-14T21:48:59.9715787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9716178Z outputs = self.model( 2025-08-14T21:48:59.9716539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9716918Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9717289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9717657Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9717998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9718348Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9718727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9719134Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9719528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:59.9719986Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:59.9720185Z 2025-08-14T21:48:59.9720294Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9720635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9720949Z return mod(**inputs) 2025-08-14T21:48:59.9721298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9721659Z outputs = self.model( 2025-08-14T21:48:59.9722011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9722387Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9722756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9723126Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9723463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9723810Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9724192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9724590Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9724986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:59.9725495Z key_states = self.k_proj(current_states) 2025-08-14T21:48:59.9725650Z 2025-08-14T21:48:59.9725767Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9726200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9726545Z return mod(**inputs) 2025-08-14T21:48:59.9726927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9727337Z outputs = self.model( 2025-08-14T21:48:59.9727702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9728092Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9728461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9728850Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9729199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9729558Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9729939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9730383Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9730786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:59.9731182Z value_states = self.v_proj(current_states) 2025-08-14T21:48:59.9731321Z 2025-08-14T21:48:59.9731404Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9731617Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9731822Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9732017Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9732248Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9732604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9732944Z return mod(**inputs) 2025-08-14T21:48:59.9733309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9733694Z outputs = self.model( 2025-08-14T21:48:59.9734053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9734437Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9734820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9735205Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9735545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9735906Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9736294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9736698Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9737094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9737510Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9738217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:59.9738707Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:59.9738892Z 2025-08-14T21:48:59.9739000Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9739368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9739698Z return mod(**inputs) 2025-08-14T21:48:59.9740059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9740568Z outputs = self.model( 2025-08-14T21:48:59.9740920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9741305Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9741671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9742039Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9742368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9742714Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9743073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9743464Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9743858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9744271Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9744727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:59.9745164Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:59.9745316Z 2025-08-14T21:48:59.9745420Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9745749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9746107Z return mod(**inputs) 2025-08-14T21:48:59.9746462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9746830Z outputs = self.model( 2025-08-14T21:48:59.9747201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9747585Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9747971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9748325Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9748656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9748998Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9749382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9749780Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9750165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:59.9750556Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:59.9750690Z 2025-08-14T21:48:59.9750796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9751149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9751469Z return mod(**inputs) 2025-08-14T21:48:59.9751835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9752225Z outputs = self.model( 2025-08-14T21:48:59.9752611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9753081Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9753455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9753832Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9754172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9754561Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9754933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9755345Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9755520Z 2025-08-14T21:48:59.9755618Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9755961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9756314Z return mod(**inputs) 2025-08-14T21:48:59.9756668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9757044Z outputs = self.model( 2025-08-14T21:48:59.9757403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9757785Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9758209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9758585Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9758918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9759262Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9759646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9760079Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9760461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:59.9760793Z return self.act(input) 2025-08-14T21:48:59.9760912Z 2025-08-14T21:48:59.9761033Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9761390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9761704Z return mod(**inputs) 2025-08-14T21:48:59.9762064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9762473Z outputs = self.model( 2025-08-14T21:48:59.9762866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9763277Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9763552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9763630Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9763860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9763957Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9764244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:48:59.9764338Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:59.9764343Z 2025-08-14T21:48:59.9764448Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9764663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9764740Z return mod(**inputs) 2025-08-14T21:48:59.9765016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9765095Z outputs = self.model( 2025-08-14T21:48:59.9765442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9765557Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9765832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9765913Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9766148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9766241Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9766512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9766618Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9766900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:59.9767051Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:59.9767055Z 2025-08-14T21:48:59.9767170Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9767368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9767482Z return mod(**inputs) 2025-08-14T21:48:59.9767735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9767801Z outputs = self.model( 2025-08-14T21:48:59.9768052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9768123Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9768361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9768439Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9768650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9768751Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9768994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9769082Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9769327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:59.9769404Z key_states = self.k_proj(current_states) 2025-08-14T21:48:59.9769407Z 2025-08-14T21:48:59.9769505Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9769700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9769764Z return mod(**inputs) 2025-08-14T21:48:59.9770013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9770658Z outputs = self.model( 2025-08-14T21:48:59.9770904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9770990Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9771233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9771311Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9771521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9771597Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9771847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9771934Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9772177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:59.9772286Z value_states = self.v_proj(current_states) 2025-08-14T21:48:59.9772290Z 2025-08-14T21:48:59.9772371Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9772456Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9772531Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9772605Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9772711Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9772902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9772964Z return mod(**inputs) 2025-08-14T21:48:59.9773217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9773284Z outputs = self.model( 2025-08-14T21:48:59.9773536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9773609Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9773866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9773961Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9774175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9774257Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9774497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9774584Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9774830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9774926Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9775225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:59.9775366Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:59.9775370Z 2025-08-14T21:48:59.9775477Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9775669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9775730Z return mod(**inputs) 2025-08-14T21:48:59.9775968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9776042Z outputs = self.model( 2025-08-14T21:48:59.9776277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9776353Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9776584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9776656Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9776869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9776943Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9777177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9777269Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9777504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9777603Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9777877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:59.9778000Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:59.9778003Z 2025-08-14T21:48:59.9778107Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9778293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9778363Z return mod(**inputs) 2025-08-14T21:48:59.9778601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9778667Z outputs = self.model( 2025-08-14T21:48:59.9778907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9778978Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9779210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9779285Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9779491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9779624Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9779862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9779945Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9780187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:59.9780263Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:59.9780267Z 2025-08-14T21:48:59.9780362Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9780555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9780617Z return mod(**inputs) 2025-08-14T21:48:59.9780884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9780955Z outputs = self.model( 2025-08-14T21:48:59.9781201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9781279Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9781525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9781599Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9781805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9781879Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9782128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9782246Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9782251Z 2025-08-14T21:48:59.9782348Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9782549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9782611Z return mod(**inputs) 2025-08-14T21:48:59.9782859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9782923Z outputs = self.model( 2025-08-14T21:48:59.9783166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9783244Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9783487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9783557Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9783808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9783883Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9784131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9784243Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9784440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:59.9784514Z return self.act(input) 2025-08-14T21:48:59.9784517Z 2025-08-14T21:48:59.9784614Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9784808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9784869Z return mod(**inputs) 2025-08-14T21:48:59.9785109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9785184Z outputs = self.model( 2025-08-14T21:48:59.9785453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9785526Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9785770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9785839Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9786053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9786126Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9786361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:48:59.9786446Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:59.9786451Z 2025-08-14T21:48:59.9786563Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9786759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9786823Z return mod(**inputs) 2025-08-14T21:48:59.9787058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9787130Z outputs = self.model( 2025-08-14T21:48:59.9787369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9787439Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9787688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9787758Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9787981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9788058Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9788308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-08-14T21:48:59.9788395Z hidden_states = residual + hidden_states 2025-08-14T21:48:59.9788399Z 2025-08-14T21:48:59.9788499Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9788702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9788766Z return mod(**inputs) 2025-08-14T21:48:59.9789016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9789089Z outputs = self.model( 2025-08-14T21:48:59.9789336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9789429Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9789692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9789767Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9789996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9790072Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9790326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9790422Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9790674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:59.9790823Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:59.9790835Z 2025-08-14T21:48:59.9790937Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9791139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9791252Z return mod(**inputs) 2025-08-14T21:48:59.9791506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9791574Z outputs = self.model( 2025-08-14T21:48:59.9791830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9791903Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9792158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9792229Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9792445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9792566Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9792817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9792910Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9793162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:59.9793240Z key_states = self.k_proj(current_states) 2025-08-14T21:48:59.9793243Z 2025-08-14T21:48:59.9793349Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9793544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9793610Z return mod(**inputs) 2025-08-14T21:48:59.9793868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9794085Z outputs = self.model( 2025-08-14T21:48:59.9794350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9794434Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9794682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9794763Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9794979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9795055Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9795310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9795400Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9795655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:59.9795764Z value_states = self.v_proj(current_states) 2025-08-14T21:48:59.9795769Z 2025-08-14T21:48:59.9795852Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9795939Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9796015Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9796092Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9796203Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9796401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9796474Z return mod(**inputs) 2025-08-14T21:48:59.9796720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9796789Z outputs = self.model( 2025-08-14T21:48:59.9797045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9797118Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9797403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9797485Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9797703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9797786Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9798031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9798120Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9798377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9798473Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9798790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:59.9798926Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:59.9798930Z 2025-08-14T21:48:59.9799032Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9799236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9799302Z return mod(**inputs) 2025-08-14T21:48:59.9799550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9799633Z outputs = self.model( 2025-08-14T21:48:59.9799890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9799963Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9800209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9800291Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9800508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9800584Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9800839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9800927Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9801181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9801276Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9801562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:59.9801709Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:59.9801713Z 2025-08-14T21:48:59.9801815Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9802017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9802083Z return mod(**inputs) 2025-08-14T21:48:59.9802332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9802408Z outputs = self.model( 2025-08-14T21:48:59.9802657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9802729Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9802983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9803053Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9803280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9803390Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9803640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9803736Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9803980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:59.9804068Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:59.9804071Z 2025-08-14T21:48:59.9804171Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9804369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9804442Z return mod(**inputs) 2025-08-14T21:48:59.9804706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9804777Z outputs = self.model( 2025-08-14T21:48:59.9805036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9805112Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9805453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9805536Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9805773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9805864Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9806133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9806270Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9806276Z 2025-08-14T21:48:59.9806396Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9806604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9806680Z return mod(**inputs) 2025-08-14T21:48:59.9806955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9807022Z outputs = self.model( 2025-08-14T21:48:59.9807273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9807345Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9807599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9807672Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9807911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9807998Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9808247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9808365Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9808580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:59.9808649Z return self.act(input) 2025-08-14T21:48:59.9808652Z 2025-08-14T21:48:59.9808759Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9808961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9809025Z return mod(**inputs) 2025-08-14T21:48:59.9809276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9809343Z outputs = self.model( 2025-08-14T21:48:59.9809630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9809703Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9809945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9810024Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9810231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9810306Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9810555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:48:59.9810635Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:59.9810639Z 2025-08-14T21:48:59.9810763Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9810960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9811026Z return mod(**inputs) 2025-08-14T21:48:59.9811275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9811339Z outputs = self.model( 2025-08-14T21:48:59.9811589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9811660Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9811902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9811979Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9812194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9812269Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9812520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9812607Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9812856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:59.9813003Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:59.9813007Z 2025-08-14T21:48:59.9813102Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9813305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9813369Z return mod(**inputs) 2025-08-14T21:48:59.9813615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9813706Z outputs = self.model( 2025-08-14T21:48:59.9813952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9814034Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9814283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9814356Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9814578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9814654Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9814908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9814997Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9815244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:59.9815349Z key_states = self.k_proj(current_states) 2025-08-14T21:48:59.9815373Z 2025-08-14T21:48:59.9815475Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9815670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9815744Z return mod(**inputs) 2025-08-14T21:48:59.9816003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9816077Z outputs = self.model( 2025-08-14T21:48:59.9816321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9816391Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9816661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9816734Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9816956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9817029Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9817270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9817365Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9817606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:59.9817689Z value_states = self.v_proj(current_states) 2025-08-14T21:48:59.9817692Z 2025-08-14T21:48:59.9817779Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9817857Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9817942Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9818021Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9818122Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9818330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9818397Z return mod(**inputs) 2025-08-14T21:48:59.9818651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9818725Z outputs = self.model( 2025-08-14T21:48:59.9818977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9819057Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9819306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9819380Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9819606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9819702Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9819956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9820052Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9820304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9820408Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9820698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:59.9820829Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:59.9820833Z 2025-08-14T21:48:59.9820939Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9821139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9821227Z return mod(**inputs) 2025-08-14T21:48:59.9821496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9821565Z outputs = self.model( 2025-08-14T21:48:59.9821888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9821958Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9822197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9822273Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9822482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9822586Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9822832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9822921Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9823170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9823264Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9823553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:59.9823656Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:59.9823660Z 2025-08-14T21:48:59.9823757Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9823960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9824028Z return mod(**inputs) 2025-08-14T21:48:59.9824275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9824352Z outputs = self.model( 2025-08-14T21:48:59.9824597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9824677Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9824923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9824996Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9825222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9825299Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9825558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9825666Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9825917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:59.9826007Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:59.9826011Z 2025-08-14T21:48:59.9826111Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9826310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9826384Z return mod(**inputs) 2025-08-14T21:48:59.9826637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9826716Z outputs = self.model( 2025-08-14T21:48:59.9826965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9827043Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9827298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9827406Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9827629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9827715Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9827964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9828089Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9828092Z 2025-08-14T21:48:59.9828192Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9828390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9828464Z return mod(**inputs) 2025-08-14T21:48:59.9828733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9828812Z outputs = self.model( 2025-08-14T21:48:59.9829068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9829141Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9829394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9829465Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9829686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9829776Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9830035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9830167Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9830390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:59.9830463Z return self.act(input) 2025-08-14T21:48:59.9830467Z 2025-08-14T21:48:59.9830579Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9830791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9830862Z return mod(**inputs) 2025-08-14T21:48:59.9831115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9831181Z outputs = self.model( 2025-08-14T21:48:59.9831439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9831511Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9831781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9831865Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9832091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9832181Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9832439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:48:59.9832523Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:59.9832527Z 2025-08-14T21:48:59.9832639Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9832848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9832916Z return mod(**inputs) 2025-08-14T21:48:59.9833186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9833258Z outputs = self.model( 2025-08-14T21:48:59.9833570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9833652Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9833920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9834000Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9834214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9834299Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9834546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-08-14T21:48:59.9834624Z hidden_states = residual + hidden_states 2025-08-14T21:48:59.9834629Z 2025-08-14T21:48:59.9834754Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9834953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9835020Z return mod(**inputs) 2025-08-14T21:48:59.9835275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9835344Z outputs = self.model( 2025-08-14T21:48:59.9835600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9835672Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9835915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9835992Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9836211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9836288Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9836543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9836634Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9836884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:59.9837033Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:59.9837037Z 2025-08-14T21:48:59.9837138Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9837338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9837403Z return mod(**inputs) 2025-08-14T21:48:59.9837808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9837936Z outputs = self.model( 2025-08-14T21:48:59.9838196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9838281Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9838533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9838606Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9838834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9838913Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9839169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9839259Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9839507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:59.9839627Z key_states = self.k_proj(current_states) 2025-08-14T21:48:59.9839663Z 2025-08-14T21:48:59.9839766Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9839972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9840037Z return mod(**inputs) 2025-08-14T21:48:59.9840290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9840366Z outputs = self.model( 2025-08-14T21:48:59.9840617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9840691Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9840976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9841052Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9841280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9841357Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9841604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9841700Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9841948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:59.9842042Z value_states = self.v_proj(current_states) 2025-08-14T21:48:59.9842046Z 2025-08-14T21:48:59.9842127Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9842208Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9842299Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9842382Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9842488Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9842704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9842772Z return mod(**inputs) 2025-08-14T21:48:59.9843046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9843125Z outputs = self.model( 2025-08-14T21:48:59.9843385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9843470Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9843730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9843805Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9844041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9844144Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9844412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9844506Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9844778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9844887Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9845194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:59.9845392Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:59.9845406Z 2025-08-14T21:48:59.9845518Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9845738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9845839Z return mod(**inputs) 2025-08-14T21:48:59.9846158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9846233Z outputs = self.model( 2025-08-14T21:48:59.9846585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9846677Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9846947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9847022Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9847253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9847363Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9847628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9847725Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9847993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9848094Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9848410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:59.9848527Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:59.9848531Z 2025-08-14T21:48:59.9848639Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9848855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9848927Z return mod(**inputs) 2025-08-14T21:48:59.9849212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9849286Z outputs = self.model( 2025-08-14T21:48:59.9849568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9849651Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9849915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9849991Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9850228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9850308Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9850577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9851449Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9851716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:59.9851812Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:59.9851816Z 2025-08-14T21:48:59.9851922Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9852142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9852211Z return mod(**inputs) 2025-08-14T21:48:59.9852477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9852554Z outputs = self.model( 2025-08-14T21:48:59.9852822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9852900Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9853172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9853287Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9853529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9853610Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9853876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9854008Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9854012Z 2025-08-14T21:48:59.9854116Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9854329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9854396Z return mod(**inputs) 2025-08-14T21:48:59.9854688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9854779Z outputs = self.model( 2025-08-14T21:48:59.9855021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9855091Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9855338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9855407Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9855625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9855700Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9855939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9856061Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9856265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:59.9856335Z return self.act(input) 2025-08-14T21:48:59.9856338Z 2025-08-14T21:48:59.9856443Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9856634Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9856705Z return mod(**inputs) 2025-08-14T21:48:59.9856949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9857016Z outputs = self.model( 2025-08-14T21:48:59.9857265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9857335Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9857598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9857679Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9857894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9857976Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9858220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:48:59.9858297Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:59.9858301Z 2025-08-14T21:48:59.9858403Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9858593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9858666Z return mod(**inputs) 2025-08-14T21:48:59.9858916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9858982Z outputs = self.model( 2025-08-14T21:48:59.9859269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9859340Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9859583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9859660Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9859880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9859962Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9860202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9860291Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9860558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:59.9860706Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:59.9860709Z 2025-08-14T21:48:59.9860812Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9861004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9861068Z return mod(**inputs) 2025-08-14T21:48:59.9861323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9861392Z outputs = self.model( 2025-08-14T21:48:59.9861639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9861716Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9861964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9862047Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9862265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9862340Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9862596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9862684Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9862941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:59.9863019Z key_states = self.k_proj(current_states) 2025-08-14T21:48:59.9863022Z 2025-08-14T21:48:59.9863121Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9863327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9863411Z return mod(**inputs) 2025-08-14T21:48:59.9863673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9863746Z outputs = self.model( 2025-08-14T21:48:59.9863988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9864064Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9864302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9864372Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9864589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9864664Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9864908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9865022Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9865281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:59.9865372Z value_states = self.v_proj(current_states) 2025-08-14T21:48:59.9865376Z 2025-08-14T21:48:59.9865452Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9865529Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9865609Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9865681Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9865776Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9865973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9866035Z return mod(**inputs) 2025-08-14T21:48:59.9866307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9866375Z outputs = self.model( 2025-08-14T21:48:59.9866623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9866701Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9866940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9867015Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9867223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9867298Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9867543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9867632Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9867873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9867976Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9868254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:59.9868388Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:59.9868392Z 2025-08-14T21:48:59.9868493Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9868689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9868760Z return mod(**inputs) 2025-08-14T21:48:59.9869018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9869112Z outputs = self.model( 2025-08-14T21:48:59.9869353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9869426Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9869672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9869740Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9869950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9870032Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9870270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9870363Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9870604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9870698Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9871026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:59.9871133Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:59.9871136Z 2025-08-14T21:48:59.9871239Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9871429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9871493Z return mod(**inputs) 2025-08-14T21:48:59.9871744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9871815Z outputs = self.model( 2025-08-14T21:48:59.9872107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9872193Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9872446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9872525Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9872745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9872821Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9873080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 321, in forward 2025-08-14T21:48:59.9873170Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:48:59.9873421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:59.9873507Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:59.9873512Z 2025-08-14T21:48:59.9873614Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9873828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9873894Z return mod(**inputs) 2025-08-14T21:48:59.9874140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9874214Z outputs = self.model( 2025-08-14T21:48:59.9874469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9874549Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9874799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9874870Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9875097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9875196Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9875448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9875575Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9875579Z 2025-08-14T21:48:59.9875677Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9875879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9875944Z return mod(**inputs) 2025-08-14T21:48:59.9876191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9876263Z outputs = self.model( 2025-08-14T21:48:59.9876513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9876597Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9876844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9876951Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9877180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9877256Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9877508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 332, in forward 2025-08-14T21:48:59.9877634Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9877847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:59.9877924Z return self.act(input) 2025-08-14T21:48:59.9877927Z 2025-08-14T21:48:59.9878046Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9878244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9878319Z return mod(**inputs) 2025-08-14T21:48:59.9878569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9878636Z outputs = self.model( 2025-08-14T21:48:59.9878891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9878964Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9879217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9879288Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9879502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9879589Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9879836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 334, in forward 2025-08-14T21:48:59.9879924Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:59.9879927Z 2025-08-14T21:48:59.9880026Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9880220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9880293Z return mod(**inputs) 2025-08-14T21:48:59.9880540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9880606Z outputs = self.model( 2025-08-14T21:48:59.9880860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1248, in forward 2025-08-14T21:48:59.9880933Z encoder_outputs = self.encoder( 2025-08-14T21:48:59.9881206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 861, in forward 2025-08-14T21:48:59.9881277Z layer_outputs = encoder_layer( 2025-08-14T21:48:59.9881498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9881580Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9881828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 336, in forward 2025-08-14T21:48:59.9881906Z hidden_states = residual + hidden_states 2025-08-14T21:48:59.9881918Z 2025-08-14T21:48:59.9882015Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9882250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9882320Z return mod(**inputs) 2025-08-14T21:48:59.9882568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9882635Z outputs = self.model( 2025-08-14T21:48:59.9882927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9883000Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9883258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9883329Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9883546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9883630Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9883878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:59.9883978Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:59.9884252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:59.9884406Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:59.9884410Z 2025-08-14T21:48:59.9884517Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9884715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9884780Z return mod(**inputs) 2025-08-14T21:48:59.9885046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9885115Z outputs = self.model( 2025-08-14T21:48:59.9885477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9885561Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9885838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9885927Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9886158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9886240Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9886514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:59.9886626Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:59.9886881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:59.9886963Z key_states = self.k_proj(current_states) 2025-08-14T21:48:59.9886967Z 2025-08-14T21:48:59.9887066Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9887271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9887362Z return mod(**inputs) 2025-08-14T21:48:59.9887627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9887695Z outputs = self.model( 2025-08-14T21:48:59.9887948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9888026Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9888278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9888350Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9888576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9888653Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9888913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:59.9889054Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:59.9889304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:59.9889396Z value_states = self.v_proj(current_states) 2025-08-14T21:48:59.9889399Z 2025-08-14T21:48:59.9889478Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9889557Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9889640Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9889713Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9889820Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9890016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9890080Z return mod(**inputs) 2025-08-14T21:48:59.9890357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9890425Z outputs = self.model( 2025-08-14T21:48:59.9890670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9890747Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9890991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9891068Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9891278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9891352Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9891600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:59.9891696Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:59.9891946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9892040Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9892318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:59.9892451Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:59.9892454Z 2025-08-14T21:48:59.9892552Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9892742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9892814Z return mod(**inputs) 2025-08-14T21:48:59.9893057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9893160Z outputs = self.model( 2025-08-14T21:48:59.9893404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9893477Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9893726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9893796Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9894014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9894088Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9894329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:59.9894427Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:59.9894667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9894760Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9895076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:59.9895183Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:59.9895187Z 2025-08-14T21:48:59.9895290Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9895484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9895546Z return mod(**inputs) 2025-08-14T21:48:59.9895797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9895863Z outputs = self.model( 2025-08-14T21:48:59.9896125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9896198Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9896441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9896519Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9896726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9896801Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9897047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:59.9897140Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:59.9897387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:59.9897464Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:59.9897471Z 2025-08-14T21:48:59.9897568Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9897769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9897832Z return mod(**inputs) 2025-08-14T21:48:59.9898070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9898141Z outputs = self.model( 2025-08-14T21:48:59.9898381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9898460Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9898704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9898773Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9898991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9899083Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9899335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:48:59.9899441Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:48:59.9899686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:59.9899839Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:59.9899843Z 2025-08-14T21:48:59.9899939Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9900133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9900202Z return mod(**inputs) 2025-08-14T21:48:59.9900448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9900522Z outputs = self.model( 2025-08-14T21:48:59.9900800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9900872Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9901121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9901191Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9901408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9901482Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9901725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:48:59.9901836Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:48:59.9902103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:59.9902185Z key_states = self.k_proj(current_states) 2025-08-14T21:48:59.9902197Z 2025-08-14T21:48:59.9902295Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9902487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9902558Z return mod(**inputs) 2025-08-14T21:48:59.9902801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9902866Z outputs = self.model( 2025-08-14T21:48:59.9903116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9903187Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9903439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9903511Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9903724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9903806Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9904049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:48:59.9904151Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:48:59.9904402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:59.9904485Z value_states = self.v_proj(current_states) 2025-08-14T21:48:59.9904489Z 2025-08-14T21:48:59.9904571Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9904648Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9904746Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9904826Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9904928Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9905119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9905190Z return mod(**inputs) 2025-08-14T21:48:59.9905434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9905504Z outputs = self.model( 2025-08-14T21:48:59.9905747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9905819Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9906068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9906139Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9906351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9906468Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9906710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:48:59.9906819Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:48:59.9907057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9907150Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9907436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:59.9907563Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:59.9907567Z 2025-08-14T21:48:59.9907690Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9907885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9907952Z return mod(**inputs) 2025-08-14T21:48:59.9908204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9908269Z outputs = self.model( 2025-08-14T21:48:59.9908518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9908594Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9908832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9908908Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9909115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9909189Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9909432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:48:59.9909532Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:48:59.9909773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9909864Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9910134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:59.9910240Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:59.9910244Z 2025-08-14T21:48:59.9910338Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9910526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9910614Z return mod(**inputs) 2025-08-14T21:48:59.9910853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9910923Z outputs = self.model( 2025-08-14T21:48:59.9911157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9911226Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9911472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9911544Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9911762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9911836Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9912079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:48:59.9912211Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:48:59.9912468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:59.9912551Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:59.9912554Z 2025-08-14T21:48:59.9912660Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9912858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9912931Z return mod(**inputs) 2025-08-14T21:48:59.9913191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9913254Z outputs = self.model( 2025-08-14T21:48:59.9913521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9913594Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9913837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9913914Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9914121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9914204Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9914443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:59.9914556Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9914560Z 2025-08-14T21:48:59.9914662Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9914850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9914921Z return mod(**inputs) 2025-08-14T21:48:59.9915163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9915231Z outputs = self.model( 2025-08-14T21:48:59.9915476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9915545Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9915784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9915862Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9916070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9916150Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9916390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:59.9916521Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9916729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:59.9916793Z return self.act(input) 2025-08-14T21:48:59.9916796Z 2025-08-14T21:48:59.9916895Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9917080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9917147Z return mod(**inputs) 2025-08-14T21:48:59.9917391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9917456Z outputs = self.model( 2025-08-14T21:48:59.9917695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9917775Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9918033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9918125Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9918338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9918413Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9918662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:48:59.9918740Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:59.9918743Z 2025-08-14T21:48:59.9918846Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9919041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9919105Z return mod(**inputs) 2025-08-14T21:48:59.9919372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9919439Z outputs = self.model( 2025-08-14T21:48:59.9919683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9919761Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9920004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9920082Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9920293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9920368Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9920623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:59.9920725Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:59.9920977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:59.9921134Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:59.9921138Z 2025-08-14T21:48:59.9921239Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9921443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9921508Z return mod(**inputs) 2025-08-14T21:48:59.9921756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9921830Z outputs = self.model( 2025-08-14T21:48:59.9922078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9922181Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9922432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9922506Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9922730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9922806Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9923053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:59.9923159Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:59.9923404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:59.9923489Z key_states = self.k_proj(current_states) 2025-08-14T21:48:59.9923492Z 2025-08-14T21:48:59.9923591Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9923791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9923899Z return mod(**inputs) 2025-08-14T21:48:59.9924151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9924223Z outputs = self.model( 2025-08-14T21:48:59.9924469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9924540Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9924795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9924865Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9925078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9925183Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9925525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:59.9925640Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:59.9925901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:59.9925992Z value_states = self.v_proj(current_states) 2025-08-14T21:48:59.9925997Z 2025-08-14T21:48:59.9926089Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9926171Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9926251Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9926341Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9926446Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9926667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9926733Z return mod(**inputs) 2025-08-14T21:48:59.9926985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9927064Z outputs = self.model( 2025-08-14T21:48:59.9927312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9927385Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9927638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9927710Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9927932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9928010Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9928262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:59.9928392Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:59.9928645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9928748Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9929037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:59.9929168Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:59.9929172Z 2025-08-14T21:48:59.9929279Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9929478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9929542Z return mod(**inputs) 2025-08-14T21:48:59.9929801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9929870Z outputs = self.model( 2025-08-14T21:48:59.9930183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9930260Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9930511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9930590Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9930807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9930884Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9931138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:59.9931252Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:59.9931508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9931606Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9931887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:59.9932002Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:59.9932005Z 2025-08-14T21:48:59.9932105Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9932306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9932370Z return mod(**inputs) 2025-08-14T21:48:59.9932621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9932694Z outputs = self.model( 2025-08-14T21:48:59.9932943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9933020Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9933281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9933353Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9933578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9933656Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9933902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:59.9934005Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:59.9934309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:59.9934417Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:59.9934420Z 2025-08-14T21:48:59.9934524Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9934721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9934796Z return mod(**inputs) 2025-08-14T21:48:59.9935042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9935110Z outputs = self.model( 2025-08-14T21:48:59.9935365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9935436Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9935689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9935759Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9935981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9936110Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9936365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 424, in forward 2025-08-14T21:48:59.9936451Z hidden_states = residual + hidden_states 2025-08-14T21:48:59.9936455Z 2025-08-14T21:48:59.9936564Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9936748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9936818Z return mod(**inputs) 2025-08-14T21:48:59.9937054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9937117Z outputs = self.model( 2025-08-14T21:48:59.9937376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9937448Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9937829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9937902Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9938111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9938193Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9938433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:48:59.9938533Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:48:59.9938775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:59.9938916Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:59.9938919Z 2025-08-14T21:48:59.9939025Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9939215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9939278Z return mod(**inputs) 2025-08-14T21:48:59.9939521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9939585Z outputs = self.model( 2025-08-14T21:48:59.9939827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9939896Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9940133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9940209Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9940467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9940541Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9940787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:48:59.9940886Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:48:59.9941127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:59.9941202Z key_states = self.k_proj(current_states) 2025-08-14T21:48:59.9941206Z 2025-08-14T21:48:59.9941299Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9941492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9941554Z return mod(**inputs) 2025-08-14T21:48:59.9941827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9941920Z outputs = self.model( 2025-08-14T21:48:59.9942198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9942281Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9942540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9942608Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9942824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9942897Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9943144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:48:59.9943272Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:48:59.9943514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:59.9943606Z value_states = self.v_proj(current_states) 2025-08-14T21:48:59.9943609Z 2025-08-14T21:48:59.9943685Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9943767Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9943842Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9943916Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9944019Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9944204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9944266Z return mod(**inputs) 2025-08-14T21:48:59.9944512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9944579Z outputs = self.model( 2025-08-14T21:48:59.9944827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9944900Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9945135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9945213Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9945424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9945498Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9945748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:48:59.9945849Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:48:59.9946102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9946219Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9946498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:59.9946633Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:59.9946637Z 2025-08-14T21:48:59.9946737Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9946937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9947001Z return mod(**inputs) 2025-08-14T21:48:59.9947261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9947331Z outputs = self.model( 2025-08-14T21:48:59.9947575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9947647Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9947942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9948014Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9948232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9948306Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9948549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:48:59.9948658Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:48:59.9948899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9948992Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9949301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:59.9949408Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:59.9949412Z 2025-08-14T21:48:59.9949517Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9949709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9949772Z return mod(**inputs) 2025-08-14T21:48:59.9950033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9950097Z outputs = self.model( 2025-08-14T21:48:59.9950341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9950409Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9950656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9950739Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9950955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9951032Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9951287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:48:59.9951391Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:48:59.9951644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:59.9951728Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:59.9951731Z 2025-08-14T21:48:59.9951832Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9952059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9952124Z return mod(**inputs) 2025-08-14T21:48:59.9952384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9952451Z outputs = self.model( 2025-08-14T21:48:59.9952704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9952781Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9953081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9953149Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9953368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9953441Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9953699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:59.9953844Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9953848Z 2025-08-14T21:48:59.9953943Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9954138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9954203Z return mod(**inputs) 2025-08-14T21:48:59.9954456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9954522Z outputs = self.model( 2025-08-14T21:48:59.9954773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9954853Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9955118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9955194Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9955422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9955499Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9955754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:59.9955871Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9956082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:59.9956158Z return self.act(input) 2025-08-14T21:48:59.9956162Z 2025-08-14T21:48:59.9956261Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9956457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9956531Z return mod(**inputs) 2025-08-14T21:48:59.9956781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9956860Z outputs = self.model( 2025-08-14T21:48:59.9957108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9957180Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9957435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9957508Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9957732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9957809Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9958058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:48:59.9958166Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:59.9958171Z 2025-08-14T21:48:59.9958272Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9958468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9958541Z return mod(**inputs) 2025-08-14T21:48:59.9958787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9958861Z outputs = self.model( 2025-08-14T21:48:59.9959110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9959182Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9959437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9959511Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9959752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9959854Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9960104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:59.9960208Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:59.9960456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:59.9960604Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:59.9960607Z 2025-08-14T21:48:59.9960715Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9960929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9961007Z return mod(**inputs) 2025-08-14T21:48:59.9961260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9961327Z outputs = self.model( 2025-08-14T21:48:59.9961587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9961658Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9961908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9961985Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9962205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9962289Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9962542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:59.9962641Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:59.9962902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:59.9962983Z key_states = self.k_proj(current_states) 2025-08-14T21:48:59.9962986Z 2025-08-14T21:48:59.9963092Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9963291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9963356Z return mod(**inputs) 2025-08-14T21:48:59.9963613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9963680Z outputs = self.model( 2025-08-14T21:48:59.9963929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9964030Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9964280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9964363Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9964580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9964655Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9964910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:59.9965007Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:59.9965467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:59.9965562Z value_states = self.v_proj(current_states) 2025-08-14T21:48:59.9965569Z 2025-08-14T21:48:59.9965651Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9965741Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9965850Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9965948Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9966066Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9966293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9966362Z return mod(**inputs) 2025-08-14T21:48:59.9966649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9966719Z outputs = self.model( 2025-08-14T21:48:59.9966996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9967074Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9967358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9967441Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9967669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9967753Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9968009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:59.9968104Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:59.9968358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9968452Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9968740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:59.9968881Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:59.9968884Z 2025-08-14T21:48:59.9968988Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9969193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9969259Z return mod(**inputs) 2025-08-14T21:48:59.9969511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9969585Z outputs = self.model( 2025-08-14T21:48:59.9969836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9969915Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9970162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9970236Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9970485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9970565Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9970815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:59.9970917Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:59.9971168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9971278Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9971550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:59.9971652Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:59.9971655Z 2025-08-14T21:48:59.9971762Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9971954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9972065Z return mod(**inputs) 2025-08-14T21:48:59.9972311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9972377Z outputs = self.model( 2025-08-14T21:48:59.9972632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9972702Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9972938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9973015Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9973219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9973321Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9973561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:59.9973655Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:59.9973902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:59.9973979Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:59.9973983Z 2025-08-14T21:48:59.9974084Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9974274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9974336Z return mod(**inputs) 2025-08-14T21:48:59.9974587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9974653Z outputs = self.model( 2025-08-14T21:48:59.9974898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9974978Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9975220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9975295Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9975504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9975584Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9975832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:48:59.9975934Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:48:59.9976178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:59.9976350Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:59.9976355Z 2025-08-14T21:48:59.9976450Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9976644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9976705Z return mod(**inputs) 2025-08-14T21:48:59.9976941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9977012Z outputs = self.model( 2025-08-14T21:48:59.9977246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9977321Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9977559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9977628Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9977862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9977956Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9978192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:48:59.9978298Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:48:59.9978536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:48:59.9978618Z key_states = self.k_proj(current_states) 2025-08-14T21:48:59.9978621Z 2025-08-14T21:48:59.9978717Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9978906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9978998Z return mod(**inputs) 2025-08-14T21:48:59.9979240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9979314Z outputs = self.model( 2025-08-14T21:48:59.9979549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9979618Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9979859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9979928Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9980132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9980214Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9980447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:48:59.9980556Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:48:59.9980792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:48:59.9980872Z value_states = self.v_proj(current_states) 2025-08-14T21:48:59.9980875Z 2025-08-14T21:48:59.9980959Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9981033Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9981105Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9981185Z cudagraph partition due to non gpu ops 2025-08-14T21:48:59.9981279Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9981473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9981535Z return mod(**inputs) 2025-08-14T21:48:59.9981771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9981871Z outputs = self.model( 2025-08-14T21:48:59.9982114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9982182Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9982423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9982490Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9982703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9982775Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9983011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:48:59.9983119Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:48:59.9983354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9983489Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9983760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:48:59.9983883Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:48:59.9983887Z 2025-08-14T21:48:59.9983990Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9984178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9984239Z return mod(**inputs) 2025-08-14T21:48:59.9984482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9984545Z outputs = self.model( 2025-08-14T21:48:59.9984805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9984877Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9985116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9985192Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9985399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9985477Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9985711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:48:59.9985811Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:48:59.9986058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:48:59.9986150Z attn_output, attn_weights = attention_interface( 2025-08-14T21:48:59.9986422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:48:59.9986534Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:48:59.9986538Z 2025-08-14T21:48:59.9986633Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9986826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9986889Z return mod(**inputs) 2025-08-14T21:48:59.9987130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9987203Z outputs = self.model( 2025-08-14T21:48:59.9987441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9987537Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9987776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9987846Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9988059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9988133Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9988367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:48:59.9988477Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:48:59.9988709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:48:59.9988792Z attn_output = self.out_proj(attn_output) 2025-08-14T21:48:59.9988796Z 2025-08-14T21:48:59.9988892Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9989076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9989185Z return mod(**inputs) 2025-08-14T21:48:59.9989425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9989496Z outputs = self.model( 2025-08-14T21:48:59.9989730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9989798Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9990040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9990107Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9990309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9990410Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9990650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 441, in forward 2025-08-14T21:48:59.9990736Z hidden_states = residual + hidden_states 2025-08-14T21:48:59.9990739Z 2025-08-14T21:48:59.9990832Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9991021Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9991090Z return mod(**inputs) 2025-08-14T21:48:59.9991329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9991393Z outputs = self.model( 2025-08-14T21:48:59.9991638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9991708Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9991954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9992024Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9992234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9992314Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9992554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:59.9992679Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9992682Z 2025-08-14T21:48:59.9992780Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9992972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9993042Z return mod(**inputs) 2025-08-14T21:48:59.9993310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9993378Z outputs = self.model( 2025-08-14T21:48:59.9993638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9993708Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9993954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9994032Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9994236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9994314Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9994544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:48:59.9994663Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:48:59.9994859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:48:59.9994959Z return self.act(input) 2025-08-14T21:48:59.9994963Z 2025-08-14T21:48:59.9995065Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9995247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9995307Z return mod(**inputs) 2025-08-14T21:48:59.9995545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9995608Z outputs = self.model( 2025-08-14T21:48:59.9995842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9995910Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9996157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9996236Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9996436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9996507Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9996745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:48:59.9996819Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:48:59.9996823Z 2025-08-14T21:48:59.9996922Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9997105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9997165Z return mod(**inputs) 2025-08-14T21:48:59.9997402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9997466Z outputs = self.model( 2025-08-14T21:48:59.9997714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:48:59.9997784Z decoder_outputs = self.decoder( 2025-08-14T21:48:59.9998022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:48:59.9998099Z layer_outputs = decoder_layer( 2025-08-14T21:48:59.9998305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:48:59.9998377Z return super().__call__(*args, **kwargs) 2025-08-14T21:48:59.9998627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:48:59.9998720Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:48:59.9998989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:48:59.9999134Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:48:59.9999138Z 2025-08-14T21:48:59.9999235Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:48:59.9999429Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:48:59.9999491Z return mod(**inputs) 2025-08-14T21:48:59.9999788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:48:59.9999852Z outputs = self.model( 2025-08-14T21:49:00.0000091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0000168Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0000408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0000478Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0000727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0000801Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0001051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0001147Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0001392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:00.0001477Z key_states = self.k_proj(current_states) 2025-08-14T21:49:00.0001480Z 2025-08-14T21:49:00.0001577Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0001798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0001865Z return mod(**inputs) 2025-08-14T21:49:00.0002119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0002194Z outputs = self.model( 2025-08-14T21:49:00.0002444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0002514Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0002770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0002841Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0003066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0003141Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0003386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0003491Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0003740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:00.0003822Z value_states = self.v_proj(current_states) 2025-08-14T21:49:00.0003834Z 2025-08-14T21:49:00.0003911Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0003988Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0004071Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0004145Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0004243Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0004443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0004507Z return mod(**inputs) 2025-08-14T21:49:00.0004781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0004857Z outputs = self.model( 2025-08-14T21:49:00.0005113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0005191Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0005519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0005597Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0005828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0005912Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0006197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0006313Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0006600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0006760Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0007085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:00.0007228Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:00.0007233Z 2025-08-14T21:49:00.0007356Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0007551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0007624Z return mod(**inputs) 2025-08-14T21:49:00.0007880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0007967Z outputs = self.model( 2025-08-14T21:49:00.0008214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0008287Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0008523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0008599Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0008804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0008885Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0009120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0009210Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0009457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0009548Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0009829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:00.0009931Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:00.0009934Z 2025-08-14T21:49:00.0010029Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0010220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0010281Z return mod(**inputs) 2025-08-14T21:49:00.0010523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0010596Z outputs = self.model( 2025-08-14T21:49:00.0010833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0010931Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0011170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0011240Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0011453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0011526Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0011769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0011862Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0012127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:00.0012214Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:00.0012219Z 2025-08-14T21:49:00.0012319Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0012533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0012629Z return mod(**inputs) 2025-08-14T21:49:00.0012892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0012968Z outputs = self.model( 2025-08-14T21:49:00.0013228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0013303Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0013578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0013654Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0013922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0014017Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0014264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0014375Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0014617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:00.0014761Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:00.0014772Z 2025-08-14T21:49:00.0014870Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0015064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0015136Z return mod(**inputs) 2025-08-14T21:49:00.0015389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0015456Z outputs = self.model( 2025-08-14T21:49:00.0015713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0015785Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0016043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0016113Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0016328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0016411Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0016659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0016762Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0017041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:00.0017124Z key_states = self.k_proj(current_states) 2025-08-14T21:49:00.0017127Z 2025-08-14T21:49:00.0017236Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0017434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0017498Z return mod(**inputs) 2025-08-14T21:49:00.0017762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0017827Z outputs = self.model( 2025-08-14T21:49:00.0018069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0018146Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0018398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0018477Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0018728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0018814Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0019061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0019161Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0019406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:00.0019489Z value_states = self.v_proj(current_states) 2025-08-14T21:49:00.0019492Z 2025-08-14T21:49:00.0019570Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0019655Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0019792Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0019867Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0019976Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0020173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0020245Z return mod(**inputs) 2025-08-14T21:49:00.0020500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0020578Z outputs = self.model( 2025-08-14T21:49:00.0020831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0020904Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0021148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0021227Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0021446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0021534Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0021797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0021900Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0022150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0022246Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0022538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:00.0022668Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:00.0022672Z 2025-08-14T21:49:00.0022794Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0022995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0023064Z return mod(**inputs) 2025-08-14T21:49:00.0023316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0023391Z outputs = self.model( 2025-08-14T21:49:00.0023640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0023720Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0023971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0024043Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0024277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0024354Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0024611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0024746Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0024986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0025083Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0025361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:00.0025463Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:00.0025467Z 2025-08-14T21:49:00.0025570Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0025779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0025854Z return mod(**inputs) 2025-08-14T21:49:00.0026101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0026169Z outputs = self.model( 2025-08-14T21:49:00.0026416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0026487Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0026731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0026806Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0027017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0027101Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0027342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0027446Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0027698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:00.0027777Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:00.0027780Z 2025-08-14T21:49:00.0027884Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0028073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0028137Z return mod(**inputs) 2025-08-14T21:49:00.0028386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0028451Z outputs = self.model( 2025-08-14T21:49:00.0028693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0028790Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0029035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0029114Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0029324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0029399Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0029647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:00.0029761Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:00.0029765Z 2025-08-14T21:49:00.0029871Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0030062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0030129Z return mod(**inputs) 2025-08-14T21:49:00.0030394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0030475Z outputs = self.model( 2025-08-14T21:49:00.0030720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0030798Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0031042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0031118Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0031331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0031405Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0031666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:00.0031785Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:00.0031996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:49:00.0032073Z return self.act(input) 2025-08-14T21:49:00.0032077Z 2025-08-14T21:49:00.0032176Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0032376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0032440Z return mod(**inputs) 2025-08-14T21:49:00.0032697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0032771Z outputs = self.model( 2025-08-14T21:49:00.0033020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0033103Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0033349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0033426Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0033648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0033736Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0033973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:49:00.0034060Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:00.0034064Z 2025-08-14T21:49:00.0034164Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0034360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0034425Z return mod(**inputs) 2025-08-14T21:49:00.0034695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0034772Z outputs = self.model( 2025-08-14T21:49:00.0035078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0035155Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0035395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0035463Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0035679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0035753Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0035989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-14T21:49:00.0036075Z hidden_states = residual + hidden_states 2025-08-14T21:49:00.0036078Z 2025-08-14T21:49:00.0036175Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0036411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0036478Z return mod(**inputs) 2025-08-14T21:49:00.0036732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0036808Z outputs = self.model( 2025-08-14T21:49:00.0037061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0037133Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0037439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0037508Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0037944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0038038Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0038275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0038377Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0038611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:00.0038759Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:00.0038763Z 2025-08-14T21:49:00.0038858Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0039045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0039114Z return mod(**inputs) 2025-08-14T21:49:00.0039353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0039419Z outputs = self.model( 2025-08-14T21:49:00.0039662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0039732Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0039975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0040044Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0040250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0040330Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0040566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0040693Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0040926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:00.0041003Z key_states = self.k_proj(current_states) 2025-08-14T21:49:00.0041006Z 2025-08-14T21:49:00.0041108Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0041293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0041353Z return mod(**inputs) 2025-08-14T21:49:00.0041595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0041659Z outputs = self.model( 2025-08-14T21:49:00.0041902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0041970Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0042215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0042332Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0042581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0042658Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0042913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0043009Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0043263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:00.0043348Z value_states = self.v_proj(current_states) 2025-08-14T21:49:00.0043352Z 2025-08-14T21:49:00.0043432Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0043534Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0043616Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0043700Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0043805Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0044005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0044078Z return mod(**inputs) 2025-08-14T21:49:00.0044332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0044401Z outputs = self.model( 2025-08-14T21:49:00.0044660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0044735Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0044993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0045068Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0045361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0045461Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0045729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0045834Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0046108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0046212Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0046526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:00.0046664Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:00.0046701Z 2025-08-14T21:49:00.0046802Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0047000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0047065Z return mod(**inputs) 2025-08-14T21:49:00.0047314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0047379Z outputs = self.model( 2025-08-14T21:49:00.0047617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0047697Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0047932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0048000Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0048214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0048289Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0048588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0048685Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0048929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0049029Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0049313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:00.0049420Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:00.0049424Z 2025-08-14T21:49:00.0049519Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0049721Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0049795Z return mod(**inputs) 2025-08-14T21:49:00.0050035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0050101Z outputs = self.model( 2025-08-14T21:49:00.0050346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0050416Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0050660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0050729Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0050934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0051017Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0051253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0051348Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0051589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:00.0051666Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:00.0051670Z 2025-08-14T21:49:00.0051771Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0051993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0052055Z return mod(**inputs) 2025-08-14T21:49:00.0052308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0052373Z outputs = self.model( 2025-08-14T21:49:00.0052622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0052712Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0052958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0053036Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0053255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0053329Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0053572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0053673Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0053916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:00.0054057Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:00.0054062Z 2025-08-14T21:49:00.0054159Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0054389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0054454Z return mod(**inputs) 2025-08-14T21:49:00.0054703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0054769Z outputs = self.model( 2025-08-14T21:49:00.0055011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0055090Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0055332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0055402Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0055636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0055714Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0055968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0056071Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0056315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:00.0056399Z key_states = self.k_proj(current_states) 2025-08-14T21:49:00.0056403Z 2025-08-14T21:49:00.0056503Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0056702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0056766Z return mod(**inputs) 2025-08-14T21:49:00.0057013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0057089Z outputs = self.model( 2025-08-14T21:49:00.0057332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0057404Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0057654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0057725Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0057943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0058019Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0058261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0058373Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0058636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:00.0058729Z value_states = self.v_proj(current_states) 2025-08-14T21:49:00.0058732Z 2025-08-14T21:49:00.0058813Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0058891Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0058972Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0059046Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0059144Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0059339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0059403Z return mod(**inputs) 2025-08-14T21:49:00.0059643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0059717Z outputs = self.model( 2025-08-14T21:49:00.0059958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0060072Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0060316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0060385Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0060604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0060680Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0060930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0061032Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0061286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0061393Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0061675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:00.0061801Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:00.0061812Z 2025-08-14T21:49:00.0061912Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0062111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0062184Z return mod(**inputs) 2025-08-14T21:49:00.0062440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0062507Z outputs = self.model( 2025-08-14T21:49:00.0062770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0062844Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0063119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0063189Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0063398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0063482Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0063721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0063823Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0064072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0064166Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0064473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:00.0064579Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:00.0064582Z 2025-08-14T21:49:00.0064681Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0064882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0064947Z return mod(**inputs) 2025-08-14T21:49:00.0065198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0065263Z outputs = self.model( 2025-08-14T21:49:00.0065505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0065582Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0065825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0065918Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0066155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0066233Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0066480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0066581Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0066827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:00.0066915Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:00.0066919Z 2025-08-14T21:49:00.0067017Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0067236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0067305Z return mod(**inputs) 2025-08-14T21:49:00.0067560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0067634Z outputs = self.model( 2025-08-14T21:49:00.0067897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0067967Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0068219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0068289Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0068509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0068585Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0068836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:00.0068960Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:00.0068964Z 2025-08-14T21:49:00.0069061Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0069261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0069324Z return mod(**inputs) 2025-08-14T21:49:00.0069569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0069644Z outputs = self.model( 2025-08-14T21:49:00.0069889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0069961Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0070217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0070315Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0070536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0070613Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0070850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:00.0070972Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:00.0071176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:49:00.0071243Z return self.act(input) 2025-08-14T21:49:00.0071247Z 2025-08-14T21:49:00.0071350Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0071541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0071614Z return mod(**inputs) 2025-08-14T21:49:00.0071878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0071959Z outputs = self.model( 2025-08-14T21:49:00.0072235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0072312Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0072594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0072679Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0072922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0073013Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0073300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:49:00.0073390Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:00.0073396Z 2025-08-14T21:49:00.0073513Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0073724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0073802Z return mod(**inputs) 2025-08-14T21:49:00.0074066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0074133Z outputs = self.model( 2025-08-14T21:49:00.0074384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0074456Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0074701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0074782Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0074997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0075082Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0075326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0075421Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0075674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:00.0075821Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:00.0075824Z 2025-08-14T21:49:00.0075928Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0076122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0076212Z return mod(**inputs) 2025-08-14T21:49:00.0076462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0076530Z outputs = self.model( 2025-08-14T21:49:00.0076771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0076851Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0077093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0077169Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0077382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0077459Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0077717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0077812Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0078096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:00.0078174Z key_states = self.k_proj(current_states) 2025-08-14T21:49:00.0078177Z 2025-08-14T21:49:00.0078273Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0078466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0078528Z return mod(**inputs) 2025-08-14T21:49:00.0078764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0078837Z outputs = self.model( 2025-08-14T21:49:00.0079072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0079177Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0079419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0079488Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0079701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0079775Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0080014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0080116Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0080357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:00.0080445Z value_states = self.v_proj(current_states) 2025-08-14T21:49:00.0080448Z 2025-08-14T21:49:00.0080529Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0080607Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0080693Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0080767Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0080864Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0081066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0081131Z return mod(**inputs) 2025-08-14T21:49:00.0081388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0081455Z outputs = self.model( 2025-08-14T21:49:00.0081703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0081783Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0082033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0082129Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0082349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0082426Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0082683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0082779Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0083041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0083152Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0083456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:00.0083605Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:00.0083609Z 2025-08-14T21:49:00.0083735Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0084064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0084148Z return mod(**inputs) 2025-08-14T21:49:00.0084419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0084497Z outputs = self.model( 2025-08-14T21:49:00.0084774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0084848Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0085108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0085180Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0085516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0085615Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0085870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0085980Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0086242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0086346Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0086659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:00.0086765Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:00.0086769Z 2025-08-14T21:49:00.0086877Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0087068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0087134Z return mod(**inputs) 2025-08-14T21:49:00.0087385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0087453Z outputs = self.model( 2025-08-14T21:49:00.0087695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0087774Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0088020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0088097Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0088303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0088401Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0088645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0088738Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0088971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:00.0089054Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:00.0089057Z 2025-08-14T21:49:00.0089150Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0089344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0089404Z return mod(**inputs) 2025-08-14T21:49:00.0089640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0089714Z outputs = self.model( 2025-08-14T21:49:00.0089956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0090067Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0090313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0090386Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0090603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0090677Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0090924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 424, in forward 2025-08-14T21:49:00.0091010Z hidden_states = residual + hidden_states 2025-08-14T21:49:00.0091014Z 2025-08-14T21:49:00.0091111Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0091340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0091417Z return mod(**inputs) 2025-08-14T21:49:00.0091657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0091728Z outputs = self.model( 2025-08-14T21:49:00.0091963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0092041Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0092282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0092353Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0092569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0092643Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0092886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0093001Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0093240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:00.0093392Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:00.0093395Z 2025-08-14T21:49:00.0093493Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0093685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0093756Z return mod(**inputs) 2025-08-14T21:49:00.0094003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0094077Z outputs = self.model( 2025-08-14T21:49:00.0094342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0094417Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0094667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0094737Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0094949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0095030Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0095269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0095376Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0095619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:00.0095698Z key_states = self.k_proj(current_states) 2025-08-14T21:49:00.0095721Z 2025-08-14T21:49:00.0095843Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0096036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0096099Z return mod(**inputs) 2025-08-14T21:49:00.0096352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0096416Z outputs = self.model( 2025-08-14T21:49:00.0096665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0096735Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0096978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0097072Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0097285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0097367Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0097609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0097710Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0097957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:00.0098040Z value_states = self.v_proj(current_states) 2025-08-14T21:49:00.0098043Z 2025-08-14T21:49:00.0098118Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0098201Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0098276Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0098357Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0098458Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0098648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0098721Z return mod(**inputs) 2025-08-14T21:49:00.0098965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0099029Z outputs = self.model( 2025-08-14T21:49:00.0099281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0099352Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0099604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0099673Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0099887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0099990Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0100241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0100344Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0100601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0100695Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0100990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:00.0101114Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:00.0101118Z 2025-08-14T21:49:00.0101217Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0101419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0101485Z return mod(**inputs) 2025-08-14T21:49:00.0101771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0101838Z outputs = self.model( 2025-08-14T21:49:00.0102081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0102164Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0102426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0102502Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0102738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0102820Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0103109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0103223Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0103489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0103601Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0103911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:00.0104043Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:00.0104047Z 2025-08-14T21:49:00.0104147Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0104342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0104416Z return mod(**inputs) 2025-08-14T21:49:00.0104666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0104735Z outputs = self.model( 2025-08-14T21:49:00.0104992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0105065Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0105328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0105398Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0105604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0105688Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0105928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0106056Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0106298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:00.0106377Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:00.0106381Z 2025-08-14T21:49:00.0106486Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0106675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0106738Z return mod(**inputs) 2025-08-14T21:49:00.0106988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0107053Z outputs = self.model( 2025-08-14T21:49:00.0107300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0107372Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0107620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0107732Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0107945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0108025Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0108267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:00.0108381Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:00.0108385Z 2025-08-14T21:49:00.0108493Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0108685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0108749Z return mod(**inputs) 2025-08-14T21:49:00.0109015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0109083Z outputs = self.model( 2025-08-14T21:49:00.0109337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0109409Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0109650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0109730Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0109939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0110013Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0110260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:00.0110373Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:00.0110596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:49:00.0110664Z return self.act(input) 2025-08-14T21:49:00.0110667Z 2025-08-14T21:49:00.0110763Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0110955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0111016Z return mod(**inputs) 2025-08-14T21:49:00.0111257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0111320Z outputs = self.model( 2025-08-14T21:49:00.0111554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0111630Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0111870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0111960Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0112190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0112266Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0112518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:49:00.0112599Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:00.0112602Z 2025-08-14T21:49:00.0112703Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0112904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0112967Z return mod(**inputs) 2025-08-14T21:49:00.0113219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0113288Z outputs = self.model( 2025-08-14T21:49:00.0113552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0113688Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0113947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0114016Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0114236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0114311Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0114558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0114652Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0114912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:00.0115071Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:00.0115075Z 2025-08-14T21:49:00.0115171Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0115369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0115432Z return mod(**inputs) 2025-08-14T21:49:00.0115681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0115755Z outputs = self.model( 2025-08-14T21:49:00.0116006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0116080Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0116343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0116418Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0116647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0116725Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0116975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0117082Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0117331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:00.0117420Z key_states = self.k_proj(current_states) 2025-08-14T21:49:00.0117431Z 2025-08-14T21:49:00.0117528Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0117722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0117812Z return mod(**inputs) 2025-08-14T21:49:00.0118060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0118126Z outputs = self.model( 2025-08-14T21:49:00.0118379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0118450Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0118700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0118770Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0118983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0119066Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0119307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0119403Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0119685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:00.0119769Z value_states = self.v_proj(current_states) 2025-08-14T21:49:00.0119772Z 2025-08-14T21:49:00.0119857Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0119935Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0120010Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0120093Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0120193Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0120386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0120457Z return mod(**inputs) 2025-08-14T21:49:00.0120718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0120795Z outputs = self.model( 2025-08-14T21:49:00.0121042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0121114Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0121364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0121434Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0121649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0121732Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0121978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0122081Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0122331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0122431Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0122728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:00.0122858Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:00.0122862Z 2025-08-14T21:49:00.0122972Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0123167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0123232Z return mod(**inputs) 2025-08-14T21:49:00.0123490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0123559Z outputs = self.model( 2025-08-14T21:49:00.0123832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0123915Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0124166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0124245Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0124468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0124543Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0124796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0124889Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0125141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0125306Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0125667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:00.0125796Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:00.0125801Z 2025-08-14T21:49:00.0125912Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0126125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0126208Z return mod(**inputs) 2025-08-14T21:49:00.0126478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0126554Z outputs = self.model( 2025-08-14T21:49:00.0126820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0126903Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0127160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0127230Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0127447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0127526Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0127775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0127878Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0128128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:00.0128208Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:00.0128214Z 2025-08-14T21:49:00.0128325Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0128520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0128595Z return mod(**inputs) 2025-08-14T21:49:00.0128845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0128912Z outputs = self.model( 2025-08-14T21:49:00.0129166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0129239Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0129488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0129566Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0129784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0129888Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0130147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0130252Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0130513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:00.0130662Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:00.0130665Z 2025-08-14T21:49:00.0130774Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0130974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0131040Z return mod(**inputs) 2025-08-14T21:49:00.0131303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0131371Z outputs = self.model( 2025-08-14T21:49:00.0131640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0131737Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0131990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0132070Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0132288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0132366Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0132621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0132726Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0132998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:00.0133080Z key_states = self.k_proj(current_states) 2025-08-14T21:49:00.0133085Z 2025-08-14T21:49:00.0133187Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0133391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0133457Z return mod(**inputs) 2025-08-14T21:49:00.0133708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0133782Z outputs = self.model( 2025-08-14T21:49:00.0134032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0134111Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0134366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0134442Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0134671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0134747Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0134999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0135104Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0135354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:00.0135446Z value_states = self.v_proj(current_states) 2025-08-14T21:49:00.0135450Z 2025-08-14T21:49:00.0135529Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0135607Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0135695Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0135800Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0135909Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0136109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0136175Z return mod(**inputs) 2025-08-14T21:49:00.0136436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0136505Z outputs = self.model( 2025-08-14T21:49:00.0136757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0136836Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0137088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0137165Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0137388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0137497Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0137924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0138047Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0138289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0138389Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0138667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:00.0138800Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:00.0138804Z 2025-08-14T21:49:00.0138928Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0139122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0139197Z return mod(**inputs) 2025-08-14T21:49:00.0139442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0139516Z outputs = self.model( 2025-08-14T21:49:00.0139761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0139834Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0140087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0140160Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0140376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0140464Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0140713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0140826Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0141070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0141163Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0141450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:00.0141552Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:00.0141556Z 2025-08-14T21:49:00.0141660Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0141853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0141949Z return mod(**inputs) 2025-08-14T21:49:00.0142205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0142273Z outputs = self.model( 2025-08-14T21:49:00.0142516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0142593Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0142879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0142959Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0143177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0143253Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0143510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0143625Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0143915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:00.0143995Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:00.0143998Z 2025-08-14T21:49:00.0144095Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0144293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0144356Z return mod(**inputs) 2025-08-14T21:49:00.0144597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0144671Z outputs = self.model( 2025-08-14T21:49:00.0144912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0145008Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0145251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0145323Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0145542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0145617Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0145857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 441, in forward 2025-08-14T21:49:00.0145942Z hidden_states = residual + hidden_states 2025-08-14T21:49:00.0145945Z 2025-08-14T21:49:00.0146041Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0146238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0146301Z return mod(**inputs) 2025-08-14T21:49:00.0146547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0146623Z outputs = self.model( 2025-08-14T21:49:00.0146864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0146942Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0147183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0147254Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0147470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0147544Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0147784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:00.0147926Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:00.0147931Z 2025-08-14T21:49:00.0148029Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0148224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0148288Z return mod(**inputs) 2025-08-14T21:49:00.0148535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0148609Z outputs = self.model( 2025-08-14T21:49:00.0148847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0148934Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0149174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0149245Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0149460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0149567Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0149816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:00.0149936Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:00.0150140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:49:00.0150214Z return self.act(input) 2025-08-14T21:49:00.0150218Z 2025-08-14T21:49:00.0150314Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0150507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0150578Z return mod(**inputs) 2025-08-14T21:49:00.0150839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0150910Z outputs = self.model( 2025-08-14T21:49:00.0151163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0151234Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0151483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0151553Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0151764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0151848Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0152088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:49:00.0152175Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:00.0152180Z 2025-08-14T21:49:00.0152280Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0152479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0152551Z return mod(**inputs) 2025-08-14T21:49:00.0152802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0152869Z outputs = self.model( 2025-08-14T21:49:00.0153137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0153208Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0153460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0153530Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0153740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0153842Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0154085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0154186Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0154435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:00.0154575Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:00.0154578Z 2025-08-14T21:49:00.0154678Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0154863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0154925Z return mod(**inputs) 2025-08-14T21:49:00.0155171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0155235Z outputs = self.model( 2025-08-14T21:49:00.0155516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0155586Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0155829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0155907Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0156117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0156197Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0156436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0156545Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0156806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:00.0156884Z key_states = self.k_proj(current_states) 2025-08-14T21:49:00.0156887Z 2025-08-14T21:49:00.0156981Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0157176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0157237Z return mod(**inputs) 2025-08-14T21:49:00.0157483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0157547Z outputs = self.model( 2025-08-14T21:49:00.0157786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0157863Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0158105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0158177Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0158396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0158469Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0158716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0158810Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0159053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:00.0159144Z value_states = self.v_proj(current_states) 2025-08-14T21:49:00.0159148Z 2025-08-14T21:49:00.0159228Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0159315Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0159422Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0159498Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0159609Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0159811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0159874Z return mod(**inputs) 2025-08-14T21:49:00.0160125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0160191Z outputs = self.model( 2025-08-14T21:49:00.0160443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0160514Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0160753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0160835Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0161047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0161157Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0161409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0161503Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0161757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0161851Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0162139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:00.0162274Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:00.0162280Z 2025-08-14T21:49:00.0162395Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0162601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0162668Z return mod(**inputs) 2025-08-14T21:49:00.0162922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0162998Z outputs = self.model( 2025-08-14T21:49:00.0163250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0163323Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0163583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0163655Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0163882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0163961Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0164214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0164316Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0164565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0164659Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0164954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:00.0165060Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:00.0165064Z 2025-08-14T21:49:00.0165175Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0165448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0165547Z return mod(**inputs) 2025-08-14T21:49:00.0165813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0165882Z outputs = self.model( 2025-08-14T21:49:00.0166144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0166221Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0166486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0166572Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0166803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0166888Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0167160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0167290Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0167576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:00.0167655Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:00.0167659Z 2025-08-14T21:49:00.0167754Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0167947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0168008Z return mod(**inputs) 2025-08-14T21:49:00.0168248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0168312Z outputs = self.model( 2025-08-14T21:49:00.0168563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0168642Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0168884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0168953Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0169168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0169241Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0169485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0169584Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0169816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:00.0169968Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:00.0169973Z 2025-08-14T21:49:00.0170068Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0170265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0170325Z return mod(**inputs) 2025-08-14T21:49:00.0170560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0170630Z outputs = self.model( 2025-08-14T21:49:00.0170862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0170931Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0171173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0171241Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0171477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0171553Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0171791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0171897Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0172138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:00.0172221Z key_states = self.k_proj(current_states) 2025-08-14T21:49:00.0172225Z 2025-08-14T21:49:00.0172323Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0172515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0172584Z return mod(**inputs) 2025-08-14T21:49:00.0172831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0172898Z outputs = self.model( 2025-08-14T21:49:00.0173190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0173261Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0173505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0173574Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0173780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0173864Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0174101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0174202Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0174471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:00.0174556Z value_states = self.v_proj(current_states) 2025-08-14T21:49:00.0174560Z 2025-08-14T21:49:00.0174643Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0174719Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0174793Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0174873Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0174969Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0175155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0175225Z return mod(**inputs) 2025-08-14T21:49:00.0175467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0175540Z outputs = self.model( 2025-08-14T21:49:00.0175787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0175859Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0176113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0176182Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0176403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0176479Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0176724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0176832Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0177077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0177192Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0177478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:00.0177603Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:00.0177606Z 2025-08-14T21:49:00.0177712Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0177900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0177962Z return mod(**inputs) 2025-08-14T21:49:00.0178210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0178275Z outputs = self.model( 2025-08-14T21:49:00.0178524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0178596Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0178854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0178948Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0179168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0179244Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0179498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0179601Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0179852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0179946Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0180260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:00.0180374Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:00.0180378Z 2025-08-14T21:49:00.0180475Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0180674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0180736Z return mod(**inputs) 2025-08-14T21:49:00.0180980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0181055Z outputs = self.model( 2025-08-14T21:49:00.0181301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0181372Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0181630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0181702Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0181932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0182008Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0182252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0182360Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0182608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:00.0182685Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:00.0182695Z 2025-08-14T21:49:00.0182792Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0182989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0183078Z return mod(**inputs) 2025-08-14T21:49:00.0183322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0183388Z outputs = self.model( 2025-08-14T21:49:00.0183636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0183706Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0183957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0184027Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0184237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0184318Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0184559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:00.0184693Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:00.0184720Z 2025-08-14T21:49:00.0184824Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0185016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0185085Z return mod(**inputs) 2025-08-14T21:49:00.0185329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0185393Z outputs = self.model( 2025-08-14T21:49:00.0185642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0185712Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0185979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0186052Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0186266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0186348Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0186594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:00.0186707Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:00.0186917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:49:00.0186984Z return self.act(input) 2025-08-14T21:49:00.0186988Z 2025-08-14T21:49:00.0187091Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0187283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0187351Z return mod(**inputs) 2025-08-14T21:49:00.0187602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0187669Z outputs = self.model( 2025-08-14T21:49:00.0187916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0187993Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0188238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0188315Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0188527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0188602Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0188852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:49:00.0188953Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:00.0188957Z 2025-08-14T21:49:00.0189065Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0189258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0189322Z return mod(**inputs) 2025-08-14T21:49:00.0189577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0189643Z outputs = self.model( 2025-08-14T21:49:00.0189890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0189971Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0190216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0190297Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0190512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0190621Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0190877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-14T21:49:00.0190955Z hidden_states = residual + hidden_states 2025-08-14T21:49:00.0190958Z 2025-08-14T21:49:00.0191062Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0191257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0191320Z return mod(**inputs) 2025-08-14T21:49:00.0191568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0191634Z outputs = self.model( 2025-08-14T21:49:00.0191897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0191978Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0192221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0192298Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0192513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0192589Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0192843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0192940Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0193186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:00.0193343Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:00.0193348Z 2025-08-14T21:49:00.0193449Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0193663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0193727Z return mod(**inputs) 2025-08-14T21:49:00.0193972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0194044Z outputs = self.model( 2025-08-14T21:49:00.0194296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0194372Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0194606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0194694Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0194908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0194986Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0195221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0195321Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0195561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:00.0195644Z key_states = self.k_proj(current_states) 2025-08-14T21:49:00.0195647Z 2025-08-14T21:49:00.0195740Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0195927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0196001Z return mod(**inputs) 2025-08-14T21:49:00.0196255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0196349Z outputs = self.model( 2025-08-14T21:49:00.0196616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0196700Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0196949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0197018Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0197228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0197311Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0197551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0197668Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0197917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:00.0198001Z value_states = self.v_proj(current_states) 2025-08-14T21:49:00.0198004Z 2025-08-14T21:49:00.0198090Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0198167Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0198242Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0198321Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0198429Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0198620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0198682Z return mod(**inputs) 2025-08-14T21:49:00.0198921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0198998Z outputs = self.model( 2025-08-14T21:49:00.0199242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0199314Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0199566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0199647Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0199861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0199935Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0200170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0200269Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0200505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0200620Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0200893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:00.0201015Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:00.0201019Z 2025-08-14T21:49:00.0201121Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0201311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0201373Z return mod(**inputs) 2025-08-14T21:49:00.0201617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0201682Z outputs = self.model( 2025-08-14T21:49:00.0201925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0201995Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0202273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0202353Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0202561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0202642Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0202883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0202975Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0203223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0203316Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0203611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:00.0203736Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:00.0203739Z 2025-08-14T21:49:00.0203837Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0204033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0204096Z return mod(**inputs) 2025-08-14T21:49:00.0204335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0204406Z outputs = self.model( 2025-08-14T21:49:00.0204645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0204722Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0204964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0205041Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0205370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0205459Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0205726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0205837Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0206104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:00.0206208Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:00.0206213Z 2025-08-14T21:49:00.0206320Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0206552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0206625Z return mod(**inputs) 2025-08-14T21:49:00.0206863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0206928Z outputs = self.model( 2025-08-14T21:49:00.0207173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0207243Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0207486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0207553Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0207756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0207835Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0208075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0208254Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0208492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:00.0208631Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:00.0208635Z 2025-08-14T21:49:00.0208737Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0208922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0208985Z return mod(**inputs) 2025-08-14T21:49:00.0209226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0209291Z outputs = self.model( 2025-08-14T21:49:00.0209552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0209624Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0209864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0209940Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0210145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0210224Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0210461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0210560Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0210802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:00.0210882Z key_states = self.k_proj(current_states) 2025-08-14T21:49:00.0210885Z 2025-08-14T21:49:00.0210985Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0211187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0211249Z return mod(**inputs) 2025-08-14T21:49:00.0211501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0211566Z outputs = self.model( 2025-08-14T21:49:00.0211807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0211887Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0212130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0212208Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0212450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0212531Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0212793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0212893Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0213126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:00.0213211Z value_states = self.v_proj(current_states) 2025-08-14T21:49:00.0213214Z 2025-08-14T21:49:00.0213289Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0213368Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0213441Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0213512Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0213618Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0213803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0213899Z return mod(**inputs) 2025-08-14T21:49:00.0214148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0214211Z outputs = self.model( 2025-08-14T21:49:00.0214457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0214527Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0214770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0214849Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0215062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0215160Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0215412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0215515Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0215763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0215857Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0216132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:00.0216263Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:00.0216267Z 2025-08-14T21:49:00.0216376Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0216571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0216635Z return mod(**inputs) 2025-08-14T21:49:00.0216875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0216949Z outputs = self.model( 2025-08-14T21:49:00.0217189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0217258Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0217503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0217573Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0217788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0217863Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0218103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0218237Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0218483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0218586Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0218866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:00.0218970Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:00.0218974Z 2025-08-14T21:49:00.0219080Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0219271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0219335Z return mod(**inputs) 2025-08-14T21:49:00.0219591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0219681Z outputs = self.model( 2025-08-14T21:49:00.0219952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0220027Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0220275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0220354Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0220568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0220650Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0220894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0221015Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0221267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:00.0221348Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:00.0221352Z 2025-08-14T21:49:00.0221448Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0221644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0221708Z return mod(**inputs) 2025-08-14T21:49:00.0221955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0222020Z outputs = self.model( 2025-08-14T21:49:00.0222262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0222342Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0222587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0222658Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0222879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0222952Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0223203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:00.0223319Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:00.0223323Z 2025-08-14T21:49:00.0223424Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0223627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0223692Z return mod(**inputs) 2025-08-14T21:49:00.0223961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0224046Z outputs = self.model( 2025-08-14T21:49:00.0224294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0224372Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0224616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0224686Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0224925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0224999Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0225246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:00.0225362Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:00.0225566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:49:00.0225687Z return self.act(input) 2025-08-14T21:49:00.0225691Z 2025-08-14T21:49:00.0225790Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0225987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0226051Z return mod(**inputs) 2025-08-14T21:49:00.0226291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0226362Z outputs = self.model( 2025-08-14T21:49:00.0226603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0226674Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0226936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0227010Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0227231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0227305Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0227546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:49:00.0227633Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:00.0227636Z 2025-08-14T21:49:00.0227732Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0227931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0227995Z return mod(**inputs) 2025-08-14T21:49:00.0228239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0228314Z outputs = self.model( 2025-08-14T21:49:00.0228559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0228629Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0228880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0228949Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0229168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0229242Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0229485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0229586Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0229830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:00.0229993Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:00.0230005Z 2025-08-14T21:49:00.0230102Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0230293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0230362Z return mod(**inputs) 2025-08-14T21:49:00.0230602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0230669Z outputs = self.model( 2025-08-14T21:49:00.0230919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0230991Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0231242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0231315Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0231566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0231654Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0231903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0232001Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0232256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:00.0232338Z key_states = self.k_proj(current_states) 2025-08-14T21:49:00.0232342Z 2025-08-14T21:49:00.0232447Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0232659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0232729Z return mod(**inputs) 2025-08-14T21:49:00.0232990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0233059Z outputs = self.model( 2025-08-14T21:49:00.0233316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0233388Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0233634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0233713Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0233929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0234006Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0234267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0234365Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0234623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:00.0234708Z value_states = self.v_proj(current_states) 2025-08-14T21:49:00.0234712Z 2025-08-14T21:49:00.0234792Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0234879Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0234956Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0235031Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0235140Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0235335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0235407Z return mod(**inputs) 2025-08-14T21:49:00.0235659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0235754Z outputs = self.model( 2025-08-14T21:49:00.0236018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0236092Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0236341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0236421Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0236648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0236735Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0237000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0237102Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0237378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0237516Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0238022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:00.0238157Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:00.0238161Z 2025-08-14T21:49:00.0238261Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0238468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0238533Z return mod(**inputs) 2025-08-14T21:49:00.0238787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0238864Z outputs = self.model( 2025-08-14T21:49:00.0239168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0239253Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0239501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0239573Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0239798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0239873Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0240128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0240222Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0240472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0240575Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0240866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:00.0240973Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:00.0240984Z 2025-08-14T21:49:00.0241085Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0241280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0241352Z return mod(**inputs) 2025-08-14T21:49:00.0241601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0241669Z outputs = self.model( 2025-08-14T21:49:00.0241926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0242031Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0242292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0242365Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0242582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0242667Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0242914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0243010Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0243266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:00.0243346Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:00.0243350Z 2025-08-14T21:49:00.0243458Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0243654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0243792Z return mod(**inputs) 2025-08-14T21:49:00.0244048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0244116Z outputs = self.model( 2025-08-14T21:49:00.0244363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0244443Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0244691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0244768Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0244986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0245078Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0245419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 424, in forward 2025-08-14T21:49:00.0245512Z hidden_states = residual + hidden_states 2025-08-14T21:49:00.0245516Z 2025-08-14T21:49:00.0245633Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0245858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0245929Z return mod(**inputs) 2025-08-14T21:49:00.0246212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0246285Z outputs = self.model( 2025-08-14T21:49:00.0246562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0246642Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0246894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0246976Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0247196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0247274Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0247529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0247635Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0247890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:00.0248039Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:00.0248043Z 2025-08-14T21:49:00.0248174Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0248378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0248446Z return mod(**inputs) 2025-08-14T21:49:00.0248700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0248775Z outputs = self.model( 2025-08-14T21:49:00.0249026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0249107Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0249358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0249429Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0249653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0249733Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0250000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0250130Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0250382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:00.0250466Z key_states = self.k_proj(current_states) 2025-08-14T21:49:00.0250469Z 2025-08-14T21:49:00.0250569Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0250769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0250847Z return mod(**inputs) 2025-08-14T21:49:00.0251113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0251224Z outputs = self.model( 2025-08-14T21:49:00.0251490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0251570Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0251838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0251913Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0252142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0252230Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0252491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0252617Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0252857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:00.0252941Z value_states = self.v_proj(current_states) 2025-08-14T21:49:00.0252946Z 2025-08-14T21:49:00.0253034Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0253110Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0253192Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0253266Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0253363Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0253562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0253626Z return mod(**inputs) 2025-08-14T21:49:00.0253870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0253943Z outputs = self.model( 2025-08-14T21:49:00.0254185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0254287Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0254533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0254603Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0254820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0254896Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0255136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0255244Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0255488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0255586Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0255869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:00.0256032Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:00.0256036Z 2025-08-14T21:49:00.0256143Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0256333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0256402Z return mod(**inputs) 2025-08-14T21:49:00.0256646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0256711Z outputs = self.model( 2025-08-14T21:49:00.0256962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0257033Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0257293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0257374Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0257589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0257670Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0257915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0258017Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0258266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0258358Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0258646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:00.0258759Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:00.0258764Z 2025-08-14T21:49:00.0258866Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0259071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0259137Z return mod(**inputs) 2025-08-14T21:49:00.0259392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0259470Z outputs = self.model( 2025-08-14T21:49:00.0259721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0259800Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0260053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0260149Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0260374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0260454Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0260700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0260813Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0261060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:00.0261149Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:00.0261153Z 2025-08-14T21:49:00.0261252Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0261449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0261522Z return mod(**inputs) 2025-08-14T21:49:00.0261778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0261870Z outputs = self.model( 2025-08-14T21:49:00.0262184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0262263Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0262543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0262619Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0262851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0262941Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0263202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:00.0263353Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:00.0263358Z 2025-08-14T21:49:00.0263466Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0263676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0263754Z return mod(**inputs) 2025-08-14T21:49:00.0264077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0264154Z outputs = self.model( 2025-08-14T21:49:00.0264407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0264483Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0264754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0264829Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0265051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0265141Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0265407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:00.0265541Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:00.0265764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:49:00.0265840Z return self.act(input) 2025-08-14T21:49:00.0265844Z 2025-08-14T21:49:00.0265960Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0266182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0266253Z return mod(**inputs) 2025-08-14T21:49:00.0266542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0266633Z outputs = self.model( 2025-08-14T21:49:00.0266892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0266964Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0267213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0267292Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0267508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0267592Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0267846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:49:00.0267925Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:00.0267931Z 2025-08-14T21:49:00.0268036Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0268273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0268344Z return mod(**inputs) 2025-08-14T21:49:00.0268615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0268686Z outputs = self.model( 2025-08-14T21:49:00.0268955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0269031Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0269292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0269375Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0269627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0269710Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0269985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0270088Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0270359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:00.0270516Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:00.0270519Z 2025-08-14T21:49:00.0270624Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0270842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0270913Z return mod(**inputs) 2025-08-14T21:49:00.0271186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0271257Z outputs = self.model( 2025-08-14T21:49:00.0271523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0271604Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0271867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0271940Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0272175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0272256Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0272526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0272628Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0272916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:00.0273010Z key_states = self.k_proj(current_states) 2025-08-14T21:49:00.0273014Z 2025-08-14T21:49:00.0273117Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0273331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0273398Z return mod(**inputs) 2025-08-14T21:49:00.0273662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0273740Z outputs = self.model( 2025-08-14T21:49:00.0274003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0274079Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0274350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0274425Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0274703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0274785Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0275044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0275152Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0275414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:00.0275511Z value_states = self.v_proj(current_states) 2025-08-14T21:49:00.0275515Z 2025-08-14T21:49:00.0275598Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0275682Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0275790Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0275871Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0275979Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0276207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0276272Z return mod(**inputs) 2025-08-14T21:49:00.0276537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0276605Z outputs = self.model( 2025-08-14T21:49:00.0276862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0276955Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0277203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0277273Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0277497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0277575Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0277826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0277919Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0278162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0278260Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0278540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:00.0278666Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:00.0278676Z 2025-08-14T21:49:00.0278793Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0278982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0279054Z return mod(**inputs) 2025-08-14T21:49:00.0279295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0279361Z outputs = self.model( 2025-08-14T21:49:00.0279608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0279679Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0279932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0280003Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0280220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0280308Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0280571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0280687Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0280944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0281038Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0281333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:00.0281439Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:00.0281443Z 2025-08-14T21:49:00.0281543Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0281763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0281835Z return mod(**inputs) 2025-08-14T21:49:00.0282108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0282179Z outputs = self.model( 2025-08-14T21:49:00.0282442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0282529Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0282791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0282868Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0283109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0283190Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0283462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0283566Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0283831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:00.0283923Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:00.0283926Z 2025-08-14T21:49:00.0284033Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0284249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0284317Z return mod(**inputs) 2025-08-14T21:49:00.0284578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0284655Z outputs = self.model( 2025-08-14T21:49:00.0284918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0285016Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0285379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0285461Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0285700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0285780Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0286042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0286164Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0286423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:00.0286588Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:00.0286594Z 2025-08-14T21:49:00.0286699Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0286959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0287034Z return mod(**inputs) 2025-08-14T21:49:00.0287280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0287346Z outputs = self.model( 2025-08-14T21:49:00.0287597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0287667Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0287918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0287988Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0288222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0288311Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0288553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0288655Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0288906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:00.0288987Z key_states = self.k_proj(current_states) 2025-08-14T21:49:00.0288991Z 2025-08-14T21:49:00.0289098Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0289292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0289359Z return mod(**inputs) 2025-08-14T21:49:00.0289610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0289681Z outputs = self.model( 2025-08-14T21:49:00.0289934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0290007Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0290251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0290332Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0290546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0290623Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0290874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0290987Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0291251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:00.0291334Z value_states = self.v_proj(current_states) 2025-08-14T21:49:00.0291337Z 2025-08-14T21:49:00.0291414Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0291495Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0291569Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0291641Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0291747Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0291935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0292006Z return mod(**inputs) 2025-08-14T21:49:00.0292250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0292318Z outputs = self.model( 2025-08-14T21:49:00.0292580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0292671Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0292946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0293027Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0293247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0293331Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0293579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0293687Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0293944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0294066Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0294347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:00.0294470Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:00.0294473Z 2025-08-14T21:49:00.0294568Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0294762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0294824Z return mod(**inputs) 2025-08-14T21:49:00.0295065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0295130Z outputs = self.model( 2025-08-14T21:49:00.0295362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0295442Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0295677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0295745Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0295957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0296028Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0296271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0296369Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0296605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0296700Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0296970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:00.0297089Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:00.0297101Z 2025-08-14T21:49:00.0297197Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0297382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0297452Z return mod(**inputs) 2025-08-14T21:49:00.0297688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0297752Z outputs = self.model( 2025-08-14T21:49:00.0297993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0298064Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0298306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0298376Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0298613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0298698Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0298935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0299034Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0299277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:00.0299353Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:00.0299356Z 2025-08-14T21:49:00.0299458Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0299660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0299725Z return mod(**inputs) 2025-08-14T21:49:00.0299971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0300036Z outputs = self.model( 2025-08-14T21:49:00.0300275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0300343Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0300575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0300650Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0300853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0300925Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0301166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 441, in forward 2025-08-14T21:49:00.0301240Z hidden_states = residual + hidden_states 2025-08-14T21:49:00.0301245Z 2025-08-14T21:49:00.0301345Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0301529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0301590Z return mod(**inputs) 2025-08-14T21:49:00.0301829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0301892Z outputs = self.model( 2025-08-14T21:49:00.0302126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0302203Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0302444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0302576Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0302790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0302865Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0303113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:00.0303228Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:00.0303232Z 2025-08-14T21:49:00.0303336Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0303527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0303590Z return mod(**inputs) 2025-08-14T21:49:00.0303851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0303915Z outputs = self.model( 2025-08-14T21:49:00.0304150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0304260Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0304499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0304577Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0304788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0304863Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0305116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:00.0305233Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:00.0305464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:49:00.0305538Z return self.act(input) 2025-08-14T21:49:00.0305541Z 2025-08-14T21:49:00.0305643Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0305844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0305906Z return mod(**inputs) 2025-08-14T21:49:00.0306148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0306224Z outputs = self.model( 2025-08-14T21:49:00.0306474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0306557Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0306809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0306891Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0307110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0307189Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0307428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:49:00.0307513Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:00.0307517Z 2025-08-14T21:49:00.0307615Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0307809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0307873Z return mod(**inputs) 2025-08-14T21:49:00.0308112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0308186Z outputs = self.model( 2025-08-14T21:49:00.0308430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0308532Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0308773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0308843Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0309060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0309132Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0309374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0309476Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0309718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:00.0309869Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:00.0309872Z 2025-08-14T21:49:00.0309987Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0310458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0310533Z return mod(**inputs) 2025-08-14T21:49:00.0310775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0310848Z outputs = self.model( 2025-08-14T21:49:00.0311088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0311160Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0311412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0311483Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0311713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0311800Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0312046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0312150Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0312395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:00.0312473Z key_states = self.k_proj(current_states) 2025-08-14T21:49:00.0312477Z 2025-08-14T21:49:00.0312584Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0312778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0312852Z return mod(**inputs) 2025-08-14T21:49:00.0313100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0313167Z outputs = self.model( 2025-08-14T21:49:00.0313420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0313491Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0313733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0313811Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0314024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0314106Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0314349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0314466Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0314715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:00.0314800Z value_states = self.v_proj(current_states) 2025-08-14T21:49:00.0314804Z 2025-08-14T21:49:00.0314880Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0314964Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0315038Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0315119Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0315214Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0315404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0315473Z return mod(**inputs) 2025-08-14T21:49:00.0315716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0315784Z outputs = self.model( 2025-08-14T21:49:00.0316031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0316142Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0316396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0316466Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0316680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0316763Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0317007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0317106Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0317363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0317459Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0317748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:00.0317873Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:00.0317876Z 2025-08-14T21:49:00.0317974Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0318172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0318236Z return mod(**inputs) 2025-08-14T21:49:00.0318481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0318547Z outputs = self.model( 2025-08-14T21:49:00.0318787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0318867Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0319110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0319181Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0319398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0319471Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0319720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0319812Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0320050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0320159Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0320449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:00.0320560Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:00.0320564Z 2025-08-14T21:49:00.0320658Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0320846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0320917Z return mod(**inputs) 2025-08-14T21:49:00.0321161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0321226Z outputs = self.model( 2025-08-14T21:49:00.0321476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0321547Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0321797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0321888Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0322119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0322207Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0322458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 415, in forward 2025-08-14T21:49:00.0322570Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:49:00.0322809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:00.0322889Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:00.0322893Z 2025-08-14T21:49:00.0323007Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0323213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0323281Z return mod(**inputs) 2025-08-14T21:49:00.0323538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0323606Z outputs = self.model( 2025-08-14T21:49:00.0323872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0323944Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0324185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0324261Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0324473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0324555Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0324806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0324912Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0325160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 225, in forward 2025-08-14T21:49:00.0325378Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:49:00.0325385Z 2025-08-14T21:49:00.0325488Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0325701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0325772Z return mod(**inputs) 2025-08-14T21:49:00.0326050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0326122Z outputs = self.model( 2025-08-14T21:49:00.0326409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0326489Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0326734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0326823Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0327029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0327102Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0327345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0327443Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0327676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 244, in forward 2025-08-14T21:49:00.0327764Z key_states = self.k_proj(current_states) 2025-08-14T21:49:00.0327768Z 2025-08-14T21:49:00.0327863Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0328096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0328160Z return mod(**inputs) 2025-08-14T21:49:00.0328402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0328473Z outputs = self.model( 2025-08-14T21:49:00.0328711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0328787Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0329025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0329094Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0329323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0329397Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0329632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0329740Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0329974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 245, in forward 2025-08-14T21:49:00.0330059Z value_states = self.v_proj(current_states) 2025-08-14T21:49:00.0330063Z 2025-08-14T21:49:00.0330137Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0330212Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0330292Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0330363Z cudagraph partition due to non gpu ops 2025-08-14T21:49:00.0330459Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0330659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0330723Z return mod(**inputs) 2025-08-14T21:49:00.0330966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0331030Z outputs = self.model( 2025-08-14T21:49:00.0331264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0331342Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0331577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0331645Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0331857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0331951Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0332194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0332293Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0332525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0332624Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0332893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:49:00.0333019Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:49:00.0333022Z 2025-08-14T21:49:00.0333117Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0333303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0333376Z return mod(**inputs) 2025-08-14T21:49:00.0336427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0336526Z outputs = self.model( 2025-08-14T21:49:00.0336784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0336857Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0337095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0337174Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0337383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0337465Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0337964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0338076Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0338350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 263, in forward 2025-08-14T21:49:00.0338443Z attn_output, attn_weights = attention_interface( 2025-08-14T21:49:00.0338719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:49:00.0338829Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:00.0338834Z 2025-08-14T21:49:00.0338931Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0339124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0339186Z return mod(**inputs) 2025-08-14T21:49:00.0339423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0339495Z outputs = self.model( 2025-08-14T21:49:00.0339732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0339803Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0340047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0340117Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0340332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0340408Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0340644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 432, in forward 2025-08-14T21:49:00.0340752Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:49:00.0341015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 277, in forward 2025-08-14T21:49:00.0341098Z attn_output = self.out_proj(attn_output) 2025-08-14T21:49:00.0341107Z 2025-08-14T21:49:00.0341202Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0341389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0341458Z return mod(**inputs) 2025-08-14T21:49:00.0341697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0341761Z outputs = self.model( 2025-08-14T21:49:00.0342004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0342072Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0342312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0342409Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0342693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0342776Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0343011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:00.0343123Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:00.0343135Z 2025-08-14T21:49:00.0343230Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0343416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0343488Z return mod(**inputs) 2025-08-14T21:49:00.0343743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0343810Z outputs = self.model( 2025-08-14T21:49:00.0344061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0344131Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0344380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0344448Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0344657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0344737Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0344974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 446, in forward 2025-08-14T21:49:00.0345084Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:49:00.0345294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:49:00.0345361Z return self.act(input) 2025-08-14T21:49:00.0345365Z 2025-08-14T21:49:00.0345469Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0345656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0345719Z return mod(**inputs) 2025-08-14T21:49:00.0345967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0346032Z outputs = self.model( 2025-08-14T21:49:00.0346272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0346349Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0346593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0346690Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0346903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0346976Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0347224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 448, in forward 2025-08-14T21:49:00.0347300Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:49:00.0347303Z 2025-08-14T21:49:00.0347404Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0347592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0347653Z return mod(**inputs) 2025-08-14T21:49:00.0347899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1438, in forward 2025-08-14T21:49:00.0347965Z outputs = self.model( 2025-08-14T21:49:00.0348207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1266, in forward 2025-08-14T21:49:00.0348331Z decoder_outputs = self.decoder( 2025-08-14T21:49:00.0348578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1109, in forward 2025-08-14T21:49:00.0348656Z layer_outputs = decoder_layer( 2025-08-14T21:49:00.0348866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:00.0348941Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:00.0349188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 450, in forward 2025-08-14T21:49:00.0349265Z hidden_states = residual + hidden_states 2025-08-14T21:49:00.0349269Z 2025-08-14T21:49:00.0349373Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0349580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0349647Z return mod(**inputs) 2025-08-14T21:49:00.0349909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1456, in forward 2025-08-14T21:49:00.0350021Z lm_logits = self.lm_head(outputs[0]) + self.final_logits_bias 2025-08-14T21:49:00.0350024Z 2025-08-14T21:49:00.0350118Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:00.0350309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:00.0350370Z return mod(**inputs) 2025-08-14T21:49:00.0350612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mbart/modeling_mbart.py", line 1461, in forward 2025-08-14T21:49:00.0350767Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:49:00.0350773Z 2025-08-14T21:49:12.7375340Z Compilation time (from dynamo_timed): 26.81445937 2025-08-14T21:49:12.7465004Z pass 2025-08-14T21:49:12.7465454Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:49:12.7466277Z TIMING: _recursive_pre_grad_passes:0.01384 _recursive_joint_graph_passes:1.11915 _recursive_post_grad_passes:0.18133 async_compile.wait:0.79544 code_gen:11.16363 inductor_compile:14.1336 backend_compile:21.16161 gc:0.00018 entire_frame_compile:26.81446 total_wall_time:26.81446 2025-08-14T21:49:12.7467185Z STATS: call_* op count: 986 | FakeTensorMode.__torch_dispatch__:33703 | FakeTensor.__torch_dispatch__:12062 | ProxyTorchDispatchMode.__torch_dispatch__:12456 2025-08-14T21:49:12.7467688Z Dynamo produced 1 graphs covering 986 ops with 0 graph breaks (0 unique) 2025-08-14T21:49:18.7532351Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:49:18.7533663Z from pkg_resources import resource_filename 2025-08-14T21:49:19.3246972Z 2025-08-14T21:49:21.8297508Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:49:21.8297935Z loading model: 0it [00:02, ?it/s] 2025-08-14T21:49:21.8319328Z cpu eval MT5ForConditionalGeneration 2025-08-14T21:49:22.4546350Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:49:22.7255048Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:49:22.9926214Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:49:35.2554904Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.2555430Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2556009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2556790Z return mod(**inputs) 2025-08-14T21:49:35.2557288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.2557706Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.2558102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2558486Z layer_outputs = layer_module( 2025-08-14T21:49:35.2558846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2559273Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2559657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2560118Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2560511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2560916Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2561313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 421, in forward 2025-08-14T21:49:35.2561778Z position_bias = position_bias + causal_mask 2025-08-14T21:49:35.2561965Z 2025-08-14T21:49:35.2562077Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2562470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2562821Z return mod(**inputs) 2025-08-14T21:49:35.2563200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.2563605Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.2564010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2564408Z layer_outputs = layer_module( 2025-08-14T21:49:35.2564793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2565181Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2565752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2566180Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2566655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:49:35.2567075Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.2567485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.2567965Z return self.weight * hidden_states 2025-08-14T21:49:35.2568104Z 2025-08-14T21:49:35.2568222Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2568610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2568981Z return mod(**inputs) 2025-08-14T21:49:35.2569364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.2569774Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.2570173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2570584Z layer_outputs = layer_module( 2025-08-14T21:49:35.2570972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2571347Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2571755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2572187Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2572614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2573032Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2573434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.2573828Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.2574125Z 2025-08-14T21:49:35.2574236Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2574585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2574925Z return mod(**inputs) 2025-08-14T21:49:35.2575323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.2575727Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.2576118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2576490Z layer_outputs = layer_module( 2025-08-14T21:49:35.2576834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2577196Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2577562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2577932Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2578308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2578677Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2579048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.2579417Z key_states = self.k(current_states) 2025-08-14T21:49:35.2579543Z 2025-08-14T21:49:35.2579645Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2579993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2580308Z return mod(**inputs) 2025-08-14T21:49:35.2580650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.2581010Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.2581381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2581744Z layer_outputs = layer_module( 2025-08-14T21:49:35.2582070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2582443Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2582813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2583192Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2583574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2583960Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2584397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.2584832Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.2585013Z 2025-08-14T21:49:35.2585116Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2585477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2585805Z return mod(**inputs) 2025-08-14T21:49:35.2586204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.2586602Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.2586966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2587356Z layer_outputs = layer_module( 2025-08-14T21:49:35.2587688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2588047Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2588427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2588876Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2589255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2589643Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2590035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.2590482Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.2590697Z 2025-08-14T21:49:35.2590799Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2591159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2591481Z return mod(**inputs) 2025-08-14T21:49:35.2591824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.2592203Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.2592575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2592938Z layer_outputs = layer_module( 2025-08-14T21:49:35.2593282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2593635Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2594003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2594368Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2594740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2595144Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2595543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.2595941Z value_states = self.v(current_states) 2025-08-14T21:49:35.2596109Z 2025-08-14T21:49:35.2596221Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2596603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2596921Z return mod(**inputs) 2025-08-14T21:49:35.2597261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.2597634Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.2597995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2598357Z layer_outputs = layer_module( 2025-08-14T21:49:35.2598700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2599056Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2599440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2599843Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2600264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2600649Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2601025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.2601457Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.2601635Z 2025-08-14T21:49:35.2601745Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2602124Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2602455Z return mod(**inputs) 2025-08-14T21:49:35.2602823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.2603239Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.2603619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2604016Z layer_outputs = layer_module( 2025-08-14T21:49:35.2604374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2604756Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2605142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2605638Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2606040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2606455Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2606843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.2607275Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.2607448Z 2025-08-14T21:49:35.2607568Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2607941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2608287Z return mod(**inputs) 2025-08-14T21:49:35.2608656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.2609055Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.2609431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2609827Z layer_outputs = layer_module( 2025-08-14T21:49:35.2610187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2610590Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2610990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2611393Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2611791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2612195Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2612586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.2613014Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.2613182Z 2025-08-14T21:49:35.2613296Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2613663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2614008Z return mod(**inputs) 2025-08-14T21:49:35.2614381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.2614829Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.2615224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2615631Z layer_outputs = layer_module( 2025-08-14T21:49:35.2615993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2616365Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2616738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2617119Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2617503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2617891Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2618301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.2618711Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.2618847Z 2025-08-14T21:49:35.2618960Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2619358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2619712Z return mod(**inputs) 2025-08-14T21:49:35.2620082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.2620477Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.2620867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2621274Z layer_outputs = layer_module( 2025-08-14T21:49:35.2621630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2622014Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2622387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.2622763Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.2623132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.2623515Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.2623892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.2624266Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.2624406Z 2025-08-14T21:49:35.2624509Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2624894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2625220Z return mod(**inputs) 2025-08-14T21:49:35.2625565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2625943Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2626308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2626679Z layer_outputs = layer_module( 2025-08-14T21:49:35.2627014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2627374Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2627744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2628123Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2628505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2628932Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2629308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.2629677Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.2629816Z 2025-08-14T21:49:35.2629919Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2630278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2630592Z return mod(**inputs) 2025-08-14T21:49:35.2630943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2631319Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2631726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2632115Z layer_outputs = layer_module( 2025-08-14T21:49:35.2632478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2632856Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2633246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2633660Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2634047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2634453Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2634839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.2635238Z key_states = self.k(current_states) 2025-08-14T21:49:35.2635382Z 2025-08-14T21:49:35.2635490Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2635869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2636201Z return mod(**inputs) 2025-08-14T21:49:35.2636565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2636962Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2637340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2638134Z layer_outputs = layer_module( 2025-08-14T21:49:35.2638507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2638890Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2639355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2639766Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2640201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2640606Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2640990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.2641434Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.2641623Z 2025-08-14T21:49:35.2641741Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2642114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2643683Z return mod(**inputs) 2025-08-14T21:49:35.2644148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2644611Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2645317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2646060Z layer_outputs = layer_module( 2025-08-14T21:49:35.2646667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2647181Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2647706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2648304Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2648880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2649411Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2649834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.2650526Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.2650743Z 2025-08-14T21:49:35.2650855Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2651363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2651780Z return mod(**inputs) 2025-08-14T21:49:35.2652301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2652662Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2653030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2653394Z layer_outputs = layer_module( 2025-08-14T21:49:35.2653741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2654167Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2654718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2655241Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2655616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2655983Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2656436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.2656874Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.2657078Z 2025-08-14T21:49:35.2657182Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2657572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2657930Z return mod(**inputs) 2025-08-14T21:49:35.2658433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2658803Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2659172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2659526Z layer_outputs = layer_module( 2025-08-14T21:49:35.2659923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2660266Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2660619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2660977Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2661328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2679439Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2680131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.2680633Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.2680873Z 2025-08-14T21:49:35.2680991Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2681402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2681792Z return mod(**inputs) 2025-08-14T21:49:35.2682198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2682614Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2683069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2683468Z layer_outputs = layer_module( 2025-08-14T21:49:35.2683848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2684239Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2684640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2685051Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2685546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2685988Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2686413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.2686812Z value_states = self.v(current_states) 2025-08-14T21:49:35.2686971Z 2025-08-14T21:49:35.2687089Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2687447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2687772Z return mod(**inputs) 2025-08-14T21:49:35.2688133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2688525Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2688883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2689252Z layer_outputs = layer_module( 2025-08-14T21:49:35.2689602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2689941Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2690332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2690699Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2691059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2691415Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2691774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.2692161Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.2692315Z 2025-08-14T21:49:35.2692414Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2692758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2693068Z return mod(**inputs) 2025-08-14T21:49:35.2693403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2693753Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2694177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2694548Z layer_outputs = layer_module( 2025-08-14T21:49:35.2694873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2695224Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2695585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2695952Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2696306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2696674Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2697064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.2697474Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.2697637Z 2025-08-14T21:49:35.2697742Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2698111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2698425Z return mod(**inputs) 2025-08-14T21:49:35.2698759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2699126Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2699485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2699845Z layer_outputs = layer_module( 2025-08-14T21:49:35.2700174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2700524Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2700891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2701249Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2701609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2701976Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2702334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.2702716Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.2702878Z 2025-08-14T21:49:35.2702978Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2703327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2703664Z return mod(**inputs) 2025-08-14T21:49:35.2704005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2704375Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2704737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2705094Z layer_outputs = layer_module( 2025-08-14T21:49:35.2705431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2705786Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2706155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2706519Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2706888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2707288Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2707676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.2708047Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.2708180Z 2025-08-14T21:49:35.2708260Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.2708493Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2708836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2709152Z return mod(**inputs) 2025-08-14T21:49:35.2709492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2709851Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2710219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2710581Z layer_outputs = layer_module( 2025-08-14T21:49:35.2710915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2711256Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2711617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.2711995Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.2712375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:49:35.2712752Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.2713135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.2713492Z return self.weight * hidden_states 2025-08-14T21:49:35.2713616Z 2025-08-14T21:49:35.2713713Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2714055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2714364Z return mod(**inputs) 2025-08-14T21:49:35.2714796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2715222Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2715581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2715941Z layer_outputs = layer_module( 2025-08-14T21:49:35.2716265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2716612Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2716975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.2717372Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.2717739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.2718137Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.2718526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:49:35.2718906Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:49:35.2719049Z 2025-08-14T21:49:35.2719146Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2719488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2719799Z return mod(**inputs) 2025-08-14T21:49:35.2720122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2720480Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2720866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2721224Z layer_outputs = layer_module( 2025-08-14T21:49:35.2721552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2721907Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2722274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.2722652Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.2723020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.2723439Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.2723843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:49:35.2724222Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:49:35.2724358Z 2025-08-14T21:49:35.2724456Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2724797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2725115Z return mod(**inputs) 2025-08-14T21:49:35.2725508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2725883Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2726262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2726666Z layer_outputs = layer_module( 2025-08-14T21:49:35.2727014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2727390Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2727755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.2728131Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.2728556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.2728959Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.2729357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:49:35.2729723Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:49:35.2729866Z 2025-08-14T21:49:35.2729965Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2730313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2730649Z return mod(**inputs) 2025-08-14T21:49:35.2731002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2731376Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2731741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2732106Z layer_outputs = layer_module( 2025-08-14T21:49:35.2732452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2732810Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2733176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.2733561Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.2733949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.2734367Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.2734777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:49:35.2735146Z hidden_states = self.wo(hidden_states) 2025-08-14T21:49:35.2735274Z 2025-08-14T21:49:35.2735360Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.2735589Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2735926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2736240Z return mod(**inputs) 2025-08-14T21:49:35.2736576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2736933Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2737303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2737885Z layer_outputs = layer_module( 2025-08-14T21:49:35.2738233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2738587Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2738951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2739315Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2739682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:49:35.2740079Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.2740460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.2740834Z return self.weight * hidden_states 2025-08-14T21:49:35.2740970Z 2025-08-14T21:49:35.2741071Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2741422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2741737Z return mod(**inputs) 2025-08-14T21:49:35.2742077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2742441Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2742796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2743152Z layer_outputs = layer_module( 2025-08-14T21:49:35.2743487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2743841Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2744294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2744673Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2745043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2745433Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2745790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.2746158Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.2746295Z 2025-08-14T21:49:35.2746408Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2746761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2747075Z return mod(**inputs) 2025-08-14T21:49:35.2747413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2747802Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2748173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2748532Z layer_outputs = layer_module( 2025-08-14T21:49:35.2748861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2749202Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2749562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2749930Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2750297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2750683Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2751051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.2751415Z key_states = self.k(current_states) 2025-08-14T21:49:35.2751541Z 2025-08-14T21:49:35.2751651Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2751991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2752365Z return mod(**inputs) 2025-08-14T21:49:35.2752691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2753036Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2753380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2753730Z layer_outputs = layer_module( 2025-08-14T21:49:35.2754054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2754394Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2754757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2755117Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2755469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2755829Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2756182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.2756588Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.2756765Z 2025-08-14T21:49:35.2756862Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2757212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2757531Z return mod(**inputs) 2025-08-14T21:49:35.2757856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2758195Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2758527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2758872Z layer_outputs = layer_module( 2025-08-14T21:49:35.2759180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2759513Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2759858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2760211Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2760562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2760975Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2761354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.2761806Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.2762002Z 2025-08-14T21:49:35.2762099Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2762441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2762756Z return mod(**inputs) 2025-08-14T21:49:35.2763087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2763458Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2763831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2764196Z layer_outputs = layer_module( 2025-08-14T21:49:35.2764521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2764876Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2765250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2765735Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2766135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2766537Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2766916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.2767350Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.2767564Z 2025-08-14T21:49:35.2767666Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2768024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2768345Z return mod(**inputs) 2025-08-14T21:49:35.2768680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2769052Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2769411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2769764Z layer_outputs = layer_module( 2025-08-14T21:49:35.2770099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2770447Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2770834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2771192Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2771553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2771918Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2772271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.2772700Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.2772900Z 2025-08-14T21:49:35.2772999Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2773343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2773648Z return mod(**inputs) 2025-08-14T21:49:35.2773982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2774364Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2774762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2775123Z layer_outputs = layer_module( 2025-08-14T21:49:35.2775454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2775804Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2776169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2776526Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2776881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2777256Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2777604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.2777966Z value_states = self.v(current_states) 2025-08-14T21:49:35.2778094Z 2025-08-14T21:49:35.2778204Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2778539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2778843Z return mod(**inputs) 2025-08-14T21:49:35.2779169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2779520Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2779857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2780209Z layer_outputs = layer_module( 2025-08-14T21:49:35.2780537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2780879Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2781225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2781582Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2781933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2782286Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2782637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.2783018Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.2783171Z 2025-08-14T21:49:35.2783275Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2783605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2783943Z return mod(**inputs) 2025-08-14T21:49:35.2784274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2784623Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2784973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2785329Z layer_outputs = layer_module( 2025-08-14T21:49:35.2785654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2785989Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2786342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2786704Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2787064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2787428Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2787800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.2788186Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.2788338Z 2025-08-14T21:49:35.2788435Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2788779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2789086Z return mod(**inputs) 2025-08-14T21:49:35.2789416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2789765Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2790140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2790504Z layer_outputs = layer_module( 2025-08-14T21:49:35.2790823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2791167Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2791519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2791881Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2792226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2792588Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2792959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.2793334Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.2793484Z 2025-08-14T21:49:35.2793585Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2793912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2794220Z return mod(**inputs) 2025-08-14T21:49:35.2794553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2794906Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2795249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2795601Z layer_outputs = layer_module( 2025-08-14T21:49:35.2795925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2796256Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2796610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2797008Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2797364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2797721Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2798068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.2798420Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.2798541Z 2025-08-14T21:49:35.2798639Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2798980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2799287Z return mod(**inputs) 2025-08-14T21:49:35.2799620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2799969Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2800340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2800708Z layer_outputs = layer_module( 2025-08-14T21:49:35.2801029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2801374Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2801727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.2802095Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.2802465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:49:35.2802851Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.2803254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.2803620Z return self.weight * hidden_states 2025-08-14T21:49:35.2803744Z 2025-08-14T21:49:35.2803843Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2804185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2804503Z return mod(**inputs) 2025-08-14T21:49:35.2804848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2805230Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2805683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2806089Z layer_outputs = layer_module( 2025-08-14T21:49:35.2806455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2806822Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2807207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.2807580Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.2807940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.2808404Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.2808807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:49:35.2809191Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:49:35.2809350Z 2025-08-14T21:49:35.2809452Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2809806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2810163Z return mod(**inputs) 2025-08-14T21:49:35.2810502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2810875Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2811238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2811601Z layer_outputs = layer_module( 2025-08-14T21:49:35.2811947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2812311Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2812679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.2813059Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.2813438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.2813850Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.2814299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:49:35.2814670Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:49:35.2814808Z 2025-08-14T21:49:35.2814911Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2815265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2815637Z return mod(**inputs) 2025-08-14T21:49:35.2815986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2816358Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2816746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2817096Z layer_outputs = layer_module( 2025-08-14T21:49:35.2817420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2817761Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2818103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.2818478Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.2818846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.2819237Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.2819632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:49:35.2819999Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:49:35.2820134Z 2025-08-14T21:49:35.2820240Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2820581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2820885Z return mod(**inputs) 2025-08-14T21:49:35.2821219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2821580Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2821917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2822275Z layer_outputs = layer_module( 2025-08-14T21:49:35.2822599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2822938Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2823287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.2823713Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.2824080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.2824464Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.2824853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:49:35.2825212Z hidden_states = self.wo(hidden_states) 2025-08-14T21:49:35.2825340Z 2025-08-14T21:49:35.2825426Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.2825646Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2825987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2826297Z return mod(**inputs) 2025-08-14T21:49:35.2826623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2826977Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2827358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2827719Z layer_outputs = layer_module( 2025-08-14T21:49:35.2828036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2828380Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2828733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2829094Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2829446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:49:35.2829849Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.2830229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.2830581Z return self.weight * hidden_states 2025-08-14T21:49:35.2830715Z 2025-08-14T21:49:35.2830813Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2831152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2831461Z return mod(**inputs) 2025-08-14T21:49:35.2831780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2832131Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2832478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2832832Z layer_outputs = layer_module( 2025-08-14T21:49:35.2833149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2833481Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2833827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2834170Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2834526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2834892Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2835250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.2835601Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.2835731Z 2025-08-14T21:49:35.2835833Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2836178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2836494Z return mod(**inputs) 2025-08-14T21:49:35.2836823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2837170Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2837510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2838080Z layer_outputs = layer_module( 2025-08-14T21:49:35.2838411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2838755Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2839103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2839467Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2839827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2840191Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2840619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.2840977Z key_states = self.k(current_states) 2025-08-14T21:49:35.2841098Z 2025-08-14T21:49:35.2841206Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2841536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2841836Z return mod(**inputs) 2025-08-14T21:49:35.2842163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2842517Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2842880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2843240Z layer_outputs = layer_module( 2025-08-14T21:49:35.2843569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2843914Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2844260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2844623Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2844983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2845388Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2846016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.2846487Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.2846673Z 2025-08-14T21:49:35.2846786Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2847130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2847449Z return mod(**inputs) 2025-08-14T21:49:35.2847792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2848162Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2848504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2848859Z layer_outputs = layer_module( 2025-08-14T21:49:35.2849178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2849510Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2849866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2850324Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2850680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2851032Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2851386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.2851811Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.2852005Z 2025-08-14T21:49:35.2852108Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2852437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2852734Z return mod(**inputs) 2025-08-14T21:49:35.2853062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2853411Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2853779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2854152Z layer_outputs = layer_module( 2025-08-14T21:49:35.2854478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2854816Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2855170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2855537Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2856782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2857152Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2857550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.2857984Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.2858180Z 2025-08-14T21:49:35.2858281Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2858635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2858947Z return mod(**inputs) 2025-08-14T21:49:35.2859284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2859640Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2859995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2860358Z layer_outputs = layer_module( 2025-08-14T21:49:35.2860682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2861035Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2861400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2861765Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2862121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2862488Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2862852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.2863275Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.2863476Z 2025-08-14T21:49:35.2863577Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2863922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2864248Z return mod(**inputs) 2025-08-14T21:49:35.2864571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2864926Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2865274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2865625Z layer_outputs = layer_module( 2025-08-14T21:49:35.2865944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2866282Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2866633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2866973Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2867318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2867666Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2868044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.2868389Z value_states = self.v(current_states) 2025-08-14T21:49:35.2868519Z 2025-08-14T21:49:35.2868616Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2868956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2869253Z return mod(**inputs) 2025-08-14T21:49:35.2869580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2869933Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2870297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2870654Z layer_outputs = layer_module( 2025-08-14T21:49:35.2870972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2871304Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2871647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2871989Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2872336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2872681Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2873015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.2873386Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.2873543Z 2025-08-14T21:49:35.2873639Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2873968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2874259Z return mod(**inputs) 2025-08-14T21:49:35.2874578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2874922Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2875250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2875594Z layer_outputs = layer_module( 2025-08-14T21:49:35.2875908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2876236Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2876572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2876946Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2877293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2877644Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2877990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.2878380Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.2878525Z 2025-08-14T21:49:35.2878628Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2878958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2879260Z return mod(**inputs) 2025-08-14T21:49:35.2879587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2879942Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2880280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2880671Z layer_outputs = layer_module( 2025-08-14T21:49:35.2881008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2881348Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2881711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2882074Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2882433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2882790Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2883169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.2883565Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.2883722Z 2025-08-14T21:49:35.2883830Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2884169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2884485Z return mod(**inputs) 2025-08-14T21:49:35.2884824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2885181Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2885622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2886004Z layer_outputs = layer_module( 2025-08-14T21:49:35.2886352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2886721Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2887079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2887450Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2887795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2888156Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2888511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.2888870Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.2888993Z 2025-08-14T21:49:35.2889073Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.2889299Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2889640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2889965Z return mod(**inputs) 2025-08-14T21:49:35.2890299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2890658Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2891010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2891361Z layer_outputs = layer_module( 2025-08-14T21:49:35.2891692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2892036Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2892392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.2892759Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.2893127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:49:35.2893505Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.2893911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.2894277Z return self.weight * hidden_states 2025-08-14T21:49:35.2894411Z 2025-08-14T21:49:35.2894510Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2894856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2895173Z return mod(**inputs) 2025-08-14T21:49:35.2895502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2895858Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2896214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2896573Z layer_outputs = layer_module( 2025-08-14T21:49:35.2896899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2897240Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2897587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.2897956Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.2898322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.2898713Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.2899091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:49:35.2899467Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:49:35.2899609Z 2025-08-14T21:49:35.2899715Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2900044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2900363Z return mod(**inputs) 2025-08-14T21:49:35.2900684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2901029Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2901362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2901705Z layer_outputs = layer_module( 2025-08-14T21:49:35.2902022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2902345Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2902690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.2903071Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.2903428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.2903803Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.2904179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:49:35.2904529Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:49:35.2904652Z 2025-08-14T21:49:35.2904754Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2905086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2905389Z return mod(**inputs) 2025-08-14T21:49:35.2905719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2906080Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2906422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2906811Z layer_outputs = layer_module( 2025-08-14T21:49:35.2907126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2907447Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2907790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.2908150Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.2908494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.2908873Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.2909266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:49:35.2909637Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:49:35.2909769Z 2025-08-14T21:49:35.2909866Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2910203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2910509Z return mod(**inputs) 2025-08-14T21:49:35.2910836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2911190Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2911526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2911869Z layer_outputs = layer_module( 2025-08-14T21:49:35.2912177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2912514Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2912864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.2913228Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.2913577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.2913965Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.2914347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:49:35.2914699Z hidden_states = self.wo(hidden_states) 2025-08-14T21:49:35.2914822Z 2025-08-14T21:49:35.2914900Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.2915120Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2915457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2915776Z return mod(**inputs) 2025-08-14T21:49:35.2916107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2916470Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2916806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2917142Z layer_outputs = layer_module( 2025-08-14T21:49:35.2917458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2917786Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2918126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2918488Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2918849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:49:35.2919256Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.2919643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.2920004Z return self.weight * hidden_states 2025-08-14T21:49:35.2920128Z 2025-08-14T21:49:35.2920232Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2920563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2920867Z return mod(**inputs) 2025-08-14T21:49:35.2921200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2921561Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2921917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2922274Z layer_outputs = layer_module( 2025-08-14T21:49:35.2922608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2922947Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2923294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2923664Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2924041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2924412Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2924790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.2925165Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.2925302Z 2025-08-14T21:49:35.2925479Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2925850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2926173Z return mod(**inputs) 2025-08-14T21:49:35.2926522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2926894Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2927258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2927614Z layer_outputs = layer_module( 2025-08-14T21:49:35.2927944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2928275Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2928633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2929027Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2929379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2929738Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2930090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.2930447Z key_states = self.k(current_states) 2025-08-14T21:49:35.2930572Z 2025-08-14T21:49:35.2930670Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2931012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2931326Z return mod(**inputs) 2025-08-14T21:49:35.2931660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2932008Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2932356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2932753Z layer_outputs = layer_module( 2025-08-14T21:49:35.2933075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2933419Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2933773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2934135Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2934480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2934841Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2935212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.2935622Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.2935799Z 2025-08-14T21:49:35.2935899Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2936238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2936546Z return mod(**inputs) 2025-08-14T21:49:35.2936872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2937233Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2937581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2938145Z layer_outputs = layer_module( 2025-08-14T21:49:35.2938472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2938829Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2939201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2939567Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2939939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2940321Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2940681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.2941110Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.2941315Z 2025-08-14T21:49:35.2941416Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2941767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2942149Z return mod(**inputs) 2025-08-14T21:49:35.2942484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2942852Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2943204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2943563Z layer_outputs = layer_module( 2025-08-14T21:49:35.2943895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2944243Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2944607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2944966Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2945333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2945703Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2946112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.2946267Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.2946271Z 2025-08-14T21:49:35.2946372Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2946573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2946637Z return mod(**inputs) 2025-08-14T21:49:35.2946867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2946945Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2947209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2947289Z layer_outputs = layer_module( 2025-08-14T21:49:35.2947507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2947585Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2947819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2947898Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2948126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2948211Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2948440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.2948591Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.2948597Z 2025-08-14T21:49:35.2948697Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2948893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2948965Z return mod(**inputs) 2025-08-14T21:49:35.2949199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2949282Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2949514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2949585Z layer_outputs = layer_module( 2025-08-14T21:49:35.2949806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2949883Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2950115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2950263Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2950498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2950588Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2950819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.2950895Z value_states = self.v(current_states) 2025-08-14T21:49:35.2950899Z 2025-08-14T21:49:35.2951006Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2951199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2951264Z return mod(**inputs) 2025-08-14T21:49:35.2951508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2951580Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2951819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2951929Z layer_outputs = layer_module( 2025-08-14T21:49:35.2952143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2952228Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2952454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2952543Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2952772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2952851Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2953100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.2953209Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.2953214Z 2025-08-14T21:49:35.2953314Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2953514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2953578Z return mod(**inputs) 2025-08-14T21:49:35.2953839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2953910Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2954142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2954219Z layer_outputs = layer_module( 2025-08-14T21:49:35.2954429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2954513Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2954744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2954825Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2955067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2955146Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2955379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.2955492Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.2955495Z 2025-08-14T21:49:35.2955595Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2955797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2955887Z return mod(**inputs) 2025-08-14T21:49:35.2956130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2956217Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2956461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2956542Z layer_outputs = layer_module( 2025-08-14T21:49:35.2956761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2956838Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2957084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2957162Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2957388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2957478Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2957740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.2957848Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.2957852Z 2025-08-14T21:49:35.2957946Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2958131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2958202Z return mod(**inputs) 2025-08-14T21:49:35.2958426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2958495Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2958728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2958811Z layer_outputs = layer_module( 2025-08-14T21:49:35.2959025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2959102Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2959324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2959409Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2959632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2959719Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2959937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.2960009Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.2960013Z 2025-08-14T21:49:35.2960118Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2960307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2960371Z return mod(**inputs) 2025-08-14T21:49:35.2960604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2960674Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2960906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2960974Z layer_outputs = layer_module( 2025-08-14T21:49:35.2961181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2961261Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2961485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2961584Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2961812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 485, in forward 2025-08-14T21:49:35.2961939Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-14T21:49:35.2961943Z 2025-08-14T21:49:35.2962026Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.2962122Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2962308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2962377Z return mod(**inputs) 2025-08-14T21:49:35.2962603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2962680Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2962907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2962978Z layer_outputs = layer_module( 2025-08-14T21:49:35.2963199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2963323Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2963557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.2963654Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.2963884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:49:35.2963984Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.2964212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.2964291Z return self.weight * hidden_states 2025-08-14T21:49:35.2964295Z 2025-08-14T21:49:35.2964427Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2964622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2964697Z return mod(**inputs) 2025-08-14T21:49:35.2964928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2965001Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2965243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2965312Z layer_outputs = layer_module( 2025-08-14T21:49:35.2965610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2965703Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2965941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.2966044Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.2966283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.2966403Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.2966647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:49:35.2966742Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:49:35.2966745Z 2025-08-14T21:49:35.2966843Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2967040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2967103Z return mod(**inputs) 2025-08-14T21:49:35.2967336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2967434Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2967662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2967744Z layer_outputs = layer_module( 2025-08-14T21:49:35.2967951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2968037Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2968263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.2968349Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.2968582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.2968694Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.2968916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:49:35.2968999Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:49:35.2969019Z 2025-08-14T21:49:35.2969133Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2969331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2969392Z return mod(**inputs) 2025-08-14T21:49:35.2969618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2969697Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2969921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2969997Z layer_outputs = layer_module( 2025-08-14T21:49:35.2970207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2971180Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2971431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.2971517Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.2971743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.2971862Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.2972090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:49:35.2972182Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:49:35.2972185Z 2025-08-14T21:49:35.2972282Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2972471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2972542Z return mod(**inputs) 2025-08-14T21:49:35.2972785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2972855Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2973087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2973154Z layer_outputs = layer_module( 2025-08-14T21:49:35.2973365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2973436Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2973656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.2973745Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.2973967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.2974110Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.2974327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:49:35.2974401Z hidden_states = self.wo(hidden_states) 2025-08-14T21:49:35.2974404Z 2025-08-14T21:49:35.2974487Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.2974582Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2974766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2974836Z return mod(**inputs) 2025-08-14T21:49:35.2975063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2975137Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2975363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2975430Z layer_outputs = layer_module( 2025-08-14T21:49:35.2975682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2975756Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2975975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2976058Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2976277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:49:35.2976381Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.2976602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.2976692Z return self.weight * hidden_states 2025-08-14T21:49:35.2976697Z 2025-08-14T21:49:35.2976803Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2976992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2977060Z return mod(**inputs) 2025-08-14T21:49:35.2977291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2977359Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2977592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2977660Z layer_outputs = layer_module( 2025-08-14T21:49:35.2977868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2977949Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2978175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2978260Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2978486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2978564Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2978799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.2978873Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.2978876Z 2025-08-14T21:49:35.2978976Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2979162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2979222Z return mod(**inputs) 2025-08-14T21:49:35.2979461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2979547Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2979769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2979848Z layer_outputs = layer_module( 2025-08-14T21:49:35.2980046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2980127Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2980342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2980442Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2980664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2980740Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2980956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.2981037Z key_states = self.k(current_states) 2025-08-14T21:49:35.2981055Z 2025-08-14T21:49:35.2981166Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2981356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2981416Z return mod(**inputs) 2025-08-14T21:49:35.2981634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2981708Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2981935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2982008Z layer_outputs = layer_module( 2025-08-14T21:49:35.2982205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2982323Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2982550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2982628Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2982843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2982925Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2983139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.2983267Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.2983271Z 2025-08-14T21:49:35.2983364Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2983545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2983618Z return mod(**inputs) 2025-08-14T21:49:35.2983837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2983914Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2984133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2984198Z layer_outputs = layer_module( 2025-08-14T21:49:35.2984409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2984483Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2984704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2984789Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2985010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2985114Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2985339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.2985484Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.2985487Z 2025-08-14T21:49:35.2985593Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2985778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2985847Z return mod(**inputs) 2025-08-14T21:49:35.2986073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2986142Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2986375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2986445Z layer_outputs = layer_module( 2025-08-14T21:49:35.2986648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2986772Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2986995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2987080Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2987299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2987375Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2987606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.2987751Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.2987756Z 2025-08-14T21:49:35.2987878Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2988064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2988129Z return mod(**inputs) 2025-08-14T21:49:35.2988361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2988442Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2988661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2988735Z layer_outputs = layer_module( 2025-08-14T21:49:35.2988933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2989011Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2989226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2989302Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2989525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2989600Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2989812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.2989954Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.2989957Z 2025-08-14T21:49:35.2990052Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2990243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2990303Z return mod(**inputs) 2025-08-14T21:49:35.2990525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2990618Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2990846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2990922Z layer_outputs = layer_module( 2025-08-14T21:49:35.2991125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2991197Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2991424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2991499Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2991716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2991796Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2992018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.2992100Z value_states = self.v(current_states) 2025-08-14T21:49:35.2992120Z 2025-08-14T21:49:35.2992229Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2992413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2992482Z return mod(**inputs) 2025-08-14T21:49:35.2992700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2992769Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2992998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2993064Z layer_outputs = layer_module( 2025-08-14T21:49:35.2993269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2993358Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2993584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2993666Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2993885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2993966Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2994185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.2994282Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.2994286Z 2025-08-14T21:49:35.2994386Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2994569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2994634Z return mod(**inputs) 2025-08-14T21:49:35.2994870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2994942Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2995176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2995242Z layer_outputs = layer_module( 2025-08-14T21:49:35.2995451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2995533Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2995759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2995841Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2996068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2996165Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2996406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.2996509Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.2996513Z 2025-08-14T21:49:35.2996611Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2996807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2996870Z return mod(**inputs) 2025-08-14T21:49:35.2997111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2997191Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2997421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2997501Z layer_outputs = layer_module( 2025-08-14T21:49:35.2997711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.2997817Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.2998049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.2998125Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.2998354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.2998432Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.2998653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.2998760Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.2998763Z 2025-08-14T21:49:35.2998877Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.2999071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.2999136Z return mod(**inputs) 2025-08-14T21:49:35.2999363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.2999441Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.2999665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.2999733Z layer_outputs = layer_module( 2025-08-14T21:49:35.2999944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3000016Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3000249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3000327Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3000551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3000636Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3000862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.3000943Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.3000946Z 2025-08-14T21:49:35.3001023Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3001117Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3001312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3001373Z return mod(**inputs) 2025-08-14T21:49:35.3001601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3001703Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3001929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3002004Z layer_outputs = layer_module( 2025-08-14T21:49:35.3002208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3002281Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3002511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3002598Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3002821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:49:35.3002918Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3003140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3003221Z return self.weight * hidden_states 2025-08-14T21:49:35.3003247Z 2025-08-14T21:49:35.3003358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3003547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3003617Z return mod(**inputs) 2025-08-14T21:49:35.3003843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3003921Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3004147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3004214Z layer_outputs = layer_module( 2025-08-14T21:49:35.3004426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3004516Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3004738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3004836Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3005061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3005181Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3005479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:49:35.3005592Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:49:35.3005596Z 2025-08-14T21:49:35.3005705Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3005898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3005976Z return mod(**inputs) 2025-08-14T21:49:35.3006223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3006304Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3006566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3006642Z layer_outputs = layer_module( 2025-08-14T21:49:35.3006873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3006956Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3007181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3007274Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3007500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3007633Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3007866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:49:35.3007942Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:49:35.3007946Z 2025-08-14T21:49:35.3008050Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3008235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3008297Z return mod(**inputs) 2025-08-14T21:49:35.3008529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3008600Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3008826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3008903Z layer_outputs = layer_module( 2025-08-14T21:49:35.3009111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3009242Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3009466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3009548Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3009778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3009885Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3010105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:49:35.3010200Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:49:35.3010205Z 2025-08-14T21:49:35.3010319Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3010513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3010578Z return mod(**inputs) 2025-08-14T21:49:35.3010806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3010882Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3011106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3011182Z layer_outputs = layer_module( 2025-08-14T21:49:35.3011388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3011462Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3011692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3011777Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3012001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3012115Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3012338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:49:35.3012421Z hidden_states = self.wo(hidden_states) 2025-08-14T21:49:35.3012425Z 2025-08-14T21:49:35.3012501Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3012597Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3012789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3012852Z return mod(**inputs) 2025-08-14T21:49:35.3013077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3013173Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3013400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3013474Z layer_outputs = layer_module( 2025-08-14T21:49:35.3013677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3013760Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3013979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3014053Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3014275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:49:35.3014371Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3014588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3014682Z return self.weight * hidden_states 2025-08-14T21:49:35.3014701Z 2025-08-14T21:49:35.3014796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3014976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3015043Z return mod(**inputs) 2025-08-14T21:49:35.3015261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3015335Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3015552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3015617Z layer_outputs = layer_module( 2025-08-14T21:49:35.3015836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3015911Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3016139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3016212Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3016425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3016507Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3016722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.3016792Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.3016796Z 2025-08-14T21:49:35.3016897Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3017079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3017150Z return mod(**inputs) 2025-08-14T21:49:35.3017366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3017436Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3017661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3017727Z layer_outputs = layer_module( 2025-08-14T21:49:35.3017924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3018003Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3018216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3018297Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3018511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3018604Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3018831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.3018902Z key_states = self.k(current_states) 2025-08-14T21:49:35.3018905Z 2025-08-14T21:49:35.3019005Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3019187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3019248Z return mod(**inputs) 2025-08-14T21:49:35.3019475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3019542Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3019761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3019838Z layer_outputs = layer_module( 2025-08-14T21:49:35.3020039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3020150Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3020371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3020445Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3020673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3020749Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3020968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.3021116Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.3021119Z 2025-08-14T21:49:35.3021229Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3021420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3021483Z return mod(**inputs) 2025-08-14T21:49:35.3021703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3021778Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3021997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3022069Z layer_outputs = layer_module( 2025-08-14T21:49:35.3022268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3022338Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3022562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3022638Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3022869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3022955Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3023178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3023327Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3023331Z 2025-08-14T21:49:35.3023427Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3023611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3023680Z return mod(**inputs) 2025-08-14T21:49:35.3023904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3023999Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3024225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3024295Z layer_outputs = layer_module( 2025-08-14T21:49:35.3024507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3024577Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3024797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3024881Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3025101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3025184Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3025427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3025567Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3025585Z 2025-08-14T21:49:35.3025708Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3025895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3025964Z return mod(**inputs) 2025-08-14T21:49:35.3026189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3026259Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3026494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3026565Z layer_outputs = layer_module( 2025-08-14T21:49:35.3026790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3026876Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3027104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3027188Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3027412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3027487Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3027718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3027857Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3027861Z 2025-08-14T21:49:35.3027962Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3028149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3028211Z return mod(**inputs) 2025-08-14T21:49:35.3028443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3028515Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3028742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3028817Z layer_outputs = layer_module( 2025-08-14T21:49:35.3029029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3029107Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3029322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3029395Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3029621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3029719Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3029938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.3030017Z value_states = self.v(current_states) 2025-08-14T21:49:35.3030020Z 2025-08-14T21:49:35.3030115Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3030300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3030361Z return mod(**inputs) 2025-08-14T21:49:35.3030577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3030652Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3030870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3030945Z layer_outputs = layer_module( 2025-08-14T21:49:35.3031144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3031247Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3031472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3031547Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3031760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3031842Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3032058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3032163Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3032166Z 2025-08-14T21:49:35.3032275Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3032458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3032528Z return mod(**inputs) 2025-08-14T21:49:35.3032747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3032822Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3033038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3033105Z layer_outputs = layer_module( 2025-08-14T21:49:35.3033311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3033382Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3033597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3033681Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3033897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3033979Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3034197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3034296Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3034299Z 2025-08-14T21:49:35.3034403Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3034588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3034651Z return mod(**inputs) 2025-08-14T21:49:35.3034895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3034980Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3035210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3035278Z layer_outputs = layer_module( 2025-08-14T21:49:35.3035476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3035555Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3035770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3035852Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3036064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3036141Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3036362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.3036461Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.3036481Z 2025-08-14T21:49:35.3036590Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3036781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3036841Z return mod(**inputs) 2025-08-14T21:49:35.3037065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3037132Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3037349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3037425Z layer_outputs = layer_module( 2025-08-14T21:49:35.3037816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3037949Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3038169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3038247Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3038473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3038549Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3038766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.3038847Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.3038851Z 2025-08-14T21:49:35.3038946Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3039138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3039200Z return mod(**inputs) 2025-08-14T21:49:35.3039423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3039504Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3039730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3039799Z layer_outputs = layer_module( 2025-08-14T21:49:35.3040013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3040088Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3040320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3040396Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3040617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 485, in forward 2025-08-14T21:49:35.3040802Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-14T21:49:35.3040806Z 2025-08-14T21:49:35.3040882Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3040986Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3041172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3041233Z return mod(**inputs) 2025-08-14T21:49:35.3041466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3041535Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3041758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3041834Z layer_outputs = layer_module( 2025-08-14T21:49:35.3042039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3042121Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3042343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3042480Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3042713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:49:35.3042804Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3043022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3043101Z return self.weight * hidden_states 2025-08-14T21:49:35.3043104Z 2025-08-14T21:49:35.3043198Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3043393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3043472Z return mod(**inputs) 2025-08-14T21:49:35.3043702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3043783Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3044010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3044083Z layer_outputs = layer_module( 2025-08-14T21:49:35.3044287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3044359Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3044591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3044675Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3044906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3045027Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3045257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:49:35.3045360Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:49:35.3045364Z 2025-08-14T21:49:35.3045512Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3045709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3045784Z return mod(**inputs) 2025-08-14T21:49:35.3046035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3046122Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3046374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3046475Z layer_outputs = layer_module( 2025-08-14T21:49:35.3046704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3046781Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3047007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3047101Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3047325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3047443Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3047667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:49:35.3047742Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:49:35.3047746Z 2025-08-14T21:49:35.3047854Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3048041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3048151Z return mod(**inputs) 2025-08-14T21:49:35.3048379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3048449Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3048683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3048751Z layer_outputs = layer_module( 2025-08-14T21:49:35.3048954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3049037Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3049259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3049377Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3049603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3049711Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3049939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:49:35.3050022Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:49:35.3050025Z 2025-08-14T21:49:35.3050128Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3050314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3050376Z return mod(**inputs) 2025-08-14T21:49:35.3050610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3050682Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3050906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3050982Z layer_outputs = layer_module( 2025-08-14T21:49:35.3051185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3051265Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3051485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3051569Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3051797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3051902Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3052121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:49:35.3052225Z hidden_states = self.wo(hidden_states) 2025-08-14T21:49:35.3052230Z 2025-08-14T21:49:35.3052308Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3052414Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3052598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3052660Z return mod(**inputs) 2025-08-14T21:49:35.3052894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3052964Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3053192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3053260Z layer_outputs = layer_module( 2025-08-14T21:49:35.3053467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3053551Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3053821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3053899Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3054133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:49:35.3054231Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3054457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3054531Z return self.weight * hidden_states 2025-08-14T21:49:35.3054534Z 2025-08-14T21:49:35.3054629Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3054836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3054902Z return mod(**inputs) 2025-08-14T21:49:35.3055127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3055207Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3055431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3055507Z layer_outputs = layer_module( 2025-08-14T21:49:35.3055713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3055785Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3056012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3056087Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3056315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3056394Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3056619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.3056699Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.3056703Z 2025-08-14T21:49:35.3056797Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3056983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3057053Z return mod(**inputs) 2025-08-14T21:49:35.3057276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3057351Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3057575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3057662Z layer_outputs = layer_module( 2025-08-14T21:49:35.3057874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3057951Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3058174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3058257Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3058478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3058566Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3058789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.3058862Z key_states = self.k(current_states) 2025-08-14T21:49:35.3058865Z 2025-08-14T21:49:35.3058976Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3059164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3059250Z return mod(**inputs) 2025-08-14T21:49:35.3059495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3059566Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3059799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3059867Z layer_outputs = layer_module( 2025-08-14T21:49:35.3060070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3060149Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3060379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3060493Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3060711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3060790Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3061015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.3061136Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.3061140Z 2025-08-14T21:49:35.3061244Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3061425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3061486Z return mod(**inputs) 2025-08-14T21:49:35.3061714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3061786Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3062006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3062083Z layer_outputs = layer_module( 2025-08-14T21:49:35.3062286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3062368Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3062583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3062658Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3062882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3062957Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3063174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3063349Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3063354Z 2025-08-14T21:49:35.3063448Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3063640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3063702Z return mod(**inputs) 2025-08-14T21:49:35.3063922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3063997Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3064219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3064292Z layer_outputs = layer_module( 2025-08-14T21:49:35.3064500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3064575Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3064809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3064918Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3065146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3065230Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3065451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3065600Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3065604Z 2025-08-14T21:49:35.3065701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3065885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3065972Z return mod(**inputs) 2025-08-14T21:49:35.3066199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3066280Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3066504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3066571Z layer_outputs = layer_module( 2025-08-14T21:49:35.3066789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3066861Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3067077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3067160Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3067382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3067467Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3067688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3067825Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3067829Z 2025-08-14T21:49:35.3067932Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3068112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3068182Z return mod(**inputs) 2025-08-14T21:49:35.3068406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3068474Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3068705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3068790Z layer_outputs = layer_module( 2025-08-14T21:49:35.3068991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3069074Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3069292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3069373Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3069589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3069664Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3069887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.3069959Z value_states = self.v(current_states) 2025-08-14T21:49:35.3069962Z 2025-08-14T21:49:35.3070066Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3070247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3070390Z return mod(**inputs) 2025-08-14T21:49:35.3070621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3070690Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3070911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3070987Z layer_outputs = layer_module( 2025-08-14T21:49:35.3071187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3071266Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3071498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3071580Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3071808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3071883Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3072099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3072209Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3072212Z 2025-08-14T21:49:35.3072304Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3072494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3072555Z return mod(**inputs) 2025-08-14T21:49:35.3072775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3072858Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3073078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3073155Z layer_outputs = layer_module( 2025-08-14T21:49:35.3073358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3073431Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3073656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3073730Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3073951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3074035Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3074252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3074377Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3074381Z 2025-08-14T21:49:35.3074477Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3074661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3074735Z return mod(**inputs) 2025-08-14T21:49:35.3074963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3075040Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3075271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3075338Z layer_outputs = layer_module( 2025-08-14T21:49:35.3075556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3075635Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3075856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3075978Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3076204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3076294Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3076524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.3076622Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.3076625Z 2025-08-14T21:49:35.3076726Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3076909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3076969Z return mod(**inputs) 2025-08-14T21:49:35.3077215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3077285Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3077512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3077580Z layer_outputs = layer_module( 2025-08-14T21:49:35.3077786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3077871Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3078092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3078176Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3078402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3078483Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3078717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.3078797Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.3078801Z 2025-08-14T21:49:35.3078879Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3078983Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3079173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3079242Z return mod(**inputs) 2025-08-14T21:49:35.3079470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3079540Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3079775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3079863Z layer_outputs = layer_module( 2025-08-14T21:49:35.3080071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3080158Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3080383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3080478Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3080702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:49:35.3080794Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3081025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3081100Z return self.weight * hidden_states 2025-08-14T21:49:35.3081103Z 2025-08-14T21:49:35.3081211Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3081401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3081485Z return mod(**inputs) 2025-08-14T21:49:35.3081737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3081809Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3082041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3082118Z layer_outputs = layer_module( 2025-08-14T21:49:35.3082327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3082410Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3082634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3082734Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3082968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3083078Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3083331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:49:35.3083427Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:49:35.3083431Z 2025-08-14T21:49:35.3083529Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3083727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3083792Z return mod(**inputs) 2025-08-14T21:49:35.3084023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3084106Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3084337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3084418Z layer_outputs = layer_module( 2025-08-14T21:49:35.3084628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3084703Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3084945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3085033Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3085267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3085387Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3085706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:49:35.3085823Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:49:35.3085828Z 2025-08-14T21:49:35.3085943Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3086139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3086216Z return mod(**inputs) 2025-08-14T21:49:35.3086455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3086536Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3086770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3086839Z layer_outputs = layer_module( 2025-08-14T21:49:35.3087061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3087152Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3087378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3087508Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3087761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3087881Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3088134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:49:35.3088226Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:49:35.3088229Z 2025-08-14T21:49:35.3088345Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3088535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3088631Z return mod(**inputs) 2025-08-14T21:49:35.3088867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3088954Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3089189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3089258Z layer_outputs = layer_module( 2025-08-14T21:49:35.3089469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3089552Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3089774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3089865Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3090091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3090198Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3090432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:49:35.3090509Z hidden_states = self.wo(hidden_states) 2025-08-14T21:49:35.3090512Z 2025-08-14T21:49:35.3090595Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3090692Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3090883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3090954Z return mod(**inputs) 2025-08-14T21:49:35.3091181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3091256Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3091496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3091583Z layer_outputs = layer_module( 2025-08-14T21:49:35.3091799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3091876Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3092099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3092186Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3092408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:49:35.3092508Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3092739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3092814Z return self.weight * hidden_states 2025-08-14T21:49:35.3092820Z 2025-08-14T21:49:35.3092928Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3093161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3093226Z return mod(**inputs) 2025-08-14T21:49:35.3093468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3093540Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3093777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3093846Z layer_outputs = layer_module( 2025-08-14T21:49:35.3094058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3094140Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3094382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3094467Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3094707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3094787Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3095046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.3095122Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.3095126Z 2025-08-14T21:49:35.3095225Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3095423Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3095487Z return mod(**inputs) 2025-08-14T21:49:35.3095725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3095811Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3096049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3096134Z layer_outputs = layer_module( 2025-08-14T21:49:35.3096361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3096438Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3096676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3096757Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3096998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3097077Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3097305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.3097411Z key_states = self.k(current_states) 2025-08-14T21:49:35.3097416Z 2025-08-14T21:49:35.3097516Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3097705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3097778Z return mod(**inputs) 2025-08-14T21:49:35.3098006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3098083Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3098311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3098377Z layer_outputs = layer_module( 2025-08-14T21:49:35.3098592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3098670Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3098904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3099021Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3099252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3099338Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3099563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.3099689Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.3099692Z 2025-08-14T21:49:35.3099800Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3099989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3100080Z return mod(**inputs) 2025-08-14T21:49:35.3100309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3100384Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3100622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3100691Z layer_outputs = layer_module( 2025-08-14T21:49:35.3100900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3100985Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3101213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3101298Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3101525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3101606Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3101843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3101990Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3101994Z 2025-08-14T21:49:35.3102101Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3102291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3102355Z return mod(**inputs) 2025-08-14T21:49:35.3102593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3102664Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3102896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3102993Z layer_outputs = layer_module( 2025-08-14T21:49:35.3103206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3103293Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3103520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3103599Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3103838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3103916Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3104151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3104296Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3104301Z 2025-08-14T21:49:35.3104401Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3104600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3104696Z return mod(**inputs) 2025-08-14T21:49:35.3104935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3105012Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3105242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3105317Z layer_outputs = layer_module( 2025-08-14T21:49:35.3105527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3105601Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3105850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3105931Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3106169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3106246Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3106475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3106622Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3106626Z 2025-08-14T21:49:35.3106724Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3106912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3106983Z return mod(**inputs) 2025-08-14T21:49:35.3107215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3107296Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3107527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3107599Z layer_outputs = layer_module( 2025-08-14T21:49:35.3107819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3107895Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3108126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3108210Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3108438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3108525Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3108755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.3108852Z value_states = self.v(current_states) 2025-08-14T21:49:35.3108857Z 2025-08-14T21:49:35.3108968Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3109171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3109242Z return mod(**inputs) 2025-08-14T21:49:35.3109476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3109546Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3109786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3109853Z layer_outputs = layer_module( 2025-08-14T21:49:35.3110062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3110145Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3110373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3110491Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3110720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3110797Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3111029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3111130Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3111134Z 2025-08-14T21:49:35.3111236Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3111421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3111499Z return mod(**inputs) 2025-08-14T21:49:35.3111731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3111803Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3112024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3112100Z layer_outputs = layer_module( 2025-08-14T21:49:35.3112304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3112384Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3112603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3112679Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3112906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3112982Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3113202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3113310Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3113313Z 2025-08-14T21:49:35.3113408Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3113599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3113660Z return mod(**inputs) 2025-08-14T21:49:35.3113883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3113958Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3114181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3114283Z layer_outputs = layer_module( 2025-08-14T21:49:35.3114490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3114567Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3114803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3114882Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3115115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3115204Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3115440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.3115553Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.3115556Z 2025-08-14T21:49:35.3115660Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3115858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3115951Z return mod(**inputs) 2025-08-14T21:49:35.3116211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3116290Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3116519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3116588Z layer_outputs = layer_module( 2025-08-14T21:49:35.3116802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3116875Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3117098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3117199Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3117426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3117515Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3117737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.3117812Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.3117819Z 2025-08-14T21:49:35.3117924Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3118113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3118175Z return mod(**inputs) 2025-08-14T21:49:35.3118409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3118478Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3118711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3118780Z layer_outputs = layer_module( 2025-08-14T21:49:35.3118989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3119071Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3119293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3119378Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3119599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 485, in forward 2025-08-14T21:49:35.3119725Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-14T21:49:35.3119729Z 2025-08-14T21:49:35.3119814Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3119934Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3120121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3120193Z return mod(**inputs) 2025-08-14T21:49:35.3120421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3120498Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3120722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3120789Z layer_outputs = layer_module( 2025-08-14T21:49:35.3121007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3121081Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3121308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3121409Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3121635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:49:35.3121769Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3121999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3122074Z return self.weight * hidden_states 2025-08-14T21:49:35.3122078Z 2025-08-14T21:49:35.3122184Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3122371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3122445Z return mod(**inputs) 2025-08-14T21:49:35.3122676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3122763Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3123005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3123077Z layer_outputs = layer_module( 2025-08-14T21:49:35.3123288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3123373Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3123603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3123697Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3123924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3124037Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3124273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:49:35.3124373Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:49:35.3124378Z 2025-08-14T21:49:35.3124494Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3124702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3124773Z return mod(**inputs) 2025-08-14T21:49:35.3125033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3125108Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3125355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3125518Z layer_outputs = layer_module( 2025-08-14T21:49:35.3125755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3125871Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3126123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3126220Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3126476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3126586Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3126820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:49:35.3126898Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:49:35.3126902Z 2025-08-14T21:49:35.3127009Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3127225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3127298Z return mod(**inputs) 2025-08-14T21:49:35.3127547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3127652Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3127919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3128005Z layer_outputs = layer_module( 2025-08-14T21:49:35.3128235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3128317Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3128570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3128664Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3128931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3129059Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3129308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:49:35.3129409Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:49:35.3129413Z 2025-08-14T21:49:35.3129520Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3129726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3129803Z return mod(**inputs) 2025-08-14T21:49:35.3130055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3130138Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3130385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3130464Z layer_outputs = layer_module( 2025-08-14T21:49:35.3130697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3130781Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3131027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3131128Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3131373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3131497Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3131746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:49:35.3131828Z hidden_states = self.wo(hidden_states) 2025-08-14T21:49:35.3131832Z 2025-08-14T21:49:35.3131926Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3132060Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3132279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3132350Z return mod(**inputs) 2025-08-14T21:49:35.3132603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1750, in forward 2025-08-14T21:49:35.3132687Z encoder_outputs = self.encoder( 2025-08-14T21:49:35.3132941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1115, in forward 2025-08-14T21:49:35.3133051Z hidden_states = self.final_layer_norm(hidden_states) 2025-08-14T21:49:35.3133310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3133390Z return self.weight * hidden_states 2025-08-14T21:49:35.3133394Z 2025-08-14T21:49:35.3133515Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3133703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3133784Z return mod(**inputs) 2025-08-14T21:49:35.3134037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3134109Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3134340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3134416Z layer_outputs = layer_module( 2025-08-14T21:49:35.3134625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3134706Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3134932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3135037Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3135272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3135357Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3135595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.3135671Z key_states = self.k(current_states) 2025-08-14T21:49:35.3135674Z 2025-08-14T21:49:35.3135775Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3135976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3136044Z return mod(**inputs) 2025-08-14T21:49:35.3136277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3136360Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3136606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3136686Z layer_outputs = layer_module( 2025-08-14T21:49:35.3136896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3136973Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3137204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3137283Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3137513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3137773Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3138006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.3138191Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.3138196Z 2025-08-14T21:49:35.3138296Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3138487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3138558Z return mod(**inputs) 2025-08-14T21:49:35.3138785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3138864Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3139091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3139159Z layer_outputs = layer_module( 2025-08-14T21:49:35.3139372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3139448Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3139668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3139802Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3140026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3140114Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3140337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3140480Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3140484Z 2025-08-14T21:49:35.3140587Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3140772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3140868Z return mod(**inputs) 2025-08-14T21:49:35.3141100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3141172Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3141410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3141476Z layer_outputs = layer_module( 2025-08-14T21:49:35.3141687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3141765Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3141989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3142070Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3142294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3142376Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3142610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.3142685Z value_states = self.v(current_states) 2025-08-14T21:49:35.3142688Z 2025-08-14T21:49:35.3142789Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3142976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3143035Z return mod(**inputs) 2025-08-14T21:49:35.3143270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3143339Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3143567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3143644Z layer_outputs = layer_module( 2025-08-14T21:49:35.3143867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3143950Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3144173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3144249Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3144480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3144558Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3144781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3144891Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3144895Z 2025-08-14T21:49:35.3144989Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3145185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3145267Z return mod(**inputs) 2025-08-14T21:49:35.3145532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3145614Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3145839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3145914Z layer_outputs = layer_module( 2025-08-14T21:49:35.3146117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3146189Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3146418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3146510Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3146732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3146823Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3147046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3147149Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3147153Z 2025-08-14T21:49:35.3147248Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3147433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3147503Z return mod(**inputs) 2025-08-14T21:49:35.3147726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3147814Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3148043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3148113Z layer_outputs = layer_module( 2025-08-14T21:49:35.3148328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3148401Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3148621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3148704Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3148920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3149003Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3149225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.3149344Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.3149347Z 2025-08-14T21:49:35.3149449Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3149639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3149708Z return mod(**inputs) 2025-08-14T21:49:35.3149936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3150006Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3150241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3150308Z layer_outputs = layer_module( 2025-08-14T21:49:35.3150516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3150596Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3150823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3150923Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3151171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3151248Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3151472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.3151542Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.3151546Z 2025-08-14T21:49:35.3151619Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3151721Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3151902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3151969Z return mod(**inputs) 2025-08-14T21:49:35.3152206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3152277Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3152507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3152574Z layer_outputs = layer_module( 2025-08-14T21:49:35.3152784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3152854Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3153068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3153157Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3153373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:49:35.3153462Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3153689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3153762Z return self.weight * hidden_states 2025-08-14T21:49:35.3153765Z 2025-08-14T21:49:35.3153862Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3154041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3154102Z return mod(**inputs) 2025-08-14T21:49:35.3154328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3154395Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3154615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3154691Z layer_outputs = layer_module( 2025-08-14T21:49:35.3154922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3155005Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3155227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3155312Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3155545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3155653Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3155886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:49:35.3155980Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:49:35.3155984Z 2025-08-14T21:49:35.3156080Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3156274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3156352Z return mod(**inputs) 2025-08-14T21:49:35.3156615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3156694Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3156912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3156986Z layer_outputs = layer_module( 2025-08-14T21:49:35.3157184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3157256Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3157478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3157575Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3157799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3157905Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3158120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:49:35.3158200Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:49:35.3158203Z 2025-08-14T21:49:35.3158297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3158478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3158548Z return mod(**inputs) 2025-08-14T21:49:35.3158770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3158847Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3159074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3159143Z layer_outputs = layer_module( 2025-08-14T21:49:35.3159354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3159426Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3159651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3159741Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3159962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3160074Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3160298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:49:35.3160400Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:49:35.3160406Z 2025-08-14T21:49:35.3160513Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3160707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3160777Z return mod(**inputs) 2025-08-14T21:49:35.3161006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3161079Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3161316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3161384Z layer_outputs = layer_module( 2025-08-14T21:49:35.3161595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3161683Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3161909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3162063Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3162290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3162399Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3162630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:49:35.3162707Z hidden_states = self.wo(hidden_states) 2025-08-14T21:49:35.3162710Z 2025-08-14T21:49:35.3162817Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3163005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3163068Z return mod(**inputs) 2025-08-14T21:49:35.3163322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3163397Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3163630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3163706Z layer_outputs = layer_module( 2025-08-14T21:49:35.3163919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3164004Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3164236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3164316Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3164554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:49:35.3164660Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3164925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3165008Z return self.weight * hidden_states 2025-08-14T21:49:35.3165012Z 2025-08-14T21:49:35.3165120Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3165339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3165470Z return mod(**inputs) 2025-08-14T21:49:35.3165732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3165820Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3166069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3166154Z layer_outputs = layer_module( 2025-08-14T21:49:35.3166410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3166493Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3166753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3166831Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3167056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3167144Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3167366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.3167446Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.3167450Z 2025-08-14T21:49:35.3167548Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3167740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3167839Z return mod(**inputs) 2025-08-14T21:49:35.3168138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3168276Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3168666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3168774Z layer_outputs = layer_module( 2025-08-14T21:49:35.3169156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3169293Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3169591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3169683Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3169957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3170056Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3170301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.3170381Z key_states = self.k(current_states) 2025-08-14T21:49:35.3170385Z 2025-08-14T21:49:35.3170502Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3170713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3170789Z return mod(**inputs) 2025-08-14T21:49:35.3171044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3171121Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3171388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3171467Z layer_outputs = layer_module( 2025-08-14T21:49:35.3171712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3171800Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3172046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3172137Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3172382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3172465Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3172719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.3172857Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.3172880Z 2025-08-14T21:49:35.3172990Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3173207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3173276Z return mod(**inputs) 2025-08-14T21:49:35.3173535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3173611Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3173865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3173948Z layer_outputs = layer_module( 2025-08-14T21:49:35.3174178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3174267Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3174519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3174622Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3174894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3174982Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3175229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3175399Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3175403Z 2025-08-14T21:49:35.3175509Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3175723Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3175793Z return mod(**inputs) 2025-08-14T21:49:35.3176064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3176152Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3176412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3176495Z layer_outputs = layer_module( 2025-08-14T21:49:35.3176730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3176812Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3177072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3177156Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3177410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3177504Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3177762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.3177853Z value_states = self.v(current_states) 2025-08-14T21:49:35.3177858Z 2025-08-14T21:49:35.3177966Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3178176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3178251Z return mod(**inputs) 2025-08-14T21:49:35.3178509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3178586Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3178850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3178925Z layer_outputs = layer_module( 2025-08-14T21:49:35.3179164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3179263Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3179520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3179613Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3179862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3179956Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3180202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3180308Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3180312Z 2025-08-14T21:49:35.3180419Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3180615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3180681Z return mod(**inputs) 2025-08-14T21:49:35.3180976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3181050Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3181304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3181379Z layer_outputs = layer_module( 2025-08-14T21:49:35.3181622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3181739Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3182121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3182231Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3182618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3182727Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3183114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3183257Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3183262Z 2025-08-14T21:49:35.3183398Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3183699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3183780Z return mod(**inputs) 2025-08-14T21:49:35.3184168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3184259Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3184644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3184745Z layer_outputs = layer_module( 2025-08-14T21:49:35.3185110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3185213Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3185586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3185706Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3185974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3186057Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3186356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.3186477Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.3186507Z 2025-08-14T21:49:35.3186614Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3186831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3186899Z return mod(**inputs) 2025-08-14T21:49:35.3187148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3187232Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3187481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3187555Z layer_outputs = layer_module( 2025-08-14T21:49:35.3187790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3187879Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3188121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3188202Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3188473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3188563Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3188795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.3188880Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.3188884Z 2025-08-14T21:49:35.3188968Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3189074Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3189289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3189358Z return mod(**inputs) 2025-08-14T21:49:35.3189624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3189714Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3189969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3190052Z layer_outputs = layer_module( 2025-08-14T21:49:35.3190282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3190363Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3190617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3190703Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3190949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-08-14T21:49:35.3191062Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3191297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3191382Z return self.weight * hidden_states 2025-08-14T21:49:35.3191388Z 2025-08-14T21:49:35.3191489Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3191686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3191759Z return mod(**inputs) 2025-08-14T21:49:35.3191994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3192074Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3192311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3192381Z layer_outputs = layer_module( 2025-08-14T21:49:35.3192608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3192703Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3192941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3193030Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3193260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3193350Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3193581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.3193657Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.3193661Z 2025-08-14T21:49:35.3193767Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3193963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3194030Z return mod(**inputs) 2025-08-14T21:49:35.3194290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3195176Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3195432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3195503Z layer_outputs = layer_module( 2025-08-14T21:49:35.3195714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3195799Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3196025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3196110Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3196361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3196446Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3196685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.3196761Z key_states = self.k(current_states) 2025-08-14T21:49:35.3196768Z 2025-08-14T21:49:35.3196866Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3197066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3197131Z return mod(**inputs) 2025-08-14T21:49:35.3197368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3197439Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3197673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3197751Z layer_outputs = layer_module( 2025-08-14T21:49:35.3197962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3198039Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3198274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3198349Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3198585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3198665Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3198891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.3199025Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.3199046Z 2025-08-14T21:49:35.3199146Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3199345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3199412Z return mod(**inputs) 2025-08-14T21:49:35.3199645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3199723Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3199954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3200022Z layer_outputs = layer_module( 2025-08-14T21:49:35.3200238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3200312Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3200548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3200625Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3200890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3200979Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3201207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3201359Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3201363Z 2025-08-14T21:49:35.3201461Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3201651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3201722Z return mod(**inputs) 2025-08-14T21:49:35.3201968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3202042Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3202285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3202357Z layer_outputs = layer_module( 2025-08-14T21:49:35.3202573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3202646Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3202875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3202961Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3203192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3203279Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3203508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.3203585Z value_states = self.v(current_states) 2025-08-14T21:49:35.3203590Z 2025-08-14T21:49:35.3203695Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3203886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3203949Z return mod(**inputs) 2025-08-14T21:49:35.3204188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3204257Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3204492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3204563Z layer_outputs = layer_module( 2025-08-14T21:49:35.3204779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3204895Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3205138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3205226Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3205717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3205815Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3206077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3206190Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3206194Z 2025-08-14T21:49:35.3206300Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3206519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3206589Z return mod(**inputs) 2025-08-14T21:49:35.3206828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3206941Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3207175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3207254Z layer_outputs = layer_module( 2025-08-14T21:49:35.3207465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3207540Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3207779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3207859Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3208126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3208212Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3208443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3208552Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3208556Z 2025-08-14T21:49:35.3208654Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3208855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3208918Z return mod(**inputs) 2025-08-14T21:49:35.3209149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3209227Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3209456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3209528Z layer_outputs = layer_module( 2025-08-14T21:49:35.3209746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3209823Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3210059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3210136Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3210369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3210457Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3210688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.3210790Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.3210802Z 2025-08-14T21:49:35.3210921Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3211112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3211183Z return mod(**inputs) 2025-08-14T21:49:35.3211414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3211484Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3211737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3211804Z layer_outputs = layer_module( 2025-08-14T21:49:35.3212014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3212088Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3212309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3212394Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3212614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3212724Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3212962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.3213034Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.3213037Z 2025-08-14T21:49:35.3213119Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3213216Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3213402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3213472Z return mod(**inputs) 2025-08-14T21:49:35.3213701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3213787Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3214019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3214088Z layer_outputs = layer_module( 2025-08-14T21:49:35.3214299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3214370Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3214588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3214679Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3214904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:49:35.3215003Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3215231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3215307Z return self.weight * hidden_states 2025-08-14T21:49:35.3215310Z 2025-08-14T21:49:35.3215428Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3215611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3215673Z return mod(**inputs) 2025-08-14T21:49:35.3215906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3215975Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3216208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3216275Z layer_outputs = layer_module( 2025-08-14T21:49:35.3216483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3216581Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3216808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3216901Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3217122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3217232Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3217461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:49:35.3217554Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:49:35.3217558Z 2025-08-14T21:49:35.3217653Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3217845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3217910Z return mod(**inputs) 2025-08-14T21:49:35.3218144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3218248Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3218476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3218552Z layer_outputs = layer_module( 2025-08-14T21:49:35.3218757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3218840Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3219062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3219147Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3219393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3219505Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3219729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:49:35.3219813Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:49:35.3219816Z 2025-08-14T21:49:35.3219910Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3220115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3220178Z return mod(**inputs) 2025-08-14T21:49:35.3220403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3220479Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3220700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3220769Z layer_outputs = layer_module( 2025-08-14T21:49:35.3220980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3221054Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3221280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3221364Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3221582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3221696Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3221915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:49:35.3222004Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:49:35.3222026Z 2025-08-14T21:49:35.3222126Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3222310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3222384Z return mod(**inputs) 2025-08-14T21:49:35.3222616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3222686Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3222917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3222986Z layer_outputs = layer_module( 2025-08-14T21:49:35.3223202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3223275Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3223500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3223592Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3223854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3223970Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3224196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:49:35.3224271Z hidden_states = self.wo(hidden_states) 2025-08-14T21:49:35.3224274Z 2025-08-14T21:49:35.3224359Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3224457Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3224652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3224724Z return mod(**inputs) 2025-08-14T21:49:35.3224981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3225064Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3225299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3225370Z layer_outputs = layer_module( 2025-08-14T21:49:35.3225592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3225667Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3225899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3225986Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3226226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:49:35.3226334Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3226556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3226631Z return self.weight * hidden_states 2025-08-14T21:49:35.3226635Z 2025-08-14T21:49:35.3226741Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3226927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3226994Z return mod(**inputs) 2025-08-14T21:49:35.3227219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3227288Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3227520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3227587Z layer_outputs = layer_module( 2025-08-14T21:49:35.3227794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3227895Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3228116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3228201Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3228425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3228502Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3228733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.3228806Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.3228809Z 2025-08-14T21:49:35.3228912Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3229102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3229166Z return mod(**inputs) 2025-08-14T21:49:35.3229417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3229503Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3229731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3229807Z layer_outputs = layer_module( 2025-08-14T21:49:35.3230014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3230094Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3230320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3230396Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3230641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3230724Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3230951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.3231033Z key_states = self.k(current_states) 2025-08-14T21:49:35.3231037Z 2025-08-14T21:49:35.3231137Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3231336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3231400Z return mod(**inputs) 2025-08-14T21:49:35.3231625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3231704Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3231929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3232007Z layer_outputs = layer_module( 2025-08-14T21:49:35.3232213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3232289Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3232528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3232604Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3232828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3232914Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3233139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.3233274Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.3233297Z 2025-08-14T21:49:35.3233397Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3233594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3233668Z return mod(**inputs) 2025-08-14T21:49:35.3233889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3233963Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3234186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3234251Z layer_outputs = layer_module( 2025-08-14T21:49:35.3234463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3234535Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3234775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3234861Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3235140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3235225Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3235453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3235599Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3235602Z 2025-08-14T21:49:35.3235705Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3235893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3235955Z return mod(**inputs) 2025-08-14T21:49:35.3236191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3236278Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3236558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3236629Z layer_outputs = layer_module( 2025-08-14T21:49:35.3236834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3236919Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3237142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3237227Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3237449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3237527Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3238052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.3238136Z value_states = self.v(current_states) 2025-08-14T21:49:35.3238141Z 2025-08-14T21:49:35.3238245Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3238475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3238544Z return mod(**inputs) 2025-08-14T21:49:35.3238802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3238881Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3239138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3239223Z layer_outputs = layer_module( 2025-08-14T21:49:35.3239462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3239606Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3239875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3239962Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3240221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3240305Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3240545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3240656Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3240659Z 2025-08-14T21:49:35.3240757Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3240958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3241025Z return mod(**inputs) 2025-08-14T21:49:35.3241254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3241398Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3241630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3241699Z layer_outputs = layer_module( 2025-08-14T21:49:35.3241918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3241993Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3242229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3242305Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3242554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3242642Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3242875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3242987Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3242990Z 2025-08-14T21:49:35.3243090Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3243283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3243355Z return mod(**inputs) 2025-08-14T21:49:35.3243588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3243658Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3243900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3243971Z layer_outputs = layer_module( 2025-08-14T21:49:35.3244189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3244268Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3244503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3244591Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3244825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3244916Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3245152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.3245259Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.3245262Z 2025-08-14T21:49:35.3245371Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3245663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3245737Z return mod(**inputs) 2025-08-14T21:49:35.3245985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3246059Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3246318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3246392Z layer_outputs = layer_module( 2025-08-14T21:49:35.3246623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3246712Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3246975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3247064Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3247332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3247475Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3247711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.3247785Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.3247789Z 2025-08-14T21:49:35.3247888Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3248098Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3248161Z return mod(**inputs) 2025-08-14T21:49:35.3248388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3248456Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3248700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3248778Z layer_outputs = layer_module( 2025-08-14T21:49:35.3248987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3249061Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3249304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3249378Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3249603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 485, in forward 2025-08-14T21:49:35.3249724Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-14T21:49:35.3249728Z 2025-08-14T21:49:35.3249803Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3249907Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3250089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3250157Z return mod(**inputs) 2025-08-14T21:49:35.3250379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3250449Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3250684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3250750Z layer_outputs = layer_module( 2025-08-14T21:49:35.3250954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3251036Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3251262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3251370Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3251592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-08-14T21:49:35.3251703Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3251926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3251997Z return self.weight * hidden_states 2025-08-14T21:49:35.3252000Z 2025-08-14T21:49:35.3252094Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3252286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3252347Z return mod(**inputs) 2025-08-14T21:49:35.3252575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3252643Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3252863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3252967Z layer_outputs = layer_module( 2025-08-14T21:49:35.3253169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3253248Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3253467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3253540Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3253764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3253842Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3254071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.3254154Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.3254159Z 2025-08-14T21:49:35.3254254Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3254446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3254508Z return mod(**inputs) 2025-08-14T21:49:35.3254736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3254811Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3255038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3255106Z layer_outputs = layer_module( 2025-08-14T21:49:35.3255322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3255396Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3255625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3255703Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3255924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3256011Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3256240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.3256317Z key_states = self.k(current_states) 2025-08-14T21:49:35.3256320Z 2025-08-14T21:49:35.3256413Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3256593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3256659Z return mod(**inputs) 2025-08-14T21:49:35.3256922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3256989Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3257214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3257280Z layer_outputs = layer_module( 2025-08-14T21:49:35.3257486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3257557Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3257771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3257851Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3258064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3258150Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3258364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.3258517Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.3258521Z 2025-08-14T21:49:35.3258624Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3258804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3258866Z return mod(**inputs) 2025-08-14T21:49:35.3259092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3259159Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3259383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3259449Z layer_outputs = layer_module( 2025-08-14T21:49:35.3259667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3259752Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3259985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3260058Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3260281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3260357Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3260579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3260719Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3260723Z 2025-08-14T21:49:35.3260817Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3261008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3261070Z return mod(**inputs) 2025-08-14T21:49:35.3261301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3261367Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3261587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3261665Z layer_outputs = layer_module( 2025-08-14T21:49:35.3261864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3261936Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3262161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3262238Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3262481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3262562Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3262777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.3262857Z value_states = self.v(current_states) 2025-08-14T21:49:35.3262860Z 2025-08-14T21:49:35.3262952Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3263139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3263198Z return mod(**inputs) 2025-08-14T21:49:35.3263417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3263503Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3263722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3263807Z layer_outputs = layer_module( 2025-08-14T21:49:35.3264034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3264104Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3264339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3264416Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3264638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3264725Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3264950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3265068Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3265079Z 2025-08-14T21:49:35.3265175Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3265365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3265434Z return mod(**inputs) 2025-08-14T21:49:35.3265662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3265732Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3265967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3266033Z layer_outputs = layer_module( 2025-08-14T21:49:35.3266248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3266320Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3266547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3266632Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3266857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3266936Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3267168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3267266Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3267270Z 2025-08-14T21:49:35.3267392Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3267580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3267644Z return mod(**inputs) 2025-08-14T21:49:35.3267879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3267964Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3268203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3268269Z layer_outputs = layer_module( 2025-08-14T21:49:35.3268476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3268552Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3268777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3268852Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3269088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3269164Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3269397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.3269539Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.3269543Z 2025-08-14T21:49:35.3269640Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3269836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3269897Z return mod(**inputs) 2025-08-14T21:49:35.3270128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3270195Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3270419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3270494Z layer_outputs = layer_module( 2025-08-14T21:49:35.3270722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3270796Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3271027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3271103Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3271332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3271408Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3271631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.3271711Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.3271714Z 2025-08-14T21:49:35.3271791Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3271886Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3272081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3272145Z return mod(**inputs) 2025-08-14T21:49:35.3272378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3272448Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3272673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3272748Z layer_outputs = layer_module( 2025-08-14T21:49:35.3272953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3273025Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3273301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3273390Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3273643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:49:35.3273736Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3273957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3274039Z return self.weight * hidden_states 2025-08-14T21:49:35.3274042Z 2025-08-14T21:49:35.3274139Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3274336Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3274398Z return mod(**inputs) 2025-08-14T21:49:35.3274630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3274710Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3274946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3275033Z layer_outputs = layer_module( 2025-08-14T21:49:35.3275268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3275345Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3275581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3275672Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3275902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3276024Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3276249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:49:35.3276384Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:49:35.3276388Z 2025-08-14T21:49:35.3276487Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3276677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3276747Z return mod(**inputs) 2025-08-14T21:49:35.3276972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3277041Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3277273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3277341Z layer_outputs = layer_module( 2025-08-14T21:49:35.3277554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3277627Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3277853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3277945Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3278169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3278284Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3278504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:49:35.3278578Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:49:35.3278581Z 2025-08-14T21:49:35.3278684Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3278871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3278934Z return mod(**inputs) 2025-08-14T21:49:35.3279195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3279267Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3279505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3279572Z layer_outputs = layer_module( 2025-08-14T21:49:35.3279781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3279861Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3280087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3280171Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3280406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3280515Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3280747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:49:35.3280864Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:49:35.3280868Z 2025-08-14T21:49:35.3280966Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3281165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3281228Z return mod(**inputs) 2025-08-14T21:49:35.3281462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3281533Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3281756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3281832Z layer_outputs = layer_module( 2025-08-14T21:49:35.3282055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3282131Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3282362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3282445Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3282677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3282782Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3283002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:49:35.3283084Z hidden_states = self.wo(hidden_states) 2025-08-14T21:49:35.3283087Z 2025-08-14T21:49:35.3283162Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3283269Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3283460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3283526Z return mod(**inputs) 2025-08-14T21:49:35.3283763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3283833Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3284064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3284140Z layer_outputs = layer_module( 2025-08-14T21:49:35.3284350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3284433Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3284660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3284759Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3284995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:49:35.3285097Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3285324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3285496Z return self.weight * hidden_states 2025-08-14T21:49:35.3285503Z 2025-08-14T21:49:35.3285646Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3285859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3285926Z return mod(**inputs) 2025-08-14T21:49:35.3286168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3286253Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3286494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3286626Z layer_outputs = layer_module( 2025-08-14T21:49:35.3286849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3286936Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3287175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3287255Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3287485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3287573Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3287819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.3287906Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.3287911Z 2025-08-14T21:49:35.3288011Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3288205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3288277Z return mod(**inputs) 2025-08-14T21:49:35.3288509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3288588Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3288818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3288888Z layer_outputs = layer_module( 2025-08-14T21:49:35.3289107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3289184Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3289413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3289500Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3289740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3289860Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3290088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.3290164Z key_states = self.k(current_states) 2025-08-14T21:49:35.3290168Z 2025-08-14T21:49:35.3290273Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3290462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3290527Z return mod(**inputs) 2025-08-14T21:49:35.3290768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3290868Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3291107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3291178Z layer_outputs = layer_module( 2025-08-14T21:49:35.3291387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3291470Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3291697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3291783Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3292009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3292091Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3292327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.3292488Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.3292493Z 2025-08-14T21:49:35.3292594Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3292789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3292853Z return mod(**inputs) 2025-08-14T21:49:35.3293088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3293158Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3293386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3293464Z layer_outputs = layer_module( 2025-08-14T21:49:35.3293694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3293774Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3294011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3294089Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3294319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3294399Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3294623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3294779Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3294782Z 2025-08-14T21:49:35.3294880Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3295078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3295145Z return mod(**inputs) 2025-08-14T21:49:35.3295376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3295456Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3295686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3295754Z layer_outputs = layer_module( 2025-08-14T21:49:35.3295972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3296047Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3296283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3296362Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3296611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3296700Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3296926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.3297010Z value_states = self.v(current_states) 2025-08-14T21:49:35.3297013Z 2025-08-14T21:49:35.3297110Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3297301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3297373Z return mod(**inputs) 2025-08-14T21:49:35.3297605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3297676Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3297918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3298008Z layer_outputs = layer_module( 2025-08-14T21:49:35.3298241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3298317Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3298551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3298637Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3298850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3298932Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3299145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3299260Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3299266Z 2025-08-14T21:49:35.3299367Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3299552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3299613Z return mod(**inputs) 2025-08-14T21:49:35.3299839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3299927Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3300156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3300222Z layer_outputs = layer_module( 2025-08-14T21:49:35.3300422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3300505Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3300723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3300797Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3301027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3301104Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3301335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3301435Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3301438Z 2025-08-14T21:49:35.3301534Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3301729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3301793Z return mod(**inputs) 2025-08-14T21:49:35.3302027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3302115Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3302344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3302421Z layer_outputs = layer_module( 2025-08-14T21:49:35.3302625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3302698Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3302937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3303011Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3303236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3303311Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3303531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.3303654Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.3303673Z 2025-08-14T21:49:35.3303769Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3303957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3304019Z return mod(**inputs) 2025-08-14T21:49:35.3304237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3304311Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3304535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3304601Z layer_outputs = layer_module( 2025-08-14T21:49:35.3304829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3304904Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3305134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3305209Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3305430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3305515Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3305736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.3305810Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.3305821Z 2025-08-14T21:49:35.3305895Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3305990Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3306184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3306257Z return mod(**inputs) 2025-08-14T21:49:35.3306476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3306552Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3306768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3306843Z layer_outputs = layer_module( 2025-08-14T21:49:35.3307041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3307111Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3307333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3307407Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3307643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-08-14T21:49:35.3307753Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3307972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3308050Z return self.weight * hidden_states 2025-08-14T21:49:35.3308053Z 2025-08-14T21:49:35.3308147Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3308329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3308398Z return mod(**inputs) 2025-08-14T21:49:35.3308620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3308688Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3308919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3308986Z layer_outputs = layer_module( 2025-08-14T21:49:35.3309230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3309301Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3309524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3309606Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3309818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3309901Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3310116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.3310214Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.3310220Z 2025-08-14T21:49:35.3310322Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3310506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3310566Z return mod(**inputs) 2025-08-14T21:49:35.3310795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3310863Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3311091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3311156Z layer_outputs = layer_module( 2025-08-14T21:49:35.3311356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3311436Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3311655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3311736Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3311956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3312036Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3312261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.3312333Z key_states = self.k(current_states) 2025-08-14T21:49:35.3312337Z 2025-08-14T21:49:35.3312430Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3312620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3312681Z return mod(**inputs) 2025-08-14T21:49:35.3312910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3312996Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3313221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3313297Z layer_outputs = layer_module( 2025-08-14T21:49:35.3313497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3313569Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3313797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3313872Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3314102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3314180Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3314402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.3314548Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.3314553Z 2025-08-14T21:49:35.3314666Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3314861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3314925Z return mod(**inputs) 2025-08-14T21:49:35.3315147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3315225Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3315453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3315522Z layer_outputs = layer_module( 2025-08-14T21:49:35.3315748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3315824Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3316060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3316137Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3316359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3316445Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3316669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3316822Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3316826Z 2025-08-14T21:49:35.3316922Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3317111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3317182Z return mod(**inputs) 2025-08-14T21:49:35.3317414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3317484Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3317721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3317789Z layer_outputs = layer_module( 2025-08-14T21:49:35.3318001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3318073Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3318295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3318390Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3318610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3318723Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3318955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.3319027Z value_states = self.v(current_states) 2025-08-14T21:49:35.3319031Z 2025-08-14T21:49:35.3319134Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3319320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3319381Z return mod(**inputs) 2025-08-14T21:49:35.3319615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3319686Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3319920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3319989Z layer_outputs = layer_module( 2025-08-14T21:49:35.3320215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3320312Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3320534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3320611Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3320839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3320918Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3321148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3321250Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3321255Z 2025-08-14T21:49:35.3321371Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3321570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3321636Z return mod(**inputs) 2025-08-14T21:49:35.3321873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3321944Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3322168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3322245Z layer_outputs = layer_module( 2025-08-14T21:49:35.3322448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3322520Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3322749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3322824Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3323052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3323130Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3323349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3323456Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3323460Z 2025-08-14T21:49:35.3323555Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3323750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3323813Z return mod(**inputs) 2025-08-14T21:49:35.3324043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3324138Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3324369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3324439Z layer_outputs = layer_module( 2025-08-14T21:49:35.3324656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3324729Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3324962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3325039Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3325264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3325349Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3325684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.3325794Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.3325832Z 2025-08-14T21:49:35.3325959Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3326171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3326250Z return mod(**inputs) 2025-08-14T21:49:35.3326505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3326588Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3326836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3326908Z layer_outputs = layer_module( 2025-08-14T21:49:35.3327151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3327232Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3327478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3327563Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3327796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3327877Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3328116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.3328191Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.3328195Z 2025-08-14T21:49:35.3328300Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3328494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3328561Z return mod(**inputs) 2025-08-14T21:49:35.3328803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3328879Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3329117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3329194Z layer_outputs = layer_module( 2025-08-14T21:49:35.3329412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3329497Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3329734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3329812Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3330054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 524, in forward 2025-08-14T21:49:35.3330198Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-08-14T21:49:35.3330204Z 2025-08-14T21:49:35.3330291Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3330390Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3330582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3330656Z return mod(**inputs) 2025-08-14T21:49:35.3330889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3330961Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3331203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3331273Z layer_outputs = layer_module( 2025-08-14T21:49:35.3331493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3331568Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3331850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3331949Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3332181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:49:35.3332283Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3332516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3332591Z return self.weight * hidden_states 2025-08-14T21:49:35.3332594Z 2025-08-14T21:49:35.3332700Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3332893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3332977Z return mod(**inputs) 2025-08-14T21:49:35.3333219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3333292Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3333531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3333600Z layer_outputs = layer_module( 2025-08-14T21:49:35.3333812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3333892Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3334125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3334211Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3334450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3334563Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3334806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:49:35.3334905Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:49:35.3334909Z 2025-08-14T21:49:35.3335010Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3335215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3335280Z return mod(**inputs) 2025-08-14T21:49:35.3335526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3335600Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3335844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3335943Z layer_outputs = layer_module( 2025-08-14T21:49:35.3336162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3336241Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3336491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3336575Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3336811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3336922Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3337148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:49:35.3337235Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:49:35.3337239Z 2025-08-14T21:49:35.3337341Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3337541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3337836Z return mod(**inputs) 2025-08-14T21:49:35.3338129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3338211Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3338453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3338523Z layer_outputs = layer_module( 2025-08-14T21:49:35.3338754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3338841Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3339111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3339202Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3339432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3339552Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3339777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:49:35.3339862Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:49:35.3339874Z 2025-08-14T21:49:35.3339976Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3340165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3340235Z return mod(**inputs) 2025-08-14T21:49:35.3340465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3340538Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3340780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3340848Z layer_outputs = layer_module( 2025-08-14T21:49:35.3341065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3341140Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3341365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3341458Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3341683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3341791Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3342063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:49:35.3342144Z hidden_states = self.wo(hidden_states) 2025-08-14T21:49:35.3342147Z 2025-08-14T21:49:35.3342236Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3342335Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3342527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3342598Z return mod(**inputs) 2025-08-14T21:49:35.3342832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3342912Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3343145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3343214Z layer_outputs = layer_module( 2025-08-14T21:49:35.3343435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3343513Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3343785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3343874Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3344104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:49:35.3344214Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3344441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3344516Z return self.weight * hidden_states 2025-08-14T21:49:35.3344520Z 2025-08-14T21:49:35.3344626Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3344864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3344933Z return mod(**inputs) 2025-08-14T21:49:35.3345178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3345250Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3345491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3345563Z layer_outputs = layer_module( 2025-08-14T21:49:35.3345774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3345858Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3346084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3346171Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3346401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3346483Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3346721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.3346797Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.3346800Z 2025-08-14T21:49:35.3346898Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3347119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3347209Z return mod(**inputs) 2025-08-14T21:49:35.3347537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3347607Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3347828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3347919Z layer_outputs = layer_module( 2025-08-14T21:49:35.3348125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3348202Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3348421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3348494Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3348718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3348795Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3349011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.3349089Z key_states = self.k(current_states) 2025-08-14T21:49:35.3349093Z 2025-08-14T21:49:35.3349191Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3349378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3349474Z return mod(**inputs) 2025-08-14T21:49:35.3349696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3349771Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3349989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3350054Z layer_outputs = layer_module( 2025-08-14T21:49:35.3350262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3350334Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3350572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3350649Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3350865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3350949Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3351166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.3351291Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.3351294Z 2025-08-14T21:49:35.3351385Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3351566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3351632Z return mod(**inputs) 2025-08-14T21:49:35.3351848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3351918Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3352141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3352210Z layer_outputs = layer_module( 2025-08-14T21:49:35.3352416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3352486Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3352699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3352780Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3352994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3353067Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3353288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3353447Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3353451Z 2025-08-14T21:49:35.3353552Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3353733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3353793Z return mod(**inputs) 2025-08-14T21:49:35.3354019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3354086Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3354311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3354376Z layer_outputs = layer_module( 2025-08-14T21:49:35.3354583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3354663Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3354927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3355004Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3355238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3355315Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3355597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.3355671Z value_states = self.v(current_states) 2025-08-14T21:49:35.3355674Z 2025-08-14T21:49:35.3355772Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3355965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3356045Z return mod(**inputs) 2025-08-14T21:49:35.3356281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3356352Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3356578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3356653Z layer_outputs = layer_module( 2025-08-14T21:49:35.3356859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3356930Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3357173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3357248Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3357484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3357561Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3357784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3357921Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3357928Z 2025-08-14T21:49:35.3358053Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3358325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3358391Z return mod(**inputs) 2025-08-14T21:49:35.3358621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3358699Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3358926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3359014Z layer_outputs = layer_module( 2025-08-14T21:49:35.3359228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3359304Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3359534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3359609Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3359832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3359915Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3360137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3360235Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3360247Z 2025-08-14T21:49:35.3360345Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3360529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3360615Z return mod(**inputs) 2025-08-14T21:49:35.3360858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3360928Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3361162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3361229Z layer_outputs = layer_module( 2025-08-14T21:49:35.3361444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3361517Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3361741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3361851Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3362070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3362147Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3362374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.3362471Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.3362475Z 2025-08-14T21:49:35.3362576Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3362757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3362818Z return mod(**inputs) 2025-08-14T21:49:35.3363045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3363117Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3363344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3363411Z layer_outputs = layer_module( 2025-08-14T21:49:35.3363614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3363692Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3363908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3363981Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3364205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3364281Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3364506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.3364596Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.3364601Z 2025-08-14T21:49:35.3364676Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3364785Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3364977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3365040Z return mod(**inputs) 2025-08-14T21:49:35.3365288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3365359Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3365665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3365739Z layer_outputs = layer_module( 2025-08-14T21:49:35.3365953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3366051Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3366325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3366437Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3366687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-08-14T21:49:35.3366786Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3367013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3367086Z return self.weight * hidden_states 2025-08-14T21:49:35.3367089Z 2025-08-14T21:49:35.3367186Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3367382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3367465Z return mod(**inputs) 2025-08-14T21:49:35.3367703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3367775Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3368008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3368081Z layer_outputs = layer_module( 2025-08-14T21:49:35.3368285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3368355Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3368583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3368659Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3368883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3368961Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3369183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.3369263Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.3369266Z 2025-08-14T21:49:35.3369360Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3369550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3369610Z return mod(**inputs) 2025-08-14T21:49:35.3369830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3369907Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3370124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3370210Z layer_outputs = layer_module( 2025-08-14T21:49:35.3370422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3370495Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3370725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3370799Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3371019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3371104Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3371324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.3371402Z key_states = self.k(current_states) 2025-08-14T21:49:35.3371406Z 2025-08-14T21:49:35.3371501Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3371687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3371772Z return mod(**inputs) 2025-08-14T21:49:35.3372008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3372078Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3372306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3372373Z layer_outputs = layer_module( 2025-08-14T21:49:35.3372581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3372650Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3372867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3372968Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3373187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3373266Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3373488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.3373606Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.3373609Z 2025-08-14T21:49:35.3373710Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3373891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3373951Z return mod(**inputs) 2025-08-14T21:49:35.3374197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3374263Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3374498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3374565Z layer_outputs = layer_module( 2025-08-14T21:49:35.3374770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3374848Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3375073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3375147Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3375377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3375454Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3375681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3375854Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3375859Z 2025-08-14T21:49:35.3375955Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3376147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3376218Z return mod(**inputs) 2025-08-14T21:49:35.3376446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3376513Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3376732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3376805Z layer_outputs = layer_module( 2025-08-14T21:49:35.3377006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3377079Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3377304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3377411Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3377639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3377716Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3377933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.3378013Z value_states = self.v(current_states) 2025-08-14T21:49:35.3378016Z 2025-08-14T21:49:35.3378108Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3378297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3378358Z return mod(**inputs) 2025-08-14T21:49:35.3378602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3378680Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3378906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3378971Z layer_outputs = layer_module( 2025-08-14T21:49:35.3379183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3379253Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3379480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3379553Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3379771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3379856Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3380075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3380175Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3380186Z 2025-08-14T21:49:35.3380278Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3380458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3380524Z return mod(**inputs) 2025-08-14T21:49:35.3380744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3380812Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3381037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3381102Z layer_outputs = layer_module( 2025-08-14T21:49:35.3381328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3381400Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3381616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3381699Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3381916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3381990Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3382237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3382334Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3382338Z 2025-08-14T21:49:35.3382441Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3382632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3382713Z return mod(**inputs) 2025-08-14T21:49:35.3382969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3383038Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3383257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3383330Z layer_outputs = layer_module( 2025-08-14T21:49:35.3383529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3383608Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3383827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3383914Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3384142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3384220Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3384495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.3384594Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.3384597Z 2025-08-14T21:49:35.3384691Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3384886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3384949Z return mod(**inputs) 2025-08-14T21:49:35.3385173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3385251Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3385477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3385552Z layer_outputs = layer_module( 2025-08-14T21:49:35.3385757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3385830Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3386062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3386137Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3386374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3386449Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3386663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.3386758Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.3386762Z 2025-08-14T21:49:35.3386837Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3386932Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3387122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3387184Z return mod(**inputs) 2025-08-14T21:49:35.3387415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3387482Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3387702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3387775Z layer_outputs = layer_module( 2025-08-14T21:49:35.3387979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3388050Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3388279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3388393Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3388620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:49:35.3388708Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3388924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3389000Z return self.weight * hidden_states 2025-08-14T21:49:35.3389003Z 2025-08-14T21:49:35.3389096Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3389285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3389345Z return mod(**inputs) 2025-08-14T21:49:35.3389583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3389660Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3389881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3389947Z layer_outputs = layer_module( 2025-08-14T21:49:35.3390158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3390229Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3390463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3390548Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3390770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3390892Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3391117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:49:35.3391220Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:49:35.3391224Z 2025-08-14T21:49:35.3391319Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3391505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3391575Z return mod(**inputs) 2025-08-14T21:49:35.3391800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3391881Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3392111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3392177Z layer_outputs = layer_module( 2025-08-14T21:49:35.3392407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3392479Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3392696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3392783Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3392997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3393102Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3393325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:49:35.3393399Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:49:35.3393402Z 2025-08-14T21:49:35.3393505Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3393689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3393770Z return mod(**inputs) 2025-08-14T21:49:35.3394018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3394089Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3394326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3394392Z layer_outputs = layer_module( 2025-08-14T21:49:35.3394603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3394685Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3394912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3395022Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3395257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3395366Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3395594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:49:35.3395678Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:49:35.3395681Z 2025-08-14T21:49:35.3395777Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3395970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3396033Z return mod(**inputs) 2025-08-14T21:49:35.3396268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3396337Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3396567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3396647Z layer_outputs = layer_module( 2025-08-14T21:49:35.3396854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3396930Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3397162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3397245Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3397481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3397587Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3397820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:49:35.3397922Z hidden_states = self.wo(hidden_states) 2025-08-14T21:49:35.3397927Z 2025-08-14T21:49:35.3398024Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3398223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3398287Z return mod(**inputs) 2025-08-14T21:49:35.3398512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3398591Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3398817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3398886Z layer_outputs = layer_module( 2025-08-14T21:49:35.3399100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3399175Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3399407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3399525Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3399750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 217, in forward 2025-08-14T21:49:35.3399880Z hidden_states = hidden_states + self.dropout(forwarded_states) 2025-08-14T21:49:35.3399884Z 2025-08-14T21:49:35.3399959Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3400056Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3400251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3400312Z return mod(**inputs) 2025-08-14T21:49:35.3400546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3400637Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3400864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3400943Z layer_outputs = layer_module( 2025-08-14T21:49:35.3401149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3401230Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3401451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3401527Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3401799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:49:35.3401897Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3402120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3402203Z return self.weight * hidden_states 2025-08-14T21:49:35.3402208Z 2025-08-14T21:49:35.3402304Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3402496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3402558Z return mod(**inputs) 2025-08-14T21:49:35.3402783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3402862Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3403086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3403152Z layer_outputs = layer_module( 2025-08-14T21:49:35.3403363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3403457Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3403686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3403765Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3403986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3404075Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3404294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.3404376Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.3404379Z 2025-08-14T21:49:35.3404475Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3404659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3404730Z return mod(**inputs) 2025-08-14T21:49:35.3404961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3405055Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3405313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3405383Z layer_outputs = layer_module( 2025-08-14T21:49:35.3405687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3405767Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3405997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3406085Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3406314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3406427Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3406666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.3406742Z key_states = self.k(current_states) 2025-08-14T21:49:35.3406746Z 2025-08-14T21:49:35.3406849Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3407034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3407096Z return mod(**inputs) 2025-08-14T21:49:35.3407327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3407397Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3407628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3407695Z layer_outputs = layer_module( 2025-08-14T21:49:35.3407904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3407987Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3408208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3408283Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3408508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3408594Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3408815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.3408933Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.3408936Z 2025-08-14T21:49:35.3409031Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3409236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3409298Z return mod(**inputs) 2025-08-14T21:49:35.3409526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3409593Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3409812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3409885Z layer_outputs = layer_module( 2025-08-14T21:49:35.3410081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3410152Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3410375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3410450Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3410673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3410783Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3411000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3411151Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3411154Z 2025-08-14T21:49:35.3411246Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3411434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3411496Z return mod(**inputs) 2025-08-14T21:49:35.3411714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3411788Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3412021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3412091Z layer_outputs = layer_module( 2025-08-14T21:49:35.3412305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3412378Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3412602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3412677Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3412895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3412980Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3413195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.3413271Z value_states = self.v(current_states) 2025-08-14T21:49:35.3413282Z 2025-08-14T21:49:35.3413383Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3413565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3413632Z return mod(**inputs) 2025-08-14T21:49:35.3413855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3413922Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3414150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3414215Z layer_outputs = layer_module( 2025-08-14T21:49:35.3414424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3414496Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3414738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3414825Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3415048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3415125Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3415355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3415456Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3415459Z 2025-08-14T21:49:35.3415561Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3415748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3415810Z return mod(**inputs) 2025-08-14T21:49:35.3416044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3416114Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3416663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3416732Z layer_outputs = layer_module( 2025-08-14T21:49:35.3416934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3417016Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3417232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3417307Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3417535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3417626Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3417852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3417953Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3417956Z 2025-08-14T21:49:35.3418050Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3418246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3418309Z return mod(**inputs) 2025-08-14T21:49:35.3418532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3418610Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3418834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3418908Z layer_outputs = layer_module( 2025-08-14T21:49:35.3419116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3419190Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3419420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3419494Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3419723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3419801Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3420022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.3420129Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.3420133Z 2025-08-14T21:49:35.3420228Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3420421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3420505Z return mod(**inputs) 2025-08-14T21:49:35.3420728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3420805Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3421025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3421092Z layer_outputs = layer_module( 2025-08-14T21:49:35.3421300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3421372Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3421599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3421676Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3421902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3422005Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3422251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.3422326Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.3422329Z 2025-08-14T21:49:35.3422413Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3422509Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3422704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3422765Z return mod(**inputs) 2025-08-14T21:49:35.3422992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3423068Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3423315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3423385Z layer_outputs = layer_module( 2025-08-14T21:49:35.3423600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3423674Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3423904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3423979Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3424220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-08-14T21:49:35.3424329Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3424562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3424647Z return self.weight * hidden_states 2025-08-14T21:49:35.3424650Z 2025-08-14T21:49:35.3424750Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3424945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3425019Z return mod(**inputs) 2025-08-14T21:49:35.3425253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3425323Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3425563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3425632Z layer_outputs = layer_module( 2025-08-14T21:49:35.3425854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3425929Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3426180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3426268Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3426504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3426584Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3426817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.3426891Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.3426895Z 2025-08-14T21:49:35.3426999Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3427189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3427253Z return mod(**inputs) 2025-08-14T21:49:35.3427490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3427562Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3427835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3427905Z layer_outputs = layer_module( 2025-08-14T21:49:35.3428114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3428196Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3428420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3428496Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3428740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3428819Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3429084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.3429160Z key_states = self.k(current_states) 2025-08-14T21:49:35.3429165Z 2025-08-14T21:49:35.3429260Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3429456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3429519Z return mod(**inputs) 2025-08-14T21:49:35.3429751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3429822Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3430049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3430125Z layer_outputs = layer_module( 2025-08-14T21:49:35.3430335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3430409Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3430641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3430715Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3430945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3431024Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3431246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.3431375Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.3431378Z 2025-08-14T21:49:35.3431475Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3431671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3431751Z return mod(**inputs) 2025-08-14T21:49:35.3431987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3432068Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3432299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3432371Z layer_outputs = layer_module( 2025-08-14T21:49:35.3432590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3432663Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3432899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3432975Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3433207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3433311Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3433576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3433725Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3433735Z 2025-08-14T21:49:35.3433831Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3434017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3434085Z return mod(**inputs) 2025-08-14T21:49:35.3434310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3434379Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3434627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3434700Z layer_outputs = layer_module( 2025-08-14T21:49:35.3434921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3434994Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3435224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3435309Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3435541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3435620Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3435862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.3435938Z value_states = self.v(current_states) 2025-08-14T21:49:35.3435944Z 2025-08-14T21:49:35.3436049Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3436242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3436306Z return mod(**inputs) 2025-08-14T21:49:35.3436557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3436629Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3436850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3436923Z layer_outputs = layer_module( 2025-08-14T21:49:35.3437127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3437206Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3437431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3437526Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3438076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3438178Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3438446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3438560Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3438564Z 2025-08-14T21:49:35.3438671Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3438890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3438959Z return mod(**inputs) 2025-08-14T21:49:35.3439212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3439301Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3439611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3439724Z layer_outputs = layer_module( 2025-08-14T21:49:35.3439954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3440037Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3440301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3440379Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3440613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3440694Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3440971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3441095Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3441099Z 2025-08-14T21:49:35.3441206Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3441413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3441492Z return mod(**inputs) 2025-08-14T21:49:35.3441742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3441831Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3442083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3442159Z layer_outputs = layer_module( 2025-08-14T21:49:35.3442400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3442484Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3442735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3442828Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3443073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3443168Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3443414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.3443524Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.3443528Z 2025-08-14T21:49:35.3443643Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3443847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3443955Z return mod(**inputs) 2025-08-14T21:49:35.3444210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3444291Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3444551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3444626Z layer_outputs = layer_module( 2025-08-14T21:49:35.3444857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3444944Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3445192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3445281Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3445606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3445700Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3445997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.3446081Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.3446085Z 2025-08-14T21:49:35.3446177Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3446285Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3446492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3446570Z return mod(**inputs) 2025-08-14T21:49:35.3446824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3446900Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3447174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3447255Z layer_outputs = layer_module( 2025-08-14T21:49:35.3447501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3447582Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3447831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3447935Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3448182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:49:35.3448283Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3448540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3448621Z return self.weight * hidden_states 2025-08-14T21:49:35.3448627Z 2025-08-14T21:49:35.3448742Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3448956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3449025Z return mod(**inputs) 2025-08-14T21:49:35.3449284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3449362Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3449620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3449695Z layer_outputs = layer_module( 2025-08-14T21:49:35.3449921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3449997Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3450215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3464853Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3465201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3465331Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3465570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:49:35.3465674Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:49:35.3465682Z 2025-08-14T21:49:35.3465785Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3465993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3466057Z return mod(**inputs) 2025-08-14T21:49:35.3466288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3466362Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3466708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3466782Z layer_outputs = layer_module( 2025-08-14T21:49:35.3466990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3467072Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3467296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3467389Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3467617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3467726Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3467976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:49:35.3468059Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:49:35.3468064Z 2025-08-14T21:49:35.3468162Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3468349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3468417Z return mod(**inputs) 2025-08-14T21:49:35.3468641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3468718Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3468957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3469022Z layer_outputs = layer_module( 2025-08-14T21:49:35.3469232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3469319Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3469547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3469641Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3469856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3469958Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3470179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:49:35.3470264Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:49:35.3470268Z 2025-08-14T21:49:35.3470366Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3470554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3470648Z return mod(**inputs) 2025-08-14T21:49:35.3470890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3470961Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3471191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3471270Z layer_outputs = layer_module( 2025-08-14T21:49:35.3471478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3471553Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3471783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3471868Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3472097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3472222Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3472461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:49:35.3472546Z hidden_states = self.wo(hidden_states) 2025-08-14T21:49:35.3472550Z 2025-08-14T21:49:35.3472628Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3472732Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3472920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3472984Z return mod(**inputs) 2025-08-14T21:49:35.3473212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3473281Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3473522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3473599Z layer_outputs = layer_module( 2025-08-14T21:49:35.3473810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3473888Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3474120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3474199Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3474420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:49:35.3474527Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3474747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3474834Z return self.weight * hidden_states 2025-08-14T21:49:35.3474838Z 2025-08-14T21:49:35.3474934Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3475126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3475197Z return mod(**inputs) 2025-08-14T21:49:35.3475423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3475494Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3475725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3475792Z layer_outputs = layer_module( 2025-08-14T21:49:35.3476004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3476080Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3476332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3476419Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3476641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3476721Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3476950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.3477021Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.3477025Z 2025-08-14T21:49:35.3477129Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3477317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3477378Z return mod(**inputs) 2025-08-14T21:49:35.3477616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3477687Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3477949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3478019Z layer_outputs = layer_module( 2025-08-14T21:49:35.3478234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3478313Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3478529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3478605Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3478826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3478903Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3479142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.3479217Z key_states = self.k(current_states) 2025-08-14T21:49:35.3479222Z 2025-08-14T21:49:35.3479316Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3479504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3479566Z return mod(**inputs) 2025-08-14T21:49:35.3479792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3479858Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3480076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3480148Z layer_outputs = layer_module( 2025-08-14T21:49:35.3480350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3480422Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3480649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3480722Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3480946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3481018Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3481268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.3481398Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.3481402Z 2025-08-14T21:49:35.3481497Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3481689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3481769Z return mod(**inputs) 2025-08-14T21:49:35.3481999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3482077Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3482304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3482371Z layer_outputs = layer_module( 2025-08-14T21:49:35.3482583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3482655Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3482886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3482963Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3483189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3483274Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3483533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3483684Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3483695Z 2025-08-14T21:49:35.3483792Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3483979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3484050Z return mod(**inputs) 2025-08-14T21:49:35.3484279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3484352Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3484611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3484686Z layer_outputs = layer_module( 2025-08-14T21:49:35.3484910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3484987Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3485214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3485301Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3485665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3485753Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3485996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.3486075Z value_states = self.v(current_states) 2025-08-14T21:49:35.3486083Z 2025-08-14T21:49:35.3486194Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3486395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3486468Z return mod(**inputs) 2025-08-14T21:49:35.3486720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3486796Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3487032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3487117Z layer_outputs = layer_module( 2025-08-14T21:49:35.3487346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3487430Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3487649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3487750Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3487982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3488058Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3488284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3488385Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3488389Z 2025-08-14T21:49:35.3488484Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3488673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3488734Z return mod(**inputs) 2025-08-14T21:49:35.3488952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3489028Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3489247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3489361Z layer_outputs = layer_module( 2025-08-14T21:49:35.3489564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3489635Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3489859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3489933Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3490156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3490231Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3490465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3490575Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3490580Z 2025-08-14T21:49:35.3490677Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3490865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3490936Z return mod(**inputs) 2025-08-14T21:49:35.3491164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3491241Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3491466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3491534Z layer_outputs = layer_module( 2025-08-14T21:49:35.3491751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3491830Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3492053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3492139Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3492360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3492455Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3492673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.3492771Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.3492775Z 2025-08-14T21:49:35.3492875Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3493056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3493146Z return mod(**inputs) 2025-08-14T21:49:35.3493368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3493440Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3493668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3493733Z layer_outputs = layer_module( 2025-08-14T21:49:35.3493932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3494010Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3494225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3494308Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3494522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3494597Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3494862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.3494937Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.3494941Z 2025-08-14T21:49:35.3495043Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3495230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3495293Z return mod(**inputs) 2025-08-14T21:49:35.3495533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3495603Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3495838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3495943Z layer_outputs = layer_module( 2025-08-14T21:49:35.3496155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3496240Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3496478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3496555Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3496785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 485, in forward 2025-08-14T21:49:35.3496910Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-14T21:49:35.3496914Z 2025-08-14T21:49:35.3496999Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3497095Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3497282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3497356Z return mod(**inputs) 2025-08-14T21:49:35.3497581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3497655Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3497895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3497962Z layer_outputs = layer_module( 2025-08-14T21:49:35.3498175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3498250Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3498478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3498563Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3498792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-08-14T21:49:35.3498913Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3499155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3499231Z return self.weight * hidden_states 2025-08-14T21:49:35.3499235Z 2025-08-14T21:49:35.3499340Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3499532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3499598Z return mod(**inputs) 2025-08-14T21:49:35.3499843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3499915Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3500162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3500234Z layer_outputs = layer_module( 2025-08-14T21:49:35.3500439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3500556Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3500785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3500860Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3501091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3501170Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3501402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.3501475Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.3501479Z 2025-08-14T21:49:35.3501591Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3501789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3501854Z return mod(**inputs) 2025-08-14T21:49:35.3502080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3502158Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3502385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3502463Z layer_outputs = layer_module( 2025-08-14T21:49:35.3502669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3502741Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3502969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3503047Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3503275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3503357Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3503580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.3503660Z key_states = self.k(current_states) 2025-08-14T21:49:35.3503663Z 2025-08-14T21:49:35.3503759Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3503943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3504012Z return mod(**inputs) 2025-08-14T21:49:35.3504239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3504317Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3504561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3504631Z layer_outputs = layer_module( 2025-08-14T21:49:35.3504851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3504925Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3505151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3505238Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3505470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3505559Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3505795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.3505926Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.3505947Z 2025-08-14T21:49:35.3506059Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3506271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3506346Z return mod(**inputs) 2025-08-14T21:49:35.3506607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3506680Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3506919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3506988Z layer_outputs = layer_module( 2025-08-14T21:49:35.3507198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3507299Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3507535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3507625Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3507860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3507938Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3508166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3508313Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3508316Z 2025-08-14T21:49:35.3508415Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3508603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3508668Z return mod(**inputs) 2025-08-14T21:49:35.3508902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3508973Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3509197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3509274Z layer_outputs = layer_module( 2025-08-14T21:49:35.3509482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3509560Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3509782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3509858Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3510089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3510182Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3510411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.3510487Z value_states = self.v(current_states) 2025-08-14T21:49:35.3510490Z 2025-08-14T21:49:35.3510584Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3510776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3510838Z return mod(**inputs) 2025-08-14T21:49:35.3511060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3511137Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3511413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3511490Z layer_outputs = layer_module( 2025-08-14T21:49:35.3511698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3511787Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3512042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3512127Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3512350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3512435Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3512657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3512764Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3512768Z 2025-08-14T21:49:35.3512863Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3513066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3513140Z return mod(**inputs) 2025-08-14T21:49:35.3513367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3513435Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3513665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3513740Z layer_outputs = layer_module( 2025-08-14T21:49:35.3513954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3514024Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3514247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3514335Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3514556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3514636Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3514867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3514967Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3514971Z 2025-08-14T21:49:35.3515073Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3515261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3515322Z return mod(**inputs) 2025-08-14T21:49:35.3515554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3515624Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3515876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3515946Z layer_outputs = layer_module( 2025-08-14T21:49:35.3516150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3516231Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3516453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3516528Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3516760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3516837Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3517066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.3517169Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.3517172Z 2025-08-14T21:49:35.3517284Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3517512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3517577Z return mod(**inputs) 2025-08-14T21:49:35.3517814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3517886Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3518111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3518188Z layer_outputs = layer_module( 2025-08-14T21:49:35.3518393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3518467Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3518714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3518793Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3519026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3519104Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3519324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.3519406Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.3519409Z 2025-08-14T21:49:35.3519485Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3519581Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3519774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3519838Z return mod(**inputs) 2025-08-14T21:49:35.3520072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3520142Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3520368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3520444Z layer_outputs = layer_module( 2025-08-14T21:49:35.3520646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3520726Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3520946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3521031Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3521260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:49:35.3521370Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3521592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3521676Z return self.weight * hidden_states 2025-08-14T21:49:35.3521679Z 2025-08-14T21:49:35.3521774Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3521965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3522029Z return mod(**inputs) 2025-08-14T21:49:35.3522264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3522341Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3522558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3522625Z layer_outputs = layer_module( 2025-08-14T21:49:35.3522833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3522919Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3523160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3523246Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3523464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3523581Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3523800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:49:35.3523899Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:49:35.3523902Z 2025-08-14T21:49:35.3524011Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3524199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3524269Z return mod(**inputs) 2025-08-14T21:49:35.3524491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3524560Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3524788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3524854Z layer_outputs = layer_module( 2025-08-14T21:49:35.3525064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3525136Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3525357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3525534Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3525767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3525890Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3526119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:49:35.3526197Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:49:35.3526200Z 2025-08-14T21:49:35.3526306Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3526500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3526568Z return mod(**inputs) 2025-08-14T21:49:35.3526816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3526892Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3527163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3527238Z layer_outputs = layer_module( 2025-08-14T21:49:35.3527457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3527543Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3527781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3527876Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3528114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3528226Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3528468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:49:35.3528559Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:49:35.3528579Z 2025-08-14T21:49:35.3528682Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3528938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3529006Z return mod(**inputs) 2025-08-14T21:49:35.3529252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3529326Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3529563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3529644Z layer_outputs = layer_module( 2025-08-14T21:49:35.3529859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3529955Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3530197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3530286Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3530527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3530636Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3530869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:49:35.3530958Z hidden_states = self.wo(hidden_states) 2025-08-14T21:49:35.3530962Z 2025-08-14T21:49:35.3531042Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3531151Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3531348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3531417Z return mod(**inputs) 2025-08-14T21:49:35.3531669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3531744Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3531977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3532059Z layer_outputs = layer_module( 2025-08-14T21:49:35.3532268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3532351Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3532577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3532655Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3532892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 474, in forward 2025-08-14T21:49:35.3533012Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3533248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3533323Z return self.weight * hidden_states 2025-08-14T21:49:35.3533326Z 2025-08-14T21:49:35.3533423Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3533621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3533684Z return mod(**inputs) 2025-08-14T21:49:35.3533913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3533993Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3534222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3534301Z layer_outputs = layer_module( 2025-08-14T21:49:35.3534509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3534625Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3534862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3534943Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3535176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3535267Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3535499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.3535585Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.3535589Z 2025-08-14T21:49:35.3535713Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3535912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3535987Z return mod(**inputs) 2025-08-14T21:49:35.3536247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3536327Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3536563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3536633Z layer_outputs = layer_module( 2025-08-14T21:49:35.3536857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3536933Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3537166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3537258Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3537490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3537581Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3538220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.3538316Z key_states = self.k(current_states) 2025-08-14T21:49:35.3538320Z 2025-08-14T21:49:35.3538440Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3538651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3538730Z return mod(**inputs) 2025-08-14T21:49:35.3538986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3539066Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3539406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3539484Z layer_outputs = layer_module( 2025-08-14T21:49:35.3539712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3539798Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3540030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3540122Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3540360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3540441Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3540682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.3540815Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.3540893Z 2025-08-14T21:49:35.3540995Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3541221Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3541290Z return mod(**inputs) 2025-08-14T21:49:35.3541537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3541610Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3541850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3541929Z layer_outputs = layer_module( 2025-08-14T21:49:35.3542147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3542262Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3542501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3542586Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3542832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3542913Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3543149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3543311Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3543315Z 2025-08-14T21:49:35.3543418Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3543622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3543687Z return mod(**inputs) 2025-08-14T21:49:35.3543929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3544011Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3544253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3544334Z layer_outputs = layer_module( 2025-08-14T21:49:35.3544553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3544631Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3544873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3544953Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3545186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3545295Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3545531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.3545620Z value_states = self.v(current_states) 2025-08-14T21:49:35.3545623Z 2025-08-14T21:49:35.3545724Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3545920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3545994Z return mod(**inputs) 2025-08-14T21:49:35.3546234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3546345Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3546599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3546671Z layer_outputs = layer_module( 2025-08-14T21:49:35.3546899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3547010Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3547253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3547339Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3547563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3547648Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3547871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3547972Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3547976Z 2025-08-14T21:49:35.3548083Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3548284Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3548350Z return mod(**inputs) 2025-08-14T21:49:35.3548589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3548659Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3548895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3548963Z layer_outputs = layer_module( 2025-08-14T21:49:35.3549171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3549254Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3549480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3549565Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3549788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3549866Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3550098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3550198Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3550202Z 2025-08-14T21:49:35.3550297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3550494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3550556Z return mod(**inputs) 2025-08-14T21:49:35.3550789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3550858Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3551106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3551184Z layer_outputs = layer_module( 2025-08-14T21:49:35.3551392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3551468Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3551702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3551781Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3552013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3552092Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3552316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.3552433Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.3552436Z 2025-08-14T21:49:35.3552534Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3552759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3552824Z return mod(**inputs) 2025-08-14T21:49:35.3553053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3553130Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3553356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3553423Z layer_outputs = layer_module( 2025-08-14T21:49:35.3553639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3553711Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3553959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 559, in forward 2025-08-14T21:49:35.3554039Z self_attention_outputs = self.layer[0]( 2025-08-14T21:49:35.3554264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 475, in forward 2025-08-14T21:49:35.3554347Z attention_output = self.SelfAttention( 2025-08-14T21:49:35.3554568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.3554649Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.3554653Z 2025-08-14T21:49:35.3554731Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3554831Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3555027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3555092Z return mod(**inputs) 2025-08-14T21:49:35.3555326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3555406Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3555638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3555714Z layer_outputs = layer_module( 2025-08-14T21:49:35.3555925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3556001Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3556236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3556316Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3556595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 511, in forward 2025-08-14T21:49:35.3556730Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3556951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3557036Z return self.weight * hidden_states 2025-08-14T21:49:35.3557039Z 2025-08-14T21:49:35.3557136Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3557323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3557396Z return mod(**inputs) 2025-08-14T21:49:35.3557620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3557694Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3557918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3557986Z layer_outputs = layer_module( 2025-08-14T21:49:35.3558199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3558291Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3558531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3558636Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3558859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3558946Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3559167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 365, in forward 2025-08-14T21:49:35.3559239Z query_states = self.q(hidden_states) 2025-08-14T21:49:35.3559243Z 2025-08-14T21:49:35.3559345Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3559547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3559632Z return mod(**inputs) 2025-08-14T21:49:35.3559860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3559936Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3560161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3560227Z layer_outputs = layer_module( 2025-08-14T21:49:35.3560439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3560512Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3560740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3560813Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3561036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3561124Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3561348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 385, in forward 2025-08-14T21:49:35.3561423Z key_states = self.k(current_states) 2025-08-14T21:49:35.3561427Z 2025-08-14T21:49:35.3561533Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3561726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3561795Z return mod(**inputs) 2025-08-14T21:49:35.3562027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3562098Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3562338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3562435Z layer_outputs = layer_module( 2025-08-14T21:49:35.3562653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3562734Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3562964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3563049Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3563281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3563359Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3563602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 401, in forward 2025-08-14T21:49:35.3563728Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:49:35.3563733Z 2025-08-14T21:49:35.3563839Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3564065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3564130Z return mod(**inputs) 2025-08-14T21:49:35.3564371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3564442Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3564672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3564748Z layer_outputs = layer_module( 2025-08-14T21:49:35.3564958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3565039Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3565284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3565367Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3565779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3565872Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3566132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 433, in forward 2025-08-14T21:49:35.3566291Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:49:35.3566295Z 2025-08-14T21:49:35.3566401Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3566622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3566688Z return mod(**inputs) 2025-08-14T21:49:35.3566926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3567010Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3567249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3567328Z layer_outputs = layer_module( 2025-08-14T21:49:35.3567543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3567618Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3567859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3567937Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3568184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3568265Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3568520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 386, in forward 2025-08-14T21:49:35.3568606Z value_states = self.v(current_states) 2025-08-14T21:49:35.3568611Z 2025-08-14T21:49:35.3568708Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3568898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3568970Z return mod(**inputs) 2025-08-14T21:49:35.3569202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3569280Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3569514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3569584Z layer_outputs = layer_module( 2025-08-14T21:49:35.3569807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3569883Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3570157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3570246Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3570484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3570573Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3570808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3570912Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3570916Z 2025-08-14T21:49:35.3571028Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3571239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3571316Z return mod(**inputs) 2025-08-14T21:49:35.3571555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3571626Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3571867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3571937Z layer_outputs = layer_module( 2025-08-14T21:49:35.3572148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3572232Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3572461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3572546Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3572778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3572861Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3573099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 440, in forward 2025-08-14T21:49:35.3573202Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:49:35.3573206Z 2025-08-14T21:49:35.3573311Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3573503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3573566Z return mod(**inputs) 2025-08-14T21:49:35.3573807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3573878Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3574113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3574207Z layer_outputs = layer_module( 2025-08-14T21:49:35.3574422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3574502Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3574729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3574804Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3575036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3575115Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3575342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 442, in forward 2025-08-14T21:49:35.3575452Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:49:35.3575457Z 2025-08-14T21:49:35.3575554Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3575790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3575857Z return mod(**inputs) 2025-08-14T21:49:35.3576097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3576178Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3576426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3576503Z layer_outputs = layer_module( 2025-08-14T21:49:35.3576716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3576789Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3577041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3577124Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3577363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 512, in forward 2025-08-14T21:49:35.3577452Z attention_output = self.EncDecAttention( 2025-08-14T21:49:35.3577686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 444, in forward 2025-08-14T21:49:35.3577782Z attn_output = self.o(attn_output) 2025-08-14T21:49:35.3577785Z 2025-08-14T21:49:35.3577884Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3578077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3578150Z return mod(**inputs) 2025-08-14T21:49:35.3578384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3578468Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3578704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3578776Z layer_outputs = layer_module( 2025-08-14T21:49:35.3578995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3579069Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3579298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 583, in forward 2025-08-14T21:49:35.3579383Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:49:35.3579617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 524, in forward 2025-08-14T21:49:35.3579755Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-08-14T21:49:35.3579774Z 2025-08-14T21:49:35.3579857Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3579957Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3580168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3580232Z return mod(**inputs) 2025-08-14T21:49:35.3580467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3580547Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3580784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3580862Z layer_outputs = layer_module( 2025-08-14T21:49:35.3581078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3581154Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3581397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3581489Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3581768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 215, in forward 2025-08-14T21:49:35.3581865Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:49:35.3582101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3582185Z return self.weight * hidden_states 2025-08-14T21:49:35.3582189Z 2025-08-14T21:49:35.3582287Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3582483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3582555Z return mod(**inputs) 2025-08-14T21:49:35.3582821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3582907Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3583154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3583227Z layer_outputs = layer_module( 2025-08-14T21:49:35.3583454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3583532Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3583772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3583871Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3584112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3584237Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3584487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 183, in forward 2025-08-14T21:49:35.3584595Z hidden_gelu = self.act(self.wi_0(hidden_states)) 2025-08-14T21:49:35.3584600Z 2025-08-14T21:49:35.3584717Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3584927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3585006Z return mod(**inputs) 2025-08-14T21:49:35.3585263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3585341Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3585612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3585684Z layer_outputs = layer_module( 2025-08-14T21:49:35.3585904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3586009Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3586246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3586343Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3586578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3586692Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3586933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 184, in forward 2025-08-14T21:49:35.3587012Z hidden_linear = self.wi_1(hidden_states) 2025-08-14T21:49:35.3587016Z 2025-08-14T21:49:35.3587125Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3587322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3587391Z return mod(**inputs) 2025-08-14T21:49:35.3587676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3587750Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3587989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3588067Z layer_outputs = layer_module( 2025-08-14T21:49:35.3588284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3588370Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3588602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3588697Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3589074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3589242Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3589509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 185, in forward 2025-08-14T21:49:35.3589602Z hidden_states = hidden_gelu * hidden_linear 2025-08-14T21:49:35.3589606Z 2025-08-14T21:49:35.3589707Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3589910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3589976Z return mod(**inputs) 2025-08-14T21:49:35.3590215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3590295Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3590533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1079, in forward 2025-08-14T21:49:35.3590613Z layer_outputs = layer_module( 2025-08-14T21:49:35.3590832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:49:35.3590908Z return super().__call__(*args, **kwargs) 2025-08-14T21:49:35.3591149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 609, in forward 2025-08-14T21:49:35.3591236Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:49:35.3591471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 216, in forward 2025-08-14T21:49:35.3591592Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:49:35.3591824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 198, in forward 2025-08-14T21:49:35.3591913Z hidden_states = self.wo(hidden_states) 2025-08-14T21:49:35.3591935Z 2025-08-14T21:49:35.3592022Z cudagraph partition due to non gpu ops 2025-08-14T21:49:35.3592131Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3592342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3592412Z return mod(**inputs) 2025-08-14T21:49:35.3592663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1787, in forward 2025-08-14T21:49:35.3592739Z decoder_outputs = self.decoder( 2025-08-14T21:49:35.3592979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1115, in forward 2025-08-14T21:49:35.3593094Z hidden_states = self.final_layer_norm(hidden_states) 2025-08-14T21:49:35.3593332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 146, in forward 2025-08-14T21:49:35.3593414Z return self.weight * hidden_states 2025-08-14T21:49:35.3593425Z 2025-08-14T21:49:35.3593529Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3593762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3593845Z return mod(**inputs) 2025-08-14T21:49:35.3594081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1816, in forward 2025-08-14T21:49:35.3594166Z lm_logits = self.lm_head(sequence_output) 2025-08-14T21:49:35.3594169Z 2025-08-14T21:49:35.3594273Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3594462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3594531Z return mod(**inputs) 2025-08-14T21:49:35.3594770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1823, in forward 2025-08-14T21:49:35.3594928Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-08-14T21:49:35.3594933Z 2025-08-14T21:49:35.3595043Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3595244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3595308Z return mod(**inputs) 2025-08-14T21:49:35.3595551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1823, in forward 2025-08-14T21:49:35.3595679Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-08-14T21:49:35.3595683Z 2025-08-14T21:49:35.3595786Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:49:35.3595977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:49:35.3596038Z return mod(**inputs) 2025-08-14T21:49:35.3596287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mt5/modeling_mt5.py", line 1823, in forward 2025-08-14T21:49:35.3596410Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-08-14T21:49:35.3596415Z 2025-08-14T21:49:46.2276991Z Compilation time (from dynamo_timed): 21.781230737 2025-08-14T21:49:46.2511242Z pass 2025-08-14T21:49:46.2511762Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:49:46.2521151Z TIMING: _recursive_pre_grad_passes:0.01537 _recursive_joint_graph_passes:0.728 _recursive_post_grad_passes:0.62689 async_compile.wait:0.7915 code_gen:10.31405 inductor_compile:12.896 backend_compile:17.7461 gc:0.00016 entire_frame_compile:21.78123 total_wall_time:21.78123 2025-08-14T21:49:46.2522226Z STATS: call_* op count: 1189 | FakeTensorMode.__torch_dispatch__:29419 | FakeTensor.__torch_dispatch__:8702 | ProxyTorchDispatchMode.__torch_dispatch__:10618 2025-08-14T21:49:46.2522726Z Dynamo produced 1 graphs covering 1189 ops with 0 graph breaks (0 unique) 2025-08-14T21:49:51.7378033Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:49:51.7379335Z from pkg_resources import resource_filename 2025-08-14T21:49:52.3753937Z 2025-08-14T21:49:52.3871425Z loading model: 0it [00:00, ?it/s]If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2025-08-14T21:49:52.3872073Z WARNING:transformers.models.megatron_bert.modeling_megatron_bert:If you want to use `MegatronBertForCausalLM` as a standalone, add `is_decoder=True.` 2025-08-14T21:49:55.6391416Z 2025-08-14T21:49:55.6397258Z loading model: 0it [00:03, ?it/s] 2025-08-14T21:49:55.6416911Z cpu eval MegatronBertForCausalLM 2025-08-14T21:49:57.3244325Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:49:57.9386822Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:49:58.5817341Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:50:12.5514851Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5520305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5522084Z return mod(**inputs) 2025-08-14T21:50:12.5522943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5523460Z outputs = self.bert( 2025-08-14T21:50:12.5523906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5524378Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5525209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5525803Z layer_outputs = layer_module( 2025-08-14T21:50:12.5526185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5526591Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5527662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5528361Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5528844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5529340Z self_outputs = self.self( 2025-08-14T21:50:12.5529889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5530318Z return func(*args, **kwargs) 2025-08-14T21:50:12.5530932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.5531405Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.5531642Z 2025-08-14T21:50:12.5531761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5532132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5532472Z return mod(**inputs) 2025-08-14T21:50:12.5532886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5533316Z outputs = self.bert( 2025-08-14T21:50:12.5533726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5534541Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5535083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5535505Z layer_outputs = layer_module( 2025-08-14T21:50:12.5535995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5536368Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5536798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5537229Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5537990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5538533Z self_outputs = self.self( 2025-08-14T21:50:12.5538898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5539386Z return func(*args, **kwargs) 2025-08-14T21:50:12.5539934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.5540352Z key_layer = self.key(current_states) 2025-08-14T21:50:12.5540537Z 2025-08-14T21:50:12.5540653Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5541051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5541362Z return mod(**inputs) 2025-08-14T21:50:12.5541738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5542205Z outputs = self.bert( 2025-08-14T21:50:12.5542615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5543110Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5543573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5543975Z layer_outputs = layer_module( 2025-08-14T21:50:12.5544307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5544651Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5545183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5545593Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5546003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5546408Z self_outputs = self.self( 2025-08-14T21:50:12.5546749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5547108Z return func(*args, **kwargs) 2025-08-14T21:50:12.5547503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.5547900Z value_layer = self.value(current_states) 2025-08-14T21:50:12.5548035Z 2025-08-14T21:50:12.5548116Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.5548332Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.5548567Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5549002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5549330Z return mod(**inputs) 2025-08-14T21:50:12.5549826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5550263Z outputs = self.bert( 2025-08-14T21:50:12.5550712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5551145Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5551577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5552046Z layer_outputs = layer_module( 2025-08-14T21:50:12.5552375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5552784Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5553190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5553691Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5554123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.5554655Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.5555185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.5555602Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5555804Z 2025-08-14T21:50:12.5555905Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5556243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5556542Z return mod(**inputs) 2025-08-14T21:50:12.5556936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5557337Z outputs = self.bert( 2025-08-14T21:50:12.5557718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5558122Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5558596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5559054Z layer_outputs = layer_module( 2025-08-14T21:50:12.5559428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5559792Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5560233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5560674Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5561074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5561472Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5561945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.5562536Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.5563060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.5563545Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5563690Z 2025-08-14T21:50:12.5563807Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5564180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5564528Z return mod(**inputs) 2025-08-14T21:50:12.5564983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5565423Z outputs = self.bert( 2025-08-14T21:50:12.5566021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5566476Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5566919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5567376Z layer_outputs = layer_module( 2025-08-14T21:50:12.5567727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5568100Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5568537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5568977Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5569559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5570001Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5570505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.5571070Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.5571566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.5572049Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.5572427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.5572785Z return self.act(input) 2025-08-14T21:50:12.5572906Z 2025-08-14T21:50:12.5573010Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5573370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5573775Z return mod(**inputs) 2025-08-14T21:50:12.5574164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5574650Z outputs = self.bert( 2025-08-14T21:50:12.5575043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5575491Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5575879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5576281Z layer_outputs = layer_module( 2025-08-14T21:50:12.5576611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5576952Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5577375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5577854Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5578328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5578698Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5579214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.5579763Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.5580240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.5580649Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5580779Z 2025-08-14T21:50:12.5580889Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5581232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5581536Z return mod(**inputs) 2025-08-14T21:50:12.5581914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5582310Z outputs = self.bert( 2025-08-14T21:50:12.5582675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5583077Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5583474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5583916Z layer_outputs = layer_module( 2025-08-14T21:50:12.5584256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5584601Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5585006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5585494Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5585893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5586384Z self_outputs = self.self( 2025-08-14T21:50:12.5586756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5587190Z return func(*args, **kwargs) 2025-08-14T21:50:12.5587667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.5588086Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.5588216Z 2025-08-14T21:50:12.5588325Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5588665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5588977Z return mod(**inputs) 2025-08-14T21:50:12.5589369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5589838Z outputs = self.bert( 2025-08-14T21:50:12.5590294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5590709Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5591199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5591610Z layer_outputs = layer_module( 2025-08-14T21:50:12.5592014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5592372Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5592864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5593362Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5593780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5594266Z self_outputs = self.self( 2025-08-14T21:50:12.5594612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5595001Z return func(*args, **kwargs) 2025-08-14T21:50:12.5595401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.5595818Z key_layer = self.key(current_states) 2025-08-14T21:50:12.5595945Z 2025-08-14T21:50:12.5596046Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5596392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5596720Z return mod(**inputs) 2025-08-14T21:50:12.5597155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5597622Z outputs = self.bert( 2025-08-14T21:50:12.5598009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5598422Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5598862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5599345Z layer_outputs = layer_module( 2025-08-14T21:50:12.5599693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5600146Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5600561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5601111Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5601677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5602139Z self_outputs = self.self( 2025-08-14T21:50:12.5602530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5603026Z return func(*args, **kwargs) 2025-08-14T21:50:12.5603468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.5603926Z value_layer = self.value(current_states) 2025-08-14T21:50:12.5604075Z 2025-08-14T21:50:12.5604162Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.5604391Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.5604710Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5605083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5605600Z return mod(**inputs) 2025-08-14T21:50:12.5606066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5606574Z outputs = self.bert( 2025-08-14T21:50:12.5607119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5607538Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5607948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5608380Z layer_outputs = layer_module( 2025-08-14T21:50:12.5608742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5609118Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5609572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5610032Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5610527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.5611034Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.5611536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.5612002Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5612159Z 2025-08-14T21:50:12.5612271Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5612649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5612989Z return mod(**inputs) 2025-08-14T21:50:12.5613420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5613957Z outputs = self.bert( 2025-08-14T21:50:12.5614390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5614937Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5615375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5615827Z layer_outputs = layer_module( 2025-08-14T21:50:12.5616185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5616563Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5617004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5617435Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5617849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5618256Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5618788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.5619345Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.5619829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.5620383Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5620527Z 2025-08-14T21:50:12.5620642Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5621099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5621435Z return mod(**inputs) 2025-08-14T21:50:12.5621902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5622386Z outputs = self.bert( 2025-08-14T21:50:12.5622763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5623280Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5623742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5624224Z layer_outputs = layer_module( 2025-08-14T21:50:12.5624631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5625028Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5625552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5626136Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5626587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5627064Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5627629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.5628205Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.5628682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.5629269Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.5629768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.5630126Z return self.act(input) 2025-08-14T21:50:12.5630281Z 2025-08-14T21:50:12.5630448Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5630871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5631286Z return mod(**inputs) 2025-08-14T21:50:12.5631695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5632235Z outputs = self.bert( 2025-08-14T21:50:12.5632738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5633186Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5633705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5634210Z layer_outputs = layer_module( 2025-08-14T21:50:12.5635235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5635702Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5636195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5636783Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5637269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5637978Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5638467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.5639170Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.5639718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.5640307Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5640455Z 2025-08-14T21:50:12.5640577Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5641066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5641404Z return mod(**inputs) 2025-08-14T21:50:12.5641938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5642393Z outputs = self.bert( 2025-08-14T21:50:12.5642812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5643269Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5643717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5644360Z layer_outputs = layer_module( 2025-08-14T21:50:12.5644847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5645231Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5645835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5646299Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5646716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5647132Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5647607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.5648145Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.5648689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:12.5649176Z return input_tensor + hidden_states 2025-08-14T21:50:12.5649316Z 2025-08-14T21:50:12.5649435Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5649816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5650146Z return mod(**inputs) 2025-08-14T21:50:12.5650547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5650968Z outputs = self.bert( 2025-08-14T21:50:12.5651361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5651830Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5652245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5652663Z layer_outputs = layer_module( 2025-08-14T21:50:12.5653008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5653367Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5653795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5654225Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5654652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5655092Z self_outputs = self.self( 2025-08-14T21:50:12.5655467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5655852Z return func(*args, **kwargs) 2025-08-14T21:50:12.5656272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.5656713Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.5656852Z 2025-08-14T21:50:12.5656958Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5657327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5657666Z return mod(**inputs) 2025-08-14T21:50:12.5658073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5658496Z outputs = self.bert( 2025-08-14T21:50:12.5658900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5659369Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5659792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5660228Z layer_outputs = layer_module( 2025-08-14T21:50:12.5660585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5660940Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5661358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5661785Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5662215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5662639Z self_outputs = self.self( 2025-08-14T21:50:12.5663006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5663411Z return func(*args, **kwargs) 2025-08-14T21:50:12.5663811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.5664217Z key_layer = self.key(current_states) 2025-08-14T21:50:12.5664357Z 2025-08-14T21:50:12.5664458Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5664808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5665117Z return mod(**inputs) 2025-08-14T21:50:12.5665500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5665916Z outputs = self.bert( 2025-08-14T21:50:12.5666339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5666770Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5667165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5667572Z layer_outputs = layer_module( 2025-08-14T21:50:12.5667906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5668302Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5668742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5669153Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5669567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5669970Z self_outputs = self.self( 2025-08-14T21:50:12.5670325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5670689Z return func(*args, **kwargs) 2025-08-14T21:50:12.5671078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.5671495Z value_layer = self.value(current_states) 2025-08-14T21:50:12.5671634Z 2025-08-14T21:50:12.5671714Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.5671927Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.5672153Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5672505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5672823Z return mod(**inputs) 2025-08-14T21:50:12.5673239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5673640Z outputs = self.bert( 2025-08-14T21:50:12.5674023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5674435Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5674832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5675258Z layer_outputs = layer_module( 2025-08-14T21:50:12.5675594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5675947Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5676354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5676777Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5677231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.5677678Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.5678118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.5678524Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5678654Z 2025-08-14T21:50:12.5678761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5679097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5679410Z return mod(**inputs) 2025-08-14T21:50:12.5679816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5680232Z outputs = self.bert( 2025-08-14T21:50:12.5680620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5681021Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5681419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5681820Z layer_outputs = layer_module( 2025-08-14T21:50:12.5682139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5682483Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5682886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5683301Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5683690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5684083Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5684537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.5685020Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.5685605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.5686100Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5686238Z 2025-08-14T21:50:12.5686356Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5686711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5687081Z return mod(**inputs) 2025-08-14T21:50:12.5687465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5687866Z outputs = self.bert( 2025-08-14T21:50:12.5688244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5688655Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5689072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5689487Z layer_outputs = layer_module( 2025-08-14T21:50:12.5689829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5690244Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5690653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5691098Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5691502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5691897Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5692329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.5692795Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.5693245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.5693686Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.5694089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.5694433Z return self.act(input) 2025-08-14T21:50:12.5694545Z 2025-08-14T21:50:12.5694651Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5695003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5695310Z return mod(**inputs) 2025-08-14T21:50:12.5695699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5696104Z outputs = self.bert( 2025-08-14T21:50:12.5696483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5696898Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5697301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5697716Z layer_outputs = layer_module( 2025-08-14T21:50:12.5698044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5698397Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5698808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5699229Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5699622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5700002Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5700436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.5700921Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.5701424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.5701849Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5701983Z 2025-08-14T21:50:12.5702093Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5702435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5702749Z return mod(**inputs) 2025-08-14T21:50:12.5703138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5703546Z outputs = self.bert( 2025-08-14T21:50:12.5703928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5704345Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5704762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5705183Z layer_outputs = layer_module( 2025-08-14T21:50:12.5705539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5705893Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5706310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5706728Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5707151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5707566Z self_outputs = self.self( 2025-08-14T21:50:12.5707930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5708298Z return func(*args, **kwargs) 2025-08-14T21:50:12.5708693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.5709111Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.5709245Z 2025-08-14T21:50:12.5709349Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5709711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5710034Z return mod(**inputs) 2025-08-14T21:50:12.5710432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5710834Z outputs = self.bert( 2025-08-14T21:50:12.5711212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5711613Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5711994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5712394Z layer_outputs = layer_module( 2025-08-14T21:50:12.5712720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5713097Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5713491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5713895Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5714294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5714687Z self_outputs = self.self( 2025-08-14T21:50:12.5715053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5715415Z return func(*args, **kwargs) 2025-08-14T21:50:12.5715801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.5716194Z key_layer = self.key(current_states) 2025-08-14T21:50:12.5716329Z 2025-08-14T21:50:12.5716426Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5716767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5717068Z return mod(**inputs) 2025-08-14T21:50:12.5717432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5717825Z outputs = self.bert( 2025-08-14T21:50:12.5718195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5718610Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5719019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5719413Z layer_outputs = layer_module( 2025-08-14T21:50:12.5719734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5720063Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5720463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5720867Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5721270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5721676Z self_outputs = self.self( 2025-08-14T21:50:12.5722024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5722391Z return func(*args, **kwargs) 2025-08-14T21:50:12.5722778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.5723195Z value_layer = self.value(current_states) 2025-08-14T21:50:12.5723332Z 2025-08-14T21:50:12.5723415Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.5723627Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.5723853Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5724205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5724525Z return mod(**inputs) 2025-08-14T21:50:12.5724908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5725316Z outputs = self.bert( 2025-08-14T21:50:12.5725843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5726300Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5726804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5727233Z layer_outputs = layer_module( 2025-08-14T21:50:12.5727574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5727930Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5728333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5728785Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5729195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.5729654Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.5730112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.5730527Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5730660Z 2025-08-14T21:50:12.5730765Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5731106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5731423Z return mod(**inputs) 2025-08-14T21:50:12.5731812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5732222Z outputs = self.bert( 2025-08-14T21:50:12.5732617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5733048Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5733450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5733855Z layer_outputs = layer_module( 2025-08-14T21:50:12.5734197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5734550Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5734962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5735371Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5735785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5736163Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5736589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.5737037Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.5737470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.5738133Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5738344Z 2025-08-14T21:50:12.5738486Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5738885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5739206Z return mod(**inputs) 2025-08-14T21:50:12.5739593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5740003Z outputs = self.bert( 2025-08-14T21:50:12.5740382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5740789Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5741192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5741586Z layer_outputs = layer_module( 2025-08-14T21:50:12.5741916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5742264Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5742664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5743147Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5743533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5743914Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5744331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.5744788Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.5745213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.5745649Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.5746006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.5746335Z return self.act(input) 2025-08-14T21:50:12.5746444Z 2025-08-14T21:50:12.5746551Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5746954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5747265Z return mod(**inputs) 2025-08-14T21:50:12.5747650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5748047Z outputs = self.bert( 2025-08-14T21:50:12.5748403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5748810Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5749219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5749631Z layer_outputs = layer_module( 2025-08-14T21:50:12.5749985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5750355Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5750752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5751151Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5751533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5751919Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5752351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.5752832Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.5753293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.5753711Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5753843Z 2025-08-14T21:50:12.5753953Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5754294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5754609Z return mod(**inputs) 2025-08-14T21:50:12.5754991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5755387Z outputs = self.bert( 2025-08-14T21:50:12.5755762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5756156Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5756550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5756987Z layer_outputs = layer_module( 2025-08-14T21:50:12.5757308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5757637Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5758020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5758422Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5758796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5759167Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5759578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.5760053Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.5760551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:12.5760960Z return input_tensor + hidden_states 2025-08-14T21:50:12.5761086Z 2025-08-14T21:50:12.5761186Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5761536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5761846Z return mod(**inputs) 2025-08-14T21:50:12.5762218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5762617Z outputs = self.bert( 2025-08-14T21:50:12.5763020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5763443Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5763859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5764264Z layer_outputs = layer_module( 2025-08-14T21:50:12.5764599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5764937Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5765350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5765851Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5766278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5766690Z self_outputs = self.self( 2025-08-14T21:50:12.5767058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5767437Z return func(*args, **kwargs) 2025-08-14T21:50:12.5767846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.5768272Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.5768421Z 2025-08-14T21:50:12.5768526Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5768889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5769211Z return mod(**inputs) 2025-08-14T21:50:12.5769600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5770018Z outputs = self.bert( 2025-08-14T21:50:12.5770412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5770863Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5771292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5771712Z layer_outputs = layer_module( 2025-08-14T21:50:12.5772055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5772407Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5772821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5773215Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5773607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5773988Z self_outputs = self.self( 2025-08-14T21:50:12.5774324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5774707Z return func(*args, **kwargs) 2025-08-14T21:50:12.5775081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.5775472Z key_layer = self.key(current_states) 2025-08-14T21:50:12.5775600Z 2025-08-14T21:50:12.5775695Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5776031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5776325Z return mod(**inputs) 2025-08-14T21:50:12.5776698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5777111Z outputs = self.bert( 2025-08-14T21:50:12.5777491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5777910Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5778303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5778699Z layer_outputs = layer_module( 2025-08-14T21:50:12.5779032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5779390Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5779814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5780232Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5780639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5781051Z self_outputs = self.self( 2025-08-14T21:50:12.5781386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5781727Z return func(*args, **kwargs) 2025-08-14T21:50:12.5782105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.5782500Z value_layer = self.value(current_states) 2025-08-14T21:50:12.5782623Z 2025-08-14T21:50:12.5782707Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.5782902Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.5783119Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5783462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5783830Z return mod(**inputs) 2025-08-14T21:50:12.5784214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5784609Z outputs = self.bert( 2025-08-14T21:50:12.5784977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5785367Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5785754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5786153Z layer_outputs = layer_module( 2025-08-14T21:50:12.5786481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5786811Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5787218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5787630Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5788054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.5788498Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.5788933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.5789333Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5789459Z 2025-08-14T21:50:12.5789554Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5789906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5790214Z return mod(**inputs) 2025-08-14T21:50:12.5790614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5791008Z outputs = self.bert( 2025-08-14T21:50:12.5791390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5791776Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5792158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5792603Z layer_outputs = layer_module( 2025-08-14T21:50:12.5792931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5793272Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5793662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5794078Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5794456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5794833Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5795247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.5795702Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.5796130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.5796554Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5796694Z 2025-08-14T21:50:12.5796795Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5797167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5797526Z return mod(**inputs) 2025-08-14T21:50:12.5797926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5798353Z outputs = self.bert( 2025-08-14T21:50:12.5798755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5799170Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5799583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5800002Z layer_outputs = layer_module( 2025-08-14T21:50:12.5800338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5800701Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5801135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5801611Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5802036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5802421Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5802873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.5803328Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.5803762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.5804208Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.5804609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.5804952Z return self.act(input) 2025-08-14T21:50:12.5805062Z 2025-08-14T21:50:12.5805165Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5805603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5805921Z return mod(**inputs) 2025-08-14T21:50:12.5806309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5806719Z outputs = self.bert( 2025-08-14T21:50:12.5807106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5807509Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5807895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5808294Z layer_outputs = layer_module( 2025-08-14T21:50:12.5808626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5808965Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5809354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5809762Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5810144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5810514Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5810929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.5811393Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.5811879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.5812289Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5812417Z 2025-08-14T21:50:12.5812511Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5812846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5813145Z return mod(**inputs) 2025-08-14T21:50:12.5813524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5813924Z outputs = self.bert( 2025-08-14T21:50:12.5814299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5814706Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5815097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5815537Z layer_outputs = layer_module( 2025-08-14T21:50:12.5815872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5816210Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5816615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5817024Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5817441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5817849Z self_outputs = self.self( 2025-08-14T21:50:12.5818231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5818607Z return func(*args, **kwargs) 2025-08-14T21:50:12.5819029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.5819441Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.5819578Z 2025-08-14T21:50:12.5819675Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5820017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5820316Z return mod(**inputs) 2025-08-14T21:50:12.5820702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5821097Z outputs = self.bert( 2025-08-14T21:50:12.5821472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5821868Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5822269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5822669Z layer_outputs = layer_module( 2025-08-14T21:50:12.5822995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5823329Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5823729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5824132Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5824523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5824924Z self_outputs = self.self( 2025-08-14T21:50:12.5825295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5825652Z return func(*args, **kwargs) 2025-08-14T21:50:12.5826034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.5826438Z key_layer = self.key(current_states) 2025-08-14T21:50:12.5826562Z 2025-08-14T21:50:12.5826666Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5827008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5827312Z return mod(**inputs) 2025-08-14T21:50:12.5827690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5828090Z outputs = self.bert( 2025-08-14T21:50:12.5828463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5828881Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5829298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5829700Z layer_outputs = layer_module( 2025-08-14T21:50:12.5830021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5830359Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5830759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5831158Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5831583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5831996Z self_outputs = self.self( 2025-08-14T21:50:12.5832348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5832703Z return func(*args, **kwargs) 2025-08-14T21:50:12.5833105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.5833511Z value_layer = self.value(current_states) 2025-08-14T21:50:12.5833637Z 2025-08-14T21:50:12.5833723Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.5833922Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.5834148Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5834485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5834785Z return mod(**inputs) 2025-08-14T21:50:12.5835163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5835558Z outputs = self.bert( 2025-08-14T21:50:12.5835928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5836321Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5836711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5837104Z layer_outputs = layer_module( 2025-08-14T21:50:12.5837425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5837962Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5838580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5839053Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5839445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.5839893Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.5840335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.5840739Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5840870Z 2025-08-14T21:50:12.5840970Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5841314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5841623Z return mod(**inputs) 2025-08-14T21:50:12.5842457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5842862Z outputs = self.bert( 2025-08-14T21:50:12.5843312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5843724Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5844121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5844515Z layer_outputs = layer_module( 2025-08-14T21:50:12.5844835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5845175Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5845643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5846110Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5846510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5846884Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5847313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.5847768Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.5848200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.5848605Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5848740Z 2025-08-14T21:50:12.5848840Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5849179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5849486Z return mod(**inputs) 2025-08-14T21:50:12.5849852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5850246Z outputs = self.bert( 2025-08-14T21:50:12.5850616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5851011Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5851418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5851831Z layer_outputs = layer_module( 2025-08-14T21:50:12.5852160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5852490Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5852889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5853330Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5853710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5854077Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5854501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.5854955Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.5855371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.5855811Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.5856171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.5856494Z return self.act(input) 2025-08-14T21:50:12.5856599Z 2025-08-14T21:50:12.5856721Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5857078Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5857397Z return mod(**inputs) 2025-08-14T21:50:12.5857778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5858163Z outputs = self.bert( 2025-08-14T21:50:12.5858537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5858937Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5859342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5859787Z layer_outputs = layer_module( 2025-08-14T21:50:12.5860131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5860485Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5860960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5861383Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5861766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5862142Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5862616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.5863092Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.5863549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.5863952Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5864090Z 2025-08-14T21:50:12.5864191Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5864540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5864854Z return mod(**inputs) 2025-08-14T21:50:12.5865227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5865641Z outputs = self.bert( 2025-08-14T21:50:12.5866006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5866395Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5866798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5867189Z layer_outputs = layer_module( 2025-08-14T21:50:12.5867522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5867857Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5868267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5868675Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5869053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5869424Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5869840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.5870310Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.5870798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:12.5871198Z return input_tensor + hidden_states 2025-08-14T21:50:12.5871331Z 2025-08-14T21:50:12.5871431Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5871768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5872077Z return mod(**inputs) 2025-08-14T21:50:12.5872448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5872833Z outputs = self.bert( 2025-08-14T21:50:12.5873213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5873608Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5873997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5874391Z layer_outputs = layer_module( 2025-08-14T21:50:12.5874712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5875042Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5875441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5875846Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5876229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5876616Z self_outputs = self.self( 2025-08-14T21:50:12.5876952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5877297Z return func(*args, **kwargs) 2025-08-14T21:50:12.5877664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.5878060Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.5878188Z 2025-08-14T21:50:12.5878293Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5878629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5878929Z return mod(**inputs) 2025-08-14T21:50:12.5879305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5879699Z outputs = self.bert( 2025-08-14T21:50:12.5880081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5880480Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5880872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5881265Z layer_outputs = layer_module( 2025-08-14T21:50:12.5881580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5881920Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5882314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5882712Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5883112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5883509Z self_outputs = self.self( 2025-08-14T21:50:12.5883887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5884244Z return func(*args, **kwargs) 2025-08-14T21:50:12.5884642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.5885058Z key_layer = self.key(current_states) 2025-08-14T21:50:12.5885187Z 2025-08-14T21:50:12.5885299Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5885737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5886065Z return mod(**inputs) 2025-08-14T21:50:12.5886471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5886885Z outputs = self.bert( 2025-08-14T21:50:12.5887261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5887661Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5888053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5888441Z layer_outputs = layer_module( 2025-08-14T21:50:12.5888779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5889117Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5889509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5889906Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5890305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5890700Z self_outputs = self.self( 2025-08-14T21:50:12.5891032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5891386Z return func(*args, **kwargs) 2025-08-14T21:50:12.5891776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.5892177Z value_layer = self.value(current_states) 2025-08-14T21:50:12.5892306Z 2025-08-14T21:50:12.5892382Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.5892582Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.5892801Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5893128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5893454Z return mod(**inputs) 2025-08-14T21:50:12.5893832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5894224Z outputs = self.bert( 2025-08-14T21:50:12.5894583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5894979Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5895370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5895764Z layer_outputs = layer_module( 2025-08-14T21:50:12.5896082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5896421Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5896821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5897242Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5897667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.5898114Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.5898557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.5898961Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5899102Z 2025-08-14T21:50:12.5899201Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5899550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5899855Z return mod(**inputs) 2025-08-14T21:50:12.5900260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5900660Z outputs = self.bert( 2025-08-14T21:50:12.5901051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5901442Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5901830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5902227Z layer_outputs = layer_module( 2025-08-14T21:50:12.5902556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5902888Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5903289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5903701Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5904084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5904471Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5904910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.5905380Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.5905801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.5906218Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5906358Z 2025-08-14T21:50:12.5906460Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5906814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5907140Z return mod(**inputs) 2025-08-14T21:50:12.5907520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5907910Z outputs = self.bert( 2025-08-14T21:50:12.5908279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5908685Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5909086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5909482Z layer_outputs = layer_module( 2025-08-14T21:50:12.5909804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5910154Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5910563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5911022Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5911408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5911792Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5912228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.5912675Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.5913100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.5913533Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.5913905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.5914226Z return self.act(input) 2025-08-14T21:50:12.5914341Z 2025-08-14T21:50:12.5914445Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5914801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5915116Z return mod(**inputs) 2025-08-14T21:50:12.5915493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5915896Z outputs = self.bert( 2025-08-14T21:50:12.5916278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5916679Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5917084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5917492Z layer_outputs = layer_module( 2025-08-14T21:50:12.5917827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5918167Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5918576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5918997Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5919386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5919756Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5920190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.5920692Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.5921140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.5921555Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5921690Z 2025-08-14T21:50:12.5921789Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5922130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5922431Z return mod(**inputs) 2025-08-14T21:50:12.5922814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5923221Z outputs = self.bert( 2025-08-14T21:50:12.5923608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5924018Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5924430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5924888Z layer_outputs = layer_module( 2025-08-14T21:50:12.5925231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5925688Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5926125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5926605Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5927073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5927481Z self_outputs = self.self( 2025-08-14T21:50:12.5927859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5928218Z return func(*args, **kwargs) 2025-08-14T21:50:12.5928599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.5929008Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.5929137Z 2025-08-14T21:50:12.5929244Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5929578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5929885Z return mod(**inputs) 2025-08-14T21:50:12.5930263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5930326Z outputs = self.bert( 2025-08-14T21:50:12.5930601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5930674Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5930944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5931022Z layer_outputs = layer_module( 2025-08-14T21:50:12.5931229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5931310Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5931577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5931654Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5931929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5932025Z self_outputs = self.self( 2025-08-14T21:50:12.5932255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5932334Z return func(*args, **kwargs) 2025-08-14T21:50:12.5932603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.5932682Z key_layer = self.key(current_states) 2025-08-14T21:50:12.5932686Z 2025-08-14T21:50:12.5932784Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5932969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5933038Z return mod(**inputs) 2025-08-14T21:50:12.5933312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5933381Z outputs = self.bert( 2025-08-14T21:50:12.5933650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5933758Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5934033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5934100Z layer_outputs = layer_module( 2025-08-14T21:50:12.5934304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5934385Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5934649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5934731Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5935016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5935084Z self_outputs = self.self( 2025-08-14T21:50:12.5935321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5935390Z return func(*args, **kwargs) 2025-08-14T21:50:12.5935669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.5935745Z value_layer = self.value(current_states) 2025-08-14T21:50:12.5935748Z 2025-08-14T21:50:12.5935825Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.5935912Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.5936011Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5936203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5936276Z return mod(**inputs) 2025-08-14T21:50:12.5936563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5936636Z outputs = self.bert( 2025-08-14T21:50:12.5936914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5936986Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5937279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5937347Z layer_outputs = layer_module( 2025-08-14T21:50:12.5937556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5937762Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5938213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5938403Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5938721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.5938848Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.5939135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.5939217Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5939221Z 2025-08-14T21:50:12.5939329Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5939524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5939589Z return mod(**inputs) 2025-08-14T21:50:12.5939881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5939949Z outputs = self.bert( 2025-08-14T21:50:12.5940304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5940386Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5940665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5940741Z layer_outputs = layer_module( 2025-08-14T21:50:12.5940957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5941036Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5941321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5941431Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5941697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5941775Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5942083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.5942193Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.5942469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.5942557Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5942561Z 2025-08-14T21:50:12.5942661Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5942853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5942925Z return mod(**inputs) 2025-08-14T21:50:12.5943203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5943269Z outputs = self.bert( 2025-08-14T21:50:12.5943552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5943624Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5943909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5943978Z layer_outputs = layer_module( 2025-08-14T21:50:12.5944188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5944269Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5944548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5944666Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5944919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5944993Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5945304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.5945404Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.5945679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.5945798Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.5946001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.5946077Z return self.act(input) 2025-08-14T21:50:12.5946100Z 2025-08-14T21:50:12.5946204Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5946416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5946489Z return mod(**inputs) 2025-08-14T21:50:12.5946772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5946844Z outputs = self.bert( 2025-08-14T21:50:12.5947121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5947191Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5947530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5947602Z layer_outputs = layer_module( 2025-08-14T21:50:12.5947814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5947900Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5948176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5948261Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5948513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5948586Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5948901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.5949024Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.5949302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.5949380Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5949383Z 2025-08-14T21:50:12.5949481Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5949673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5949734Z return mod(**inputs) 2025-08-14T21:50:12.5950006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5950075Z outputs = self.bert( 2025-08-14T21:50:12.5950342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5950419Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5950707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5950778Z layer_outputs = layer_module( 2025-08-14T21:50:12.5950993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5951065Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5951341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5951420Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5951661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5951742Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5952041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.5952161Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.5952487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:12.5952565Z return input_tensor + hidden_states 2025-08-14T21:50:12.5952568Z 2025-08-14T21:50:12.5952674Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5952862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5952928Z return mod(**inputs) 2025-08-14T21:50:12.5953217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5953281Z outputs = self.bert( 2025-08-14T21:50:12.5953591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5953664Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5953937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5954015Z layer_outputs = layer_module( 2025-08-14T21:50:12.5954221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5954296Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5954574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5954651Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5954932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5955001Z self_outputs = self.self( 2025-08-14T21:50:12.5955242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5955320Z return func(*args, **kwargs) 2025-08-14T21:50:12.5955582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.5955666Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.5955669Z 2025-08-14T21:50:12.5955778Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5955961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5956031Z return mod(**inputs) 2025-08-14T21:50:12.5956296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5956357Z outputs = self.bert( 2025-08-14T21:50:12.5956659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5956731Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5957005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5957082Z layer_outputs = layer_module( 2025-08-14T21:50:12.5957283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5957363Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5957622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5957703Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5957961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5958027Z self_outputs = self.self( 2025-08-14T21:50:12.5958298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5958367Z return func(*args, **kwargs) 2025-08-14T21:50:12.5958643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.5958723Z key_layer = self.key(current_states) 2025-08-14T21:50:12.5958726Z 2025-08-14T21:50:12.5958822Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5959019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5959081Z return mod(**inputs) 2025-08-14T21:50:12.5959371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5959445Z outputs = self.bert( 2025-08-14T21:50:12.5959713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5959793Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5960062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5960129Z layer_outputs = layer_module( 2025-08-14T21:50:12.5960342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5960416Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5960683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5960766Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5961035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5961109Z self_outputs = self.self( 2025-08-14T21:50:12.5961339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5961405Z return func(*args, **kwargs) 2025-08-14T21:50:12.5961681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.5961756Z value_layer = self.value(current_states) 2025-08-14T21:50:12.5961759Z 2025-08-14T21:50:12.5961841Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.5961917Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.5962011Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5962206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5962288Z return mod(**inputs) 2025-08-14T21:50:12.5962562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5962634Z outputs = self.bert( 2025-08-14T21:50:12.5962904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5962978Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5963247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5963314Z layer_outputs = layer_module( 2025-08-14T21:50:12.5963524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5963595Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5963868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5963977Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5964267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.5964399Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.5964680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.5964761Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5964765Z 2025-08-14T21:50:12.5964870Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5965065Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5965138Z return mod(**inputs) 2025-08-14T21:50:12.5965518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5965600Z outputs = self.bert( 2025-08-14T21:50:12.5965896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5965969Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5966253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5966335Z layer_outputs = layer_module( 2025-08-14T21:50:12.5966556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5966643Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5966930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5967016Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5967288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5967370Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5967690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.5967795Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.5968081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.5968173Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5968177Z 2025-08-14T21:50:12.5968278Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5968497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5968564Z return mod(**inputs) 2025-08-14T21:50:12.5968854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5968925Z outputs = self.bert( 2025-08-14T21:50:12.5969194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5969264Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5969544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5969611Z layer_outputs = layer_module( 2025-08-14T21:50:12.5969824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5969900Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5970171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5970293Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5970540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5970620Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5970914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.5971012Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.5971284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.5971389Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.5971603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.5971681Z return self.act(input) 2025-08-14T21:50:12.5971686Z 2025-08-14T21:50:12.5971781Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5971974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5972035Z return mod(**inputs) 2025-08-14T21:50:12.5972311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5972383Z outputs = self.bert( 2025-08-14T21:50:12.5972657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5972733Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5973008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5973077Z layer_outputs = layer_module( 2025-08-14T21:50:12.5973292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5973366Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5973634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5973720Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5973963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5974041Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5974343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.5974487Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.5974770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.5974848Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5974851Z 2025-08-14T21:50:12.5974954Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5975141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5975203Z return mod(**inputs) 2025-08-14T21:50:12.5975481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5975544Z outputs = self.bert( 2025-08-14T21:50:12.5975820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5975897Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5976185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5976279Z layer_outputs = layer_module( 2025-08-14T21:50:12.5976486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5976559Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5976835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5976912Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5977194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5977262Z self_outputs = self.self( 2025-08-14T21:50:12.5977506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5977584Z return func(*args, **kwargs) 2025-08-14T21:50:12.5977856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.5977931Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.5977942Z 2025-08-14T21:50:12.5978036Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5978223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5978291Z return mod(**inputs) 2025-08-14T21:50:12.5978589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5978652Z outputs = self.bert( 2025-08-14T21:50:12.5978938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5979010Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5979294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5979362Z layer_outputs = layer_module( 2025-08-14T21:50:12.5979573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5979653Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5979950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5980027Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5980323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5980410Z self_outputs = self.self( 2025-08-14T21:50:12.5980646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5980715Z return func(*args, **kwargs) 2025-08-14T21:50:12.5980982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.5981063Z key_layer = self.key(current_states) 2025-08-14T21:50:12.5981066Z 2025-08-14T21:50:12.5981161Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5981353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5981415Z return mod(**inputs) 2025-08-14T21:50:12.5981687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5981761Z outputs = self.bert( 2025-08-14T21:50:12.5982032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5982138Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5982421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5982488Z layer_outputs = layer_module( 2025-08-14T21:50:12.5982700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5982773Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5983043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5983128Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5983434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.5983513Z self_outputs = self.self( 2025-08-14T21:50:12.5983745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.5983811Z return func(*args, **kwargs) 2025-08-14T21:50:12.5984091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.5984166Z value_layer = self.value(current_states) 2025-08-14T21:50:12.5984170Z 2025-08-14T21:50:12.5984246Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.5984330Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.5984428Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5984620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5984687Z return mod(**inputs) 2025-08-14T21:50:12.5984960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5985035Z outputs = self.bert( 2025-08-14T21:50:12.5985304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5985373Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5985647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5985715Z layer_outputs = layer_module( 2025-08-14T21:50:12.5985934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5986007Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5986280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.5986399Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.5986674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.5986802Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.5987074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.5987153Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5987156Z 2025-08-14T21:50:12.5987259Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5987444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5987505Z return mod(**inputs) 2025-08-14T21:50:12.5987785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5987865Z outputs = self.bert( 2025-08-14T21:50:12.5988156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5988226Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5988495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5988570Z layer_outputs = layer_module( 2025-08-14T21:50:12.5988776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5988856Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5989120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5989220Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5989476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5989549Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5989849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.5989952Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.5990220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.5990304Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5990307Z 2025-08-14T21:50:12.5990402Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5990589Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5990661Z return mod(**inputs) 2025-08-14T21:50:12.5990935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5991006Z outputs = self.bert( 2025-08-14T21:50:12.5991271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5991341Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5991615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5991681Z layer_outputs = layer_module( 2025-08-14T21:50:12.5991885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5991965Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5992253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5992338Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5992582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5992653Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5992956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.5993051Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.5993324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.5993430Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.5993629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.5993705Z return self.act(input) 2025-08-14T21:50:12.5993726Z 2025-08-14T21:50:12.5993839Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5994036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5994097Z return mod(**inputs) 2025-08-14T21:50:12.5994372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5994443Z outputs = self.bert( 2025-08-14T21:50:12.5994713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5994784Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5995084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5995155Z layer_outputs = layer_module( 2025-08-14T21:50:12.5995387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5995460Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5995725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5995808Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5996052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5996125Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5996429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.5996557Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.5996834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.5996914Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.5996917Z 2025-08-14T21:50:12.5997014Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.5997211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.5997272Z return mod(**inputs) 2025-08-14T21:50:12.5997548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.5997609Z outputs = self.bert( 2025-08-14T21:50:12.5997881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.5997976Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.5998237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.5998313Z layer_outputs = layer_module( 2025-08-14T21:50:12.5998520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.5998592Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.5998871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.5998950Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.5999198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.5999279Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.5999584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.5999736Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.6000032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:12.6000109Z return input_tensor + hidden_states 2025-08-14T21:50:12.6000112Z 2025-08-14T21:50:12.6000217Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6000406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6000476Z return mod(**inputs) 2025-08-14T21:50:12.6000754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6000817Z outputs = self.bert( 2025-08-14T21:50:12.6001116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6001192Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6001473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6001548Z layer_outputs = layer_module( 2025-08-14T21:50:12.6001762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6001842Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6002118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6002197Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6002481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6002554Z self_outputs = self.self( 2025-08-14T21:50:12.6002797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6002868Z return func(*args, **kwargs) 2025-08-14T21:50:12.6003147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.6003232Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.6003236Z 2025-08-14T21:50:12.6003333Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6003523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6003597Z return mod(**inputs) 2025-08-14T21:50:12.6003877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6003967Z outputs = self.bert( 2025-08-14T21:50:12.6004247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6004323Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6004606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6004674Z layer_outputs = layer_module( 2025-08-14T21:50:12.6004894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6004970Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6005249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6005335Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6005701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6005798Z self_outputs = self.self( 2025-08-14T21:50:12.6006089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6006167Z return func(*args, **kwargs) 2025-08-14T21:50:12.6006493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.6006575Z key_layer = self.key(current_states) 2025-08-14T21:50:12.6006580Z 2025-08-14T21:50:12.6006687Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6006905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6006975Z return mod(**inputs) 2025-08-14T21:50:12.6007310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6007376Z outputs = self.bert( 2025-08-14T21:50:12.6007660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6007750Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6008022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6008090Z layer_outputs = layer_module( 2025-08-14T21:50:12.6008308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6008380Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6008660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6008739Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6009064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6009142Z self_outputs = self.self( 2025-08-14T21:50:12.6009377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6009443Z return func(*args, **kwargs) 2025-08-14T21:50:12.6009731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.6009806Z value_layer = self.value(current_states) 2025-08-14T21:50:12.6009809Z 2025-08-14T21:50:12.6009895Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6009972Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6010071Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6010272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6010354Z return mod(**inputs) 2025-08-14T21:50:12.6010648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6010709Z outputs = self.bert( 2025-08-14T21:50:12.6010976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6011051Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6011321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6011388Z layer_outputs = layer_module( 2025-08-14T21:50:12.6011601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6011674Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6011949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6012059Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6012326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.6012454Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.6012722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.6012807Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6012810Z 2025-08-14T21:50:12.6012906Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6013089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6013180Z return mod(**inputs) 2025-08-14T21:50:12.6013455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6013522Z outputs = self.bert( 2025-08-14T21:50:12.6013802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6013873Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6014172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6014243Z layer_outputs = layer_module( 2025-08-14T21:50:12.6014458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6014539Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6014840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6014930Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6015181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6015259Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6015574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6015673Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6015949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.6016038Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6016042Z 2025-08-14T21:50:12.6016140Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6016359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6016424Z return mod(**inputs) 2025-08-14T21:50:12.6016701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6016788Z outputs = self.bert( 2025-08-14T21:50:12.6017060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6017137Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6017432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6017500Z layer_outputs = layer_module( 2025-08-14T21:50:12.6017721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6017797Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6018088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6018197Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6018447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6018527Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6018832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6018931Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6019226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.6019355Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.6019577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.6019650Z return self.act(input) 2025-08-14T21:50:12.6019654Z 2025-08-14T21:50:12.6019756Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6019965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6020030Z return mod(**inputs) 2025-08-14T21:50:12.6020319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6020392Z outputs = self.bert( 2025-08-14T21:50:12.6020678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6020760Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6021049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6021119Z layer_outputs = layer_module( 2025-08-14T21:50:12.6021340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6021414Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6021695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6021774Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6022023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6022106Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6022411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.6022560Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.6022845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.6022923Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6022927Z 2025-08-14T21:50:12.6023033Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6023233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6023294Z return mod(**inputs) 2025-08-14T21:50:12.6023571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6023632Z outputs = self.bert( 2025-08-14T21:50:12.6023907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6023978Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6024284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6024360Z layer_outputs = layer_module( 2025-08-14T21:50:12.6024567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6024639Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6024916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6024992Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6025272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6025358Z self_outputs = self.self( 2025-08-14T21:50:12.6025589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6025667Z return func(*args, **kwargs) 2025-08-14T21:50:12.6025935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.6026020Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.6026024Z 2025-08-14T21:50:12.6026119Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6026307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6026376Z return mod(**inputs) 2025-08-14T21:50:12.6026645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6026719Z outputs = self.bert( 2025-08-14T21:50:12.6027031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6027101Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6027379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6027447Z layer_outputs = layer_module( 2025-08-14T21:50:12.6027657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6027740Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6028014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6028100Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6028375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6028464Z self_outputs = self.self( 2025-08-14T21:50:12.6028721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6028791Z return func(*args, **kwargs) 2025-08-14T21:50:12.6029073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.6029160Z key_layer = self.key(current_states) 2025-08-14T21:50:12.6029164Z 2025-08-14T21:50:12.6029263Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6029468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6029534Z return mod(**inputs) 2025-08-14T21:50:12.6029831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6029905Z outputs = self.bert( 2025-08-14T21:50:12.6030198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6030302Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6030572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6030639Z layer_outputs = layer_module( 2025-08-14T21:50:12.6030854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6030927Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6031194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6031279Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6031566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6031643Z self_outputs = self.self( 2025-08-14T21:50:12.6031876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6031944Z return func(*args, **kwargs) 2025-08-14T21:50:12.6032225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.6032300Z value_layer = self.value(current_states) 2025-08-14T21:50:12.6032303Z 2025-08-14T21:50:12.6032387Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6032463Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6032559Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6032754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6032818Z return mod(**inputs) 2025-08-14T21:50:12.6033093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6033165Z outputs = self.bert( 2025-08-14T21:50:12.6033435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6033511Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6033784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6033850Z layer_outputs = layer_module( 2025-08-14T21:50:12.6034066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6034140Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6034415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6034522Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6034794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.6034918Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.6035188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.6035266Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6035270Z 2025-08-14T21:50:12.6035373Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6035559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6035628Z return mod(**inputs) 2025-08-14T21:50:12.6035900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6035981Z outputs = self.bert( 2025-08-14T21:50:12.6036916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6037013Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6037296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6037364Z layer_outputs = layer_module( 2025-08-14T21:50:12.6037571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6037871Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6038419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6038515Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6038781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6038856Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6039178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6039275Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6039544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.6039629Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6039633Z 2025-08-14T21:50:12.6039729Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6039927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6039993Z return mod(**inputs) 2025-08-14T21:50:12.6040265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6040339Z outputs = self.bert( 2025-08-14T21:50:12.6040607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6040676Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6040952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6041020Z layer_outputs = layer_module( 2025-08-14T21:50:12.6041236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6041309Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6041617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6041706Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6041961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6042043Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6042347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6042444Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6042722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.6042828Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.6043034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.6043139Z return self.act(input) 2025-08-14T21:50:12.6043143Z 2025-08-14T21:50:12.6043315Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6043512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6043575Z return mod(**inputs) 2025-08-14T21:50:12.6043849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6043920Z outputs = self.bert( 2025-08-14T21:50:12.6044198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6044276Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6044574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6044646Z layer_outputs = layer_module( 2025-08-14T21:50:12.6044868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6044943Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6045220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6045306Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6045612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6045701Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6046009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.6046141Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.6046427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.6046508Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6046511Z 2025-08-14T21:50:12.6046617Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6046808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6046873Z return mod(**inputs) 2025-08-14T21:50:12.6047157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6047222Z outputs = self.bert( 2025-08-14T21:50:12.6047495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6047602Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6047880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6047958Z layer_outputs = layer_module( 2025-08-14T21:50:12.6048168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6048244Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6048527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6048607Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6048863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6048935Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6049243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.6049415Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.6049691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:12.6049772Z return input_tensor + hidden_states 2025-08-14T21:50:12.6049776Z 2025-08-14T21:50:12.6049874Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6050066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6050135Z return mod(**inputs) 2025-08-14T21:50:12.6050414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6050478Z outputs = self.bert( 2025-08-14T21:50:12.6050778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6050854Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6051139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6051207Z layer_outputs = layer_module( 2025-08-14T21:50:12.6051419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6051501Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6051775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6051853Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6052137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6052207Z self_outputs = self.self( 2025-08-14T21:50:12.6052452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6052521Z return func(*args, **kwargs) 2025-08-14T21:50:12.6052796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.6052883Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.6052887Z 2025-08-14T21:50:12.6052985Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6053182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6053244Z return mod(**inputs) 2025-08-14T21:50:12.6053524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6053617Z outputs = self.bert( 2025-08-14T21:50:12.6053896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6053971Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6054256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6054324Z layer_outputs = layer_module( 2025-08-14T21:50:12.6054544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6054618Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6054896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6054982Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6055260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6055360Z self_outputs = self.self( 2025-08-14T21:50:12.6055609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6055679Z return func(*args, **kwargs) 2025-08-14T21:50:12.6055961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.6056035Z key_layer = self.key(current_states) 2025-08-14T21:50:12.6056038Z 2025-08-14T21:50:12.6056135Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6056333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6056395Z return mod(**inputs) 2025-08-14T21:50:12.6056757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6056821Z outputs = self.bert( 2025-08-14T21:50:12.6057085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6057160Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6057422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6057493Z layer_outputs = layer_module( 2025-08-14T21:50:12.6057696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6057767Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6058035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6058111Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6058374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6058447Z self_outputs = self.self( 2025-08-14T21:50:12.6058675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6058749Z return func(*args, **kwargs) 2025-08-14T21:50:12.6059023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.6059096Z value_layer = self.value(current_states) 2025-08-14T21:50:12.6059100Z 2025-08-14T21:50:12.6059184Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6059259Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6059361Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6059569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6059632Z return mod(**inputs) 2025-08-14T21:50:12.6059911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6059975Z outputs = self.bert( 2025-08-14T21:50:12.6060248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6060325Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6060584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6060655Z layer_outputs = layer_module( 2025-08-14T21:50:12.6060853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6060928Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6061198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6061307Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6061579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.6061708Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.6061981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.6062077Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6062081Z 2025-08-14T21:50:12.6062175Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6062376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6062449Z return mod(**inputs) 2025-08-14T21:50:12.6062719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6062791Z outputs = self.bert( 2025-08-14T21:50:12.6063059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6063129Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6063407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6063474Z layer_outputs = layer_module( 2025-08-14T21:50:12.6063687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6063759Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6064034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6064121Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6064362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6064431Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6064724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6064816Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6065087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.6065161Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6065164Z 2025-08-14T21:50:12.6065258Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6065464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6065526Z return mod(**inputs) 2025-08-14T21:50:12.6065798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6065858Z outputs = self.bert( 2025-08-14T21:50:12.6066118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6066193Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6066456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6066520Z layer_outputs = layer_module( 2025-08-14T21:50:12.6066731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6066802Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6067105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6067183Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6067418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6067497Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6067785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6067886Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6068150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.6068271Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.6068475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.6068544Z return self.act(input) 2025-08-14T21:50:12.6068547Z 2025-08-14T21:50:12.6068643Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6068838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6068899Z return mod(**inputs) 2025-08-14T21:50:12.6069178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6069240Z outputs = self.bert( 2025-08-14T21:50:12.6069506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6069582Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6069855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6069932Z layer_outputs = layer_module( 2025-08-14T21:50:12.6070139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6070222Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6070488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6070563Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6070798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6070876Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6071168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.6071317Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.6071582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.6071657Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6071660Z 2025-08-14T21:50:12.6071758Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6071940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6072006Z return mod(**inputs) 2025-08-14T21:50:12.6072267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6072329Z outputs = self.bert( 2025-08-14T21:50:12.6072601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6072689Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6072992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6073067Z layer_outputs = layer_module( 2025-08-14T21:50:12.6073270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6073351Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6073613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6073686Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6073955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6074035Z self_outputs = self.self( 2025-08-14T21:50:12.6074267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6074336Z return func(*args, **kwargs) 2025-08-14T21:50:12.6074598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.6074681Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.6074684Z 2025-08-14T21:50:12.6074777Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6074960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6075028Z return mod(**inputs) 2025-08-14T21:50:12.6075293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6075365Z outputs = self.bert( 2025-08-14T21:50:12.6075631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6075704Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6075988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6076056Z layer_outputs = layer_module( 2025-08-14T21:50:12.6076272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6076347Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6076615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6076700Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6076971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6077072Z self_outputs = self.self( 2025-08-14T21:50:12.6077424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6077513Z return func(*args, **kwargs) 2025-08-14T21:50:12.6077937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.6078033Z key_layer = self.key(current_states) 2025-08-14T21:50:12.6078038Z 2025-08-14T21:50:12.6078165Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6078446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6078522Z return mod(**inputs) 2025-08-14T21:50:12.6078952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6079036Z outputs = self.bert( 2025-08-14T21:50:12.6079522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6079633Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6079969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6080042Z layer_outputs = layer_module( 2025-08-14T21:50:12.6080268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6080347Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6080657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6080741Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6081061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6081145Z self_outputs = self.self( 2025-08-14T21:50:12.6081398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6081482Z return func(*args, **kwargs) 2025-08-14T21:50:12.6081790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.6081872Z value_layer = self.value(current_states) 2025-08-14T21:50:12.6081876Z 2025-08-14T21:50:12.6081969Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6082053Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6082160Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6082374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6082444Z return mod(**inputs) 2025-08-14T21:50:12.6082756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6082825Z outputs = self.bert( 2025-08-14T21:50:12.6083124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6083207Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6083515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6083591Z layer_outputs = layer_module( 2025-08-14T21:50:12.6083825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6083905Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6084233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6084321Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6084620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.6084759Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.6085070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.6085163Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6085166Z 2025-08-14T21:50:12.6085274Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6085550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6085642Z return mod(**inputs) 2025-08-14T21:50:12.6086069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6086213Z outputs = self.bert( 2025-08-14T21:50:12.6086689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6086775Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6087213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6087302Z layer_outputs = layer_module( 2025-08-14T21:50:12.6087662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6087754Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6088059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6088151Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6088416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6088491Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6088814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6088916Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6089197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.6089287Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6089291Z 2025-08-14T21:50:12.6089391Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6089598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6089663Z return mod(**inputs) 2025-08-14T21:50:12.6089951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6090024Z outputs = self.bert( 2025-08-14T21:50:12.6090321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6090404Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6090703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6090777Z layer_outputs = layer_module( 2025-08-14T21:50:12.6091014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6091117Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6091428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6091527Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6091814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6091902Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6092247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6092356Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6092681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.6092803Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.6093030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.6093120Z return self.act(input) 2025-08-14T21:50:12.6093140Z 2025-08-14T21:50:12.6093241Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6093452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6093515Z return mod(**inputs) 2025-08-14T21:50:12.6093795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6093858Z outputs = self.bert( 2025-08-14T21:50:12.6094126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6094207Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6094492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6094565Z layer_outputs = layer_module( 2025-08-14T21:50:12.6094796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6094873Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6095164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6095245Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6095503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6095586Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6095902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.6096045Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.6096335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.6096425Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6096428Z 2025-08-14T21:50:12.6096533Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6096726Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6096790Z return mod(**inputs) 2025-08-14T21:50:12.6097084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6097150Z outputs = self.bert( 2025-08-14T21:50:12.6097443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6097535Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6097825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6097904Z layer_outputs = layer_module( 2025-08-14T21:50:12.6098124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6098209Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6098500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6098585Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6098854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6098928Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6099252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.6099426Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.6099709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:12.6099794Z return input_tensor + hidden_states 2025-08-14T21:50:12.6099798Z 2025-08-14T21:50:12.6099900Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6100095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6100170Z return mod(**inputs) 2025-08-14T21:50:12.6100454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6100555Z outputs = self.bert( 2025-08-14T21:50:12.6100837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6100912Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6101202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6101274Z layer_outputs = layer_module( 2025-08-14T21:50:12.6101489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6101574Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6101856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6101943Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6102230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6102302Z self_outputs = self.self( 2025-08-14T21:50:12.6102550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6102621Z return func(*args, **kwargs) 2025-08-14T21:50:12.6102910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.6102988Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.6102991Z 2025-08-14T21:50:12.6103090Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6103308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6103372Z return mod(**inputs) 2025-08-14T21:50:12.6103655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6103744Z outputs = self.bert( 2025-08-14T21:50:12.6104033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6104111Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6104396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6104466Z layer_outputs = layer_module( 2025-08-14T21:50:12.6104693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6104769Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6105063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6105143Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6105428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6105539Z self_outputs = self.self( 2025-08-14T21:50:12.6105781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6105852Z return func(*args, **kwargs) 2025-08-14T21:50:12.6106144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.6106222Z key_layer = self.key(current_states) 2025-08-14T21:50:12.6106226Z 2025-08-14T21:50:12.6106334Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6106533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6106600Z return mod(**inputs) 2025-08-14T21:50:12.6106914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6106984Z outputs = self.bert( 2025-08-14T21:50:12.6107324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6107393Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6107665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6107740Z layer_outputs = layer_module( 2025-08-14T21:50:12.6107946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6108019Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6108297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6108373Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6108650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6108718Z self_outputs = self.self( 2025-08-14T21:50:12.6108943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6109018Z return func(*args, **kwargs) 2025-08-14T21:50:12.6109287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.6109367Z value_layer = self.value(current_states) 2025-08-14T21:50:12.6109371Z 2025-08-14T21:50:12.6109448Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6109525Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6109629Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6109834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6109900Z return mod(**inputs) 2025-08-14T21:50:12.6110176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6110239Z outputs = self.bert( 2025-08-14T21:50:12.6110524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6110596Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6110907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6110983Z layer_outputs = layer_module( 2025-08-14T21:50:12.6111187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6111264Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6111559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6111652Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6111928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.6112048Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.6112325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.6112414Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6112417Z 2025-08-14T21:50:12.6112516Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6112733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6112800Z return mod(**inputs) 2025-08-14T21:50:12.6113086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6113159Z outputs = self.bert( 2025-08-14T21:50:12.6113436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6113509Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6113797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6113865Z layer_outputs = layer_module( 2025-08-14T21:50:12.6114088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6114160Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6114433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6114522Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6114767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6114847Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6115145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6115242Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6115529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.6115606Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6115625Z 2025-08-14T21:50:12.6115735Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6115940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6116004Z return mod(**inputs) 2025-08-14T21:50:12.6116294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6116379Z outputs = self.bert( 2025-08-14T21:50:12.6116660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6116742Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6117020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6117096Z layer_outputs = layer_module( 2025-08-14T21:50:12.6117308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6117384Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6117715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6117797Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6118038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6118117Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6118422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6118529Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6118823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.6118935Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.6119151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.6119219Z return self.act(input) 2025-08-14T21:50:12.6119222Z 2025-08-14T21:50:12.6119327Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6119517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6119580Z return mod(**inputs) 2025-08-14T21:50:12.6119864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6119928Z outputs = self.bert( 2025-08-14T21:50:12.6120204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6120284Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6120563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6120640Z layer_outputs = layer_module( 2025-08-14T21:50:12.6120854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6120928Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6121213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6121291Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6121546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6121621Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6121926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.6122078Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.6122355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.6122444Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6122448Z 2025-08-14T21:50:12.6122546Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6122736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6122808Z return mod(**inputs) 2025-08-14T21:50:12.6123090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6123157Z outputs = self.bert( 2025-08-14T21:50:12.6123450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6123543Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6123859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6123932Z layer_outputs = layer_module( 2025-08-14T21:50:12.6124152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6124238Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6124524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6124616Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6124926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6125005Z self_outputs = self.self( 2025-08-14T21:50:12.6125279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6125354Z return func(*args, **kwargs) 2025-08-14T21:50:12.6125867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.6125965Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.6125969Z 2025-08-14T21:50:12.6126072Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6126278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6126345Z return mod(**inputs) 2025-08-14T21:50:12.6126632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6126715Z outputs = self.bert( 2025-08-14T21:50:12.6127002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6127090Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6127372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6127449Z layer_outputs = layer_module( 2025-08-14T21:50:12.6127690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6127773Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6128075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6128174Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6128499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6128584Z self_outputs = self.self( 2025-08-14T21:50:12.6128840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6128916Z return func(*args, **kwargs) 2025-08-14T21:50:12.6129228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.6129309Z key_layer = self.key(current_states) 2025-08-14T21:50:12.6129313Z 2025-08-14T21:50:12.6129430Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6129639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6129707Z return mod(**inputs) 2025-08-14T21:50:12.6130017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6130088Z outputs = self.bert( 2025-08-14T21:50:12.6130427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6130513Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6130814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6130897Z layer_outputs = layer_module( 2025-08-14T21:50:12.6131128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6131210Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6131520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6131628Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6131934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6132020Z self_outputs = self.self( 2025-08-14T21:50:12.6132273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6132359Z return func(*args, **kwargs) 2025-08-14T21:50:12.6132659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.6132742Z value_layer = self.value(current_states) 2025-08-14T21:50:12.6132745Z 2025-08-14T21:50:12.6132839Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6132924Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6133037Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6133248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6133317Z return mod(**inputs) 2025-08-14T21:50:12.6133634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6133703Z outputs = self.bert( 2025-08-14T21:50:12.6134001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6134085Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6134387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6134468Z layer_outputs = layer_module( 2025-08-14T21:50:12.6134700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6134781Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6135110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6135197Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6135503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.6135633Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.6135929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.6136021Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6136025Z 2025-08-14T21:50:12.6136133Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6136338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6136419Z return mod(**inputs) 2025-08-14T21:50:12.6136717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6136838Z outputs = self.bert( 2025-08-14T21:50:12.6137139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6137215Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6137518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6137592Z layer_outputs = layer_module( 2025-08-14T21:50:12.6138177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6138282Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6138849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6139000Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6139305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6139382Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6139708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6139810Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6140105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.6140189Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6140195Z 2025-08-14T21:50:12.6140302Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6140518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6140586Z return mod(**inputs) 2025-08-14T21:50:12.6140890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6140957Z outputs = self.bert( 2025-08-14T21:50:12.6141246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6141328Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6141617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6141688Z layer_outputs = layer_module( 2025-08-14T21:50:12.6141918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6142032Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6142326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6142409Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6142649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6142731Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6143024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6143127Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6143394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.6143502Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.6143708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.6143861Z return self.act(input) 2025-08-14T21:50:12.6143865Z 2025-08-14T21:50:12.6143962Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6144154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6144216Z return mod(**inputs) 2025-08-14T21:50:12.6144494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6144557Z outputs = self.bert( 2025-08-14T21:50:12.6144824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6144914Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6145199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6145280Z layer_outputs = layer_module( 2025-08-14T21:50:12.6145493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6145568Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6145846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6145925Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6146170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6146249Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6146548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.6146684Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.6146961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.6147040Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6147043Z 2025-08-14T21:50:12.6147148Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6147338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6147409Z return mod(**inputs) 2025-08-14T21:50:12.6147689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6147752Z outputs = self.bert( 2025-08-14T21:50:12.6148034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6148124Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6148418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6148497Z layer_outputs = layer_module( 2025-08-14T21:50:12.6148719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6148803Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6149094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6149177Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6149445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6149523Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6149853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.6150032Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.6150317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:12.6150398Z return input_tensor + hidden_states 2025-08-14T21:50:12.6150401Z 2025-08-14T21:50:12.6150496Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6150691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6150753Z return mod(**inputs) 2025-08-14T21:50:12.6151023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6151109Z outputs = self.bert( 2025-08-14T21:50:12.6151384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6151454Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6151736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6151803Z layer_outputs = layer_module( 2025-08-14T21:50:12.6152017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6152090Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6152362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6152448Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6152722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6152797Z self_outputs = self.self( 2025-08-14T21:50:12.6153030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6153095Z return func(*args, **kwargs) 2025-08-14T21:50:12.6153373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.6153448Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.6153452Z 2025-08-14T21:50:12.6153546Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6153739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6153800Z return mod(**inputs) 2025-08-14T21:50:12.6154082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6154160Z outputs = self.bert( 2025-08-14T21:50:12.6154434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6154513Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6154785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6154858Z layer_outputs = layer_module( 2025-08-14T21:50:12.6155064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6155137Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6155412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6155491Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6155763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6155871Z self_outputs = self.self( 2025-08-14T21:50:12.6156104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6156178Z return func(*args, **kwargs) 2025-08-14T21:50:12.6156449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.6156523Z key_layer = self.key(current_states) 2025-08-14T21:50:12.6156527Z 2025-08-14T21:50:12.6156633Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6156819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6156882Z return mod(**inputs) 2025-08-14T21:50:12.6157180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6157245Z outputs = self.bert( 2025-08-14T21:50:12.6157529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6157597Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6157859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6157934Z layer_outputs = layer_module( 2025-08-14T21:50:12.6158138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6158218Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6158487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6158564Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6158842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6158910Z self_outputs = self.self( 2025-08-14T21:50:12.6159136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6159214Z return func(*args, **kwargs) 2025-08-14T21:50:12.6159492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.6159575Z value_layer = self.value(current_states) 2025-08-14T21:50:12.6159578Z 2025-08-14T21:50:12.6159656Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6159736Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6159862Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6160055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6160127Z return mod(**inputs) 2025-08-14T21:50:12.6160410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6160474Z outputs = self.bert( 2025-08-14T21:50:12.6160758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6160829Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6161124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6161204Z layer_outputs = layer_module( 2025-08-14T21:50:12.6161418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6161500Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6161812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6161892Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6162187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.6162307Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.6162591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.6162669Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6162673Z 2025-08-14T21:50:12.6162769Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6162981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6163045Z return mod(**inputs) 2025-08-14T21:50:12.6163324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6163397Z outputs = self.bert( 2025-08-14T21:50:12.6163672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6163751Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6164028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6164096Z layer_outputs = layer_module( 2025-08-14T21:50:12.6164316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6164394Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6164678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6164762Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6165015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6165098Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6165417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6165615Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6165942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.6166028Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6166064Z 2025-08-14T21:50:12.6166183Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6166398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6166468Z return mod(**inputs) 2025-08-14T21:50:12.6166777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6166853Z outputs = self.bert( 2025-08-14T21:50:12.6167121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6167191Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6167453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6167528Z layer_outputs = layer_module( 2025-08-14T21:50:12.6167732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6167837Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6168133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6168212Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6168464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6168537Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6168833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6168936Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6169215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.6169329Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.6169531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.6169598Z return self.act(input) 2025-08-14T21:50:12.6169602Z 2025-08-14T21:50:12.6169703Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6169889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6169952Z return mod(**inputs) 2025-08-14T21:50:12.6170231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6170298Z outputs = self.bert( 2025-08-14T21:50:12.6170575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6170649Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6170922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6170999Z layer_outputs = layer_module( 2025-08-14T21:50:12.6171205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6171287Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6171556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6171634Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6171884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6171958Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6172288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.6172421Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.6172685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.6172769Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6172772Z 2025-08-14T21:50:12.6172867Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6173049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6173119Z return mod(**inputs) 2025-08-14T21:50:12.6173388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6173458Z outputs = self.bert( 2025-08-14T21:50:12.6173725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6173828Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6174153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6174221Z layer_outputs = layer_module( 2025-08-14T21:50:12.6174427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6174509Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6174776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6174861Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6175147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6175217Z self_outputs = self.self( 2025-08-14T21:50:12.6175459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6175528Z return func(*args, **kwargs) 2025-08-14T21:50:12.6175811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.6175887Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.6175891Z 2025-08-14T21:50:12.6175985Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6176178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6176240Z return mod(**inputs) 2025-08-14T21:50:12.6176514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6176586Z outputs = self.bert( 2025-08-14T21:50:12.6176860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6176939Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6177210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6177276Z layer_outputs = layer_module( 2025-08-14T21:50:12.6177495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6177570Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6177852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6177928Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6178216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6178290Z self_outputs = self.self( 2025-08-14T21:50:12.6178525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6178594Z return func(*args, **kwargs) 2025-08-14T21:50:12.6178875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.6178950Z key_layer = self.key(current_states) 2025-08-14T21:50:12.6178953Z 2025-08-14T21:50:12.6179056Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6179243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6179305Z return mod(**inputs) 2025-08-14T21:50:12.6179584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6179664Z outputs = self.bert( 2025-08-14T21:50:12.6179960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6180031Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6180303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6180379Z layer_outputs = layer_module( 2025-08-14T21:50:12.6180585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6180657Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6180928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6181022Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6181301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6181369Z self_outputs = self.self( 2025-08-14T21:50:12.6181595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6181668Z return func(*args, **kwargs) 2025-08-14T21:50:12.6181937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.6182016Z value_layer = self.value(current_states) 2025-08-14T21:50:12.6182020Z 2025-08-14T21:50:12.6182097Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6182173Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6182276Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6182465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6182528Z return mod(**inputs) 2025-08-14T21:50:12.6182808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6182870Z outputs = self.bert( 2025-08-14T21:50:12.6183143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6183212Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6183480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6183553Z layer_outputs = layer_module( 2025-08-14T21:50:12.6183757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6183850Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6184126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6184204Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6184484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.6184603Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.6184900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.6184989Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6184992Z 2025-08-14T21:50:12.6185090Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6185286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6185354Z return mod(**inputs) 2025-08-14T21:50:12.6185652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6185740Z outputs = self.bert( 2025-08-14T21:50:12.6186019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6186088Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6186394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6186462Z layer_outputs = layer_module( 2025-08-14T21:50:12.6186684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6186758Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6187052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6187143Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6187395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6187478Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6187784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6187879Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6188153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.6188228Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6188231Z 2025-08-14T21:50:12.6188337Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6188524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6188587Z return mod(**inputs) 2025-08-14T21:50:12.6188867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6188931Z outputs = self.bert( 2025-08-14T21:50:12.6189198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6189275Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6189543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6189616Z layer_outputs = layer_module( 2025-08-14T21:50:12.6189825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6189918Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6190201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6190278Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6190533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6190605Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6190913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6191014Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6191281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.6191387Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.6191590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.6191683Z return self.act(input) 2025-08-14T21:50:12.6191687Z 2025-08-14T21:50:12.6191790Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6191974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6192036Z return mod(**inputs) 2025-08-14T21:50:12.6192315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6192377Z outputs = self.bert( 2025-08-14T21:50:12.6192652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6192746Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6193018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6193095Z layer_outputs = layer_module( 2025-08-14T21:50:12.6193301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6193374Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6193658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6193734Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6193977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6194046Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6194335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.6194465Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.6194726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.6194809Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6194813Z 2025-08-14T21:50:12.6194904Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6195088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6195157Z return mod(**inputs) 2025-08-14T21:50:12.6195419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6195480Z outputs = self.bert( 2025-08-14T21:50:12.6195750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6195835Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6196105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6196169Z layer_outputs = layer_module( 2025-08-14T21:50:12.6196368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6196446Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6196706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6196788Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6197023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6197096Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6197403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.6197537Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.6197799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:12.6197877Z return input_tensor + hidden_states 2025-08-14T21:50:12.6197881Z 2025-08-14T21:50:12.6197973Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6198161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6198220Z return mod(**inputs) 2025-08-14T21:50:12.6198499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6198571Z outputs = self.bert( 2025-08-14T21:50:12.6198838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6198913Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6199177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6199243Z layer_outputs = layer_module( 2025-08-14T21:50:12.6199453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6199525Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6199787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6199870Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6200139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6200211Z self_outputs = self.self( 2025-08-14T21:50:12.6200437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6200502Z return func(*args, **kwargs) 2025-08-14T21:50:12.6200777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.6200851Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.6200855Z 2025-08-14T21:50:12.6200956Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6201141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6201201Z return mod(**inputs) 2025-08-14T21:50:12.6201478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6201557Z outputs = self.bert( 2025-08-14T21:50:12.6201822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6201897Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6202156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6202230Z layer_outputs = layer_module( 2025-08-14T21:50:12.6202430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6202502Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6202772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6202850Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6203136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6203217Z self_outputs = self.self( 2025-08-14T21:50:12.6203440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6203518Z return func(*args, **kwargs) 2025-08-14T21:50:12.6203784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.6203858Z key_layer = self.key(current_states) 2025-08-14T21:50:12.6203869Z 2025-08-14T21:50:12.6203965Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6204148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6204235Z return mod(**inputs) 2025-08-14T21:50:12.6204510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6204577Z outputs = self.bert( 2025-08-14T21:50:12.6204858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6204930Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6205213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6205282Z layer_outputs = layer_module( 2025-08-14T21:50:12.6205615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6205749Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6206032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6206113Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6206400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6206472Z self_outputs = self.self( 2025-08-14T21:50:12.6206721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6206791Z return func(*args, **kwargs) 2025-08-14T21:50:12.6207076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.6207166Z value_layer = self.value(current_states) 2025-08-14T21:50:12.6207170Z 2025-08-14T21:50:12.6207251Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6207341Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6207472Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6207670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6207747Z return mod(**inputs) 2025-08-14T21:50:12.6208034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6208098Z outputs = self.bert( 2025-08-14T21:50:12.6208378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6208448Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6208730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6208798Z layer_outputs = layer_module( 2025-08-14T21:50:12.6209006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6209091Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6209394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6209471Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6209745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.6209862Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.6210135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.6210212Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6210216Z 2025-08-14T21:50:12.6210310Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6210523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6210588Z return mod(**inputs) 2025-08-14T21:50:12.6210877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6210939Z outputs = self.bert( 2025-08-14T21:50:12.6211210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6211285Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6211555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6211622Z layer_outputs = layer_module( 2025-08-14T21:50:12.6211838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6211913Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6212194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6212274Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6212518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6212596Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6212896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6212998Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6213271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.6213348Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6213369Z 2025-08-14T21:50:12.6213473Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6213666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6213733Z return mod(**inputs) 2025-08-14T21:50:12.6214007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6214070Z outputs = self.bert( 2025-08-14T21:50:12.6214350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6214420Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6214691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6214766Z layer_outputs = layer_module( 2025-08-14T21:50:12.6214976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6215077Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6215367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6215447Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6215698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6215770Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6216076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6216171Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6216460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.6216576Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.6216781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.6216849Z return self.act(input) 2025-08-14T21:50:12.6216853Z 2025-08-14T21:50:12.6216959Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6217153Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6217226Z return mod(**inputs) 2025-08-14T21:50:12.6217505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6217570Z outputs = self.bert( 2025-08-14T21:50:12.6217918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6217989Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6218269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6218337Z layer_outputs = layer_module( 2025-08-14T21:50:12.6218541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6218624Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6218892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6218969Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6219218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6219291Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6219618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.6219745Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.6220015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.6220100Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6220103Z 2025-08-14T21:50:12.6220201Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6220395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6220457Z return mod(**inputs) 2025-08-14T21:50:12.6220729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6220802Z outputs = self.bert( 2025-08-14T21:50:12.6221068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6221173Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6221450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6221516Z layer_outputs = layer_module( 2025-08-14T21:50:12.6221728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6221801Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6222066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6222147Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6222428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6222504Z self_outputs = self.self( 2025-08-14T21:50:12.6222733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6222800Z return func(*args, **kwargs) 2025-08-14T21:50:12.6223076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.6223152Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.6223156Z 2025-08-14T21:50:12.6223251Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6223444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6223507Z return mod(**inputs) 2025-08-14T21:50:12.6223789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6223852Z outputs = self.bert( 2025-08-14T21:50:12.6224122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6224199Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6224473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6224547Z layer_outputs = layer_module( 2025-08-14T21:50:12.6224752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6224825Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6225091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6225167Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6225445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6225522Z self_outputs = self.self( 2025-08-14T21:50:12.6225746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6225821Z return func(*args, **kwargs) 2025-08-14T21:50:12.6226089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.6226161Z key_layer = self.key(current_states) 2025-08-14T21:50:12.6226165Z 2025-08-14T21:50:12.6226267Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6226450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6226519Z return mod(**inputs) 2025-08-14T21:50:12.6226794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6226873Z outputs = self.bert( 2025-08-14T21:50:12.6227169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6227239Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6227512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6227584Z layer_outputs = layer_module( 2025-08-14T21:50:12.6227794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6227874Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6228165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6228243Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6228531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6228597Z self_outputs = self.self( 2025-08-14T21:50:12.6228832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6228908Z return func(*args, **kwargs) 2025-08-14T21:50:12.6229185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.6229268Z value_layer = self.value(current_states) 2025-08-14T21:50:12.6229272Z 2025-08-14T21:50:12.6229348Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6229426Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6229546Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6229735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6229804Z return mod(**inputs) 2025-08-14T21:50:12.6230083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6230144Z outputs = self.bert( 2025-08-14T21:50:12.6230422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6230491Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6230765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6230840Z layer_outputs = layer_module( 2025-08-14T21:50:12.6231051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6231147Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6231419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6231496Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6250315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.6250647Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.6250993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.6251084Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6251090Z 2025-08-14T21:50:12.6251209Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6251441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6251518Z return mod(**inputs) 2025-08-14T21:50:12.6252030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6252103Z outputs = self.bert( 2025-08-14T21:50:12.6252393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6252482Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6252771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6252853Z layer_outputs = layer_module( 2025-08-14T21:50:12.6253076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6253227Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6253520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6253609Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6253871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6253951Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6254259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6254372Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6254647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.6254737Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6254743Z 2025-08-14T21:50:12.6254851Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6255051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6255130Z return mod(**inputs) 2025-08-14T21:50:12.6255411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6255478Z outputs = self.bert( 2025-08-14T21:50:12.6255763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6255841Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6256129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6256201Z layer_outputs = layer_module( 2025-08-14T21:50:12.6256419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6256557Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6256839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6256921Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6257185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6257261Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6257576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6257679Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6257958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.6258080Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.6258324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.6258407Z return self.act(input) 2025-08-14T21:50:12.6258411Z 2025-08-14T21:50:12.6258517Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6258717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6258790Z return mod(**inputs) 2025-08-14T21:50:12.6259072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6259146Z outputs = self.bert( 2025-08-14T21:50:12.6259427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6259513Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6259784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6259856Z layer_outputs = layer_module( 2025-08-14T21:50:12.6260069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6260153Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6260422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6260505Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6260755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6260833Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6261135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.6261273Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.6261547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.6261628Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6261632Z 2025-08-14T21:50:12.6261730Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6261915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6261984Z return mod(**inputs) 2025-08-14T21:50:12.6262256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6262320Z outputs = self.bert( 2025-08-14T21:50:12.6262614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6262687Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6262962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6263032Z layer_outputs = layer_module( 2025-08-14T21:50:12.6263238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6263318Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6263583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6263668Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6263909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6263987Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6264327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.6264455Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.6264726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:12.6264809Z return input_tensor + hidden_states 2025-08-14T21:50:12.6264812Z 2025-08-14T21:50:12.6264911Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6265105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6265167Z return mod(**inputs) 2025-08-14T21:50:12.6265452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6265526Z outputs = self.bert( 2025-08-14T21:50:12.6265798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6265873Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6266144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6266212Z layer_outputs = layer_module( 2025-08-14T21:50:12.6266424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6266497Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6266767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6266857Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6267124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6267204Z self_outputs = self.self( 2025-08-14T21:50:12.6267436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6267506Z return func(*args, **kwargs) 2025-08-14T21:50:12.6267781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.6267859Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.6267863Z 2025-08-14T21:50:12.6267966Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6268154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6268216Z return mod(**inputs) 2025-08-14T21:50:12.6268511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6268575Z outputs = self.bert( 2025-08-14T21:50:12.6268840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6268918Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6269180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6269255Z layer_outputs = layer_module( 2025-08-14T21:50:12.6269459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6269532Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6269805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6269884Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6270195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6270266Z self_outputs = self.self( 2025-08-14T21:50:12.6270494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6270571Z return func(*args, **kwargs) 2025-08-14T21:50:12.6270845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.6270918Z key_layer = self.key(current_states) 2025-08-14T21:50:12.6270930Z 2025-08-14T21:50:12.6271026Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6271227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6271299Z return mod(**inputs) 2025-08-14T21:50:12.6271572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6271635Z outputs = self.bert( 2025-08-14T21:50:12.6271909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6271977Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6272249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6272316Z layer_outputs = layer_module( 2025-08-14T21:50:12.6272521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6272601Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6272870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6272949Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6273222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6273288Z self_outputs = self.self( 2025-08-14T21:50:12.6273520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6273588Z return func(*args, **kwargs) 2025-08-14T21:50:12.6273856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.6273938Z value_layer = self.value(current_states) 2025-08-14T21:50:12.6273942Z 2025-08-14T21:50:12.6274020Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6274134Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6274232Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6274421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6274491Z return mod(**inputs) 2025-08-14T21:50:12.6274764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6274826Z outputs = self.bert( 2025-08-14T21:50:12.6275103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6275174Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6275450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6275527Z layer_outputs = layer_module( 2025-08-14T21:50:12.6275732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6275830Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6276110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6276187Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6276460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.6276581Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.6276856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.6276933Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6276937Z 2025-08-14T21:50:12.6277058Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6277256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6277320Z return mod(**inputs) 2025-08-14T21:50:12.6277596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6277660Z outputs = self.bert( 2025-08-14T21:50:12.6277938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6278016Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6278279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6278345Z layer_outputs = layer_module( 2025-08-14T21:50:12.6278555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6278628Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6278900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6278978Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6279221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6279300Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6279597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6279704Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6279974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.6280072Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6280075Z 2025-08-14T21:50:12.6280180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6280371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6280440Z return mod(**inputs) 2025-08-14T21:50:12.6280713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6280775Z outputs = self.bert( 2025-08-14T21:50:12.6281050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6281119Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6281389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6281467Z layer_outputs = layer_module( 2025-08-14T21:50:12.6281676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6281788Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6282054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6282130Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6282379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6282452Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6282764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6282864Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6283155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.6283277Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.6283487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.6283556Z return self.act(input) 2025-08-14T21:50:12.6283559Z 2025-08-14T21:50:12.6283670Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6283867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6283939Z return mod(**inputs) 2025-08-14T21:50:12.6284228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6284305Z outputs = self.bert( 2025-08-14T21:50:12.6284590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6284665Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6284949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6285017Z layer_outputs = layer_module( 2025-08-14T21:50:12.6285229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6285312Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6285710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6285806Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6286091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6286198Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6286547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.6286691Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.6286999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.6287089Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6287093Z 2025-08-14T21:50:12.6287196Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6287415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6287478Z return mod(**inputs) 2025-08-14T21:50:12.6287759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6287832Z outputs = self.bert( 2025-08-14T21:50:12.6288123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6288210Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6288485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6288555Z layer_outputs = layer_module( 2025-08-14T21:50:12.6288766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6288839Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6289106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6289190Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6289473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6289547Z self_outputs = self.self( 2025-08-14T21:50:12.6289779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6289848Z return func(*args, **kwargs) 2025-08-14T21:50:12.6290130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.6290209Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.6290213Z 2025-08-14T21:50:12.6290311Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6290509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6290573Z return mod(**inputs) 2025-08-14T21:50:12.6290860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6290926Z outputs = self.bert( 2025-08-14T21:50:12.6291212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6291291Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6291558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6291633Z layer_outputs = layer_module( 2025-08-14T21:50:12.6291838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6291911Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6292185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6292282Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6292559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6292637Z self_outputs = self.self( 2025-08-14T21:50:12.6292872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6292951Z return func(*args, **kwargs) 2025-08-14T21:50:12.6293226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.6293302Z key_layer = self.key(current_states) 2025-08-14T21:50:12.6293305Z 2025-08-14T21:50:12.6293413Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6293603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6293677Z return mod(**inputs) 2025-08-14T21:50:12.6293956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6295042Z outputs = self.bert( 2025-08-14T21:50:12.6295333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6295404Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6295681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6295756Z layer_outputs = layer_module( 2025-08-14T21:50:12.6295967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6296047Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6296338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6296417Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6296696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6296762Z self_outputs = self.self( 2025-08-14T21:50:12.6296991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6297065Z return func(*args, **kwargs) 2025-08-14T21:50:12.6297333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.6297414Z value_layer = self.value(current_states) 2025-08-14T21:50:12.6297418Z 2025-08-14T21:50:12.6297496Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6297569Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6297676Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6297864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6297938Z return mod(**inputs) 2025-08-14T21:50:12.6298210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6298276Z outputs = self.bert( 2025-08-14T21:50:12.6298558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6298629Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6298915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6298989Z layer_outputs = layer_module( 2025-08-14T21:50:12.6299198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6299298Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6299571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6299646Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6299923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.6300044Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.6300322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.6300400Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6300404Z 2025-08-14T21:50:12.6300500Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6300700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6300781Z return mod(**inputs) 2025-08-14T21:50:12.6301076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6301149Z outputs = self.bert( 2025-08-14T21:50:12.6301417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6301495Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6301765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6301832Z layer_outputs = layer_module( 2025-08-14T21:50:12.6302048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6302140Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6302421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6302502Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6302746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6302827Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6303124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6303219Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6303495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.6303571Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6303577Z 2025-08-14T21:50:12.6303679Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6303867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6303929Z return mod(**inputs) 2025-08-14T21:50:12.6304206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6304268Z outputs = self.bert( 2025-08-14T21:50:12.6304544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6304611Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6304877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6304953Z layer_outputs = layer_module( 2025-08-14T21:50:12.6305178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6305253Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6305525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6305602Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6305851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6305922Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6306216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6306319Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6306586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.6306700Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.6306932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.6307002Z return self.act(input) 2025-08-14T21:50:12.6307006Z 2025-08-14T21:50:12.6307110Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6307299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6307361Z return mod(**inputs) 2025-08-14T21:50:12.6307639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6307702Z outputs = self.bert( 2025-08-14T21:50:12.6308000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6308074Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6308343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6308419Z layer_outputs = layer_module( 2025-08-14T21:50:12.6308626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6308706Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6308974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6309052Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6309303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6309376Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6309681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.6309822Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.6310103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.6310180Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6310183Z 2025-08-14T21:50:12.6310285Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6310471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6310533Z return mod(**inputs) 2025-08-14T21:50:12.6310808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6310870Z outputs = self.bert( 2025-08-14T21:50:12.6311160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6311239Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6311506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6311580Z layer_outputs = layer_module( 2025-08-14T21:50:12.6311794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6311865Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6312132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6312208Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6312447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6312525Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6312848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.6312979Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.6313240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:12.6313312Z return input_tensor + hidden_states 2025-08-14T21:50:12.6313316Z 2025-08-14T21:50:12.6313417Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6313598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6313666Z return mod(**inputs) 2025-08-14T21:50:12.6313946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6314010Z outputs = self.bert( 2025-08-14T21:50:12.6314284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6314352Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6314616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6314691Z layer_outputs = layer_module( 2025-08-14T21:50:12.6314894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6314973Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6315242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6315321Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6315594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6315660Z self_outputs = self.self( 2025-08-14T21:50:12.6315894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6315962Z return func(*args, **kwargs) 2025-08-14T21:50:12.6316228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.6316313Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.6316317Z 2025-08-14T21:50:12.6316411Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6316600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6316680Z return mod(**inputs) 2025-08-14T21:50:12.6316948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6317023Z outputs = self.bert( 2025-08-14T21:50:12.6317286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6317356Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6317626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6317691Z layer_outputs = layer_module( 2025-08-14T21:50:12.6317902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6317978Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6318239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6318326Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6318633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6318700Z self_outputs = self.self( 2025-08-14T21:50:12.6318939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6319007Z return func(*args, **kwargs) 2025-08-14T21:50:12.6319285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.6319359Z key_layer = self.key(current_states) 2025-08-14T21:50:12.6319362Z 2025-08-14T21:50:12.6319459Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6319671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6319736Z return mod(**inputs) 2025-08-14T21:50:12.6320018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6320081Z outputs = self.bert( 2025-08-14T21:50:12.6320354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6320433Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6320703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6320771Z layer_outputs = layer_module( 2025-08-14T21:50:12.6320985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6321059Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6321338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6321417Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6321690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6321766Z self_outputs = self.self( 2025-08-14T21:50:12.6321996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6322073Z return func(*args, **kwargs) 2025-08-14T21:50:12.6322344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.6322419Z value_layer = self.value(current_states) 2025-08-14T21:50:12.6322422Z 2025-08-14T21:50:12.6322507Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6322600Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6322694Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6322892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6322955Z return mod(**inputs) 2025-08-14T21:50:12.6323239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6323301Z outputs = self.bert( 2025-08-14T21:50:12.6323571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6323650Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6323924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6323991Z layer_outputs = layer_module( 2025-08-14T21:50:12.6324205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6324303Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6324596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6324672Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6324949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.6325081Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.6325360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.6325522Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6325532Z 2025-08-14T21:50:12.6325668Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6325879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6325961Z return mod(**inputs) 2025-08-14T21:50:12.6326263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6326339Z outputs = self.bert( 2025-08-14T21:50:12.6326650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6326727Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6327037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6327107Z layer_outputs = layer_module( 2025-08-14T21:50:12.6327327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6327414Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6327701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6327793Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6328050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6328130Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6328469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6328576Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6328892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.6328999Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6329005Z 2025-08-14T21:50:12.6329111Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6329329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6329409Z return mod(**inputs) 2025-08-14T21:50:12.6329702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6329775Z outputs = self.bert( 2025-08-14T21:50:12.6330063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6330144Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6330435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6330509Z layer_outputs = layer_module( 2025-08-14T21:50:12.6330746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6330874Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6331183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6331277Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6331533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6331619Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6331932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6332034Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6332343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.6332461Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.6332679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.6332750Z return self.act(input) 2025-08-14T21:50:12.6332754Z 2025-08-14T21:50:12.6332855Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6333062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6333127Z return mod(**inputs) 2025-08-14T21:50:12.6333422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6333489Z outputs = self.bert( 2025-08-14T21:50:12.6333778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6333861Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6334147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6334217Z layer_outputs = layer_module( 2025-08-14T21:50:12.6334444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6334521Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6334811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6334891Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6335149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6335250Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6335566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.6335707Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.6335989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.6336070Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6336074Z 2025-08-14T21:50:12.6336183Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6336381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6336446Z return mod(**inputs) 2025-08-14T21:50:12.6336740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6336806Z outputs = self.bert( 2025-08-14T21:50:12.6337128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6337203Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6337488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6337568Z layer_outputs = layer_module( 2025-08-14T21:50:12.6338106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6338250Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6338643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6338733Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6339106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6339187Z self_outputs = self.self( 2025-08-14T21:50:12.6339448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6339534Z return func(*args, **kwargs) 2025-08-14T21:50:12.6339831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:12.6339923Z query_layer = self.query(hidden_states) 2025-08-14T21:50:12.6339927Z 2025-08-14T21:50:12.6340034Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6340251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6340338Z return mod(**inputs) 2025-08-14T21:50:12.6340627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6340704Z outputs = self.bert( 2025-08-14T21:50:12.6340989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6341063Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6341356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6341430Z layer_outputs = layer_module( 2025-08-14T21:50:12.6341648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6341734Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6342020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6342138Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6342424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6342495Z self_outputs = self.self( 2025-08-14T21:50:12.6342744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6342816Z return func(*args, **kwargs) 2025-08-14T21:50:12.6343103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:12.6343181Z key_layer = self.key(current_states) 2025-08-14T21:50:12.6343185Z 2025-08-14T21:50:12.6343283Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6343482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6343549Z return mod(**inputs) 2025-08-14T21:50:12.6343834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6343963Z outputs = self.bert( 2025-08-14T21:50:12.6344251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6344331Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6344612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6344682Z layer_outputs = layer_module( 2025-08-14T21:50:12.6344909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6344982Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6345272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6345353Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6345621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:12.6345693Z self_outputs = self.self( 2025-08-14T21:50:12.6345917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:12.6345984Z return func(*args, **kwargs) 2025-08-14T21:50:12.6346259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:12.6346333Z value_layer = self.value(current_states) 2025-08-14T21:50:12.6346336Z 2025-08-14T21:50:12.6346421Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6346497Z cudagraph partition due to non gpu ops 2025-08-14T21:50:12.6346595Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6346790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6346853Z return mod(**inputs) 2025-08-14T21:50:12.6347124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6347194Z outputs = self.bert( 2025-08-14T21:50:12.6347467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6347543Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6347809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6347877Z layer_outputs = layer_module( 2025-08-14T21:50:12.6348093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6348194Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6348476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:12.6348552Z self_attention_outputs = self.attention( 2025-08-14T21:50:12.6348825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:12.6348953Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:12.6349232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:12.6349311Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6349323Z 2025-08-14T21:50:12.6349421Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6349615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6349702Z return mod(**inputs) 2025-08-14T21:50:12.6350042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6350107Z outputs = self.bert( 2025-08-14T21:50:12.6350386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6350454Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6350728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6350794Z layer_outputs = layer_module( 2025-08-14T21:50:12.6350997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6351095Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6351366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6351445Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6351694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6351767Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6352068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6352165Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6352430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:12.6352517Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6352520Z 2025-08-14T21:50:12.6352618Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6352813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6352875Z return mod(**inputs) 2025-08-14T21:50:12.6353144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6353215Z outputs = self.bert( 2025-08-14T21:50:12.6353480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6353550Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6353832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6353900Z layer_outputs = layer_module( 2025-08-14T21:50:12.6354133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6354208Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6354477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6354564Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6354808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6354887Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6355184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:12.6355280Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:12.6355557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:12.6355683Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:12.6355905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:12.6355974Z return self.act(input) 2025-08-14T21:50:12.6355978Z 2025-08-14T21:50:12.6356074Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6356269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6356330Z return mod(**inputs) 2025-08-14T21:50:12.6356611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6356679Z outputs = self.bert( 2025-08-14T21:50:12.6356956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6357036Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6357307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6357372Z layer_outputs = layer_module( 2025-08-14T21:50:12.6357585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6357656Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6357923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6358004Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6358241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6358319Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6358614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.6358739Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.6359011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:12.6359084Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6359087Z 2025-08-14T21:50:12.6359187Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6359370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6359430Z return mod(**inputs) 2025-08-14T21:50:12.6359706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1064, in forward 2025-08-14T21:50:12.6359785Z outputs = self.bert( 2025-08-14T21:50:12.6360055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:12.6360125Z encoder_outputs = self.encoder( 2025-08-14T21:50:12.6360390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:12.6360465Z layer_outputs = layer_module( 2025-08-14T21:50:12.6360669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:12.6360740Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:12.6361011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:12.6361086Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:12.6361338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:12.6361434Z return forward_fn(*input_tensors) 2025-08-14T21:50:12.6361753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:12.6361885Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:12.6362175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:12.6362248Z return input_tensor + hidden_states 2025-08-14T21:50:12.6362252Z 2025-08-14T21:50:12.6362348Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6362545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6362606Z return mod(**inputs) 2025-08-14T21:50:12.6362892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1082, in forward 2025-08-14T21:50:12.6362992Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:50:12.6363263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 652, in forward 2025-08-14T21:50:12.6363375Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:50:12.6363647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 640, in forward 2025-08-14T21:50:12.6363734Z hidden_states = self.transform(hidden_states) 2025-08-14T21:50:12.6364014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 615, in forward 2025-08-14T21:50:12.6364089Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:12.6364093Z 2025-08-14T21:50:12.6364200Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6364386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6364451Z return mod(**inputs) 2025-08-14T21:50:12.6364731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1082, in forward 2025-08-14T21:50:12.6364817Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:50:12.6365099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 652, in forward 2025-08-14T21:50:12.6365205Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:50:12.6365560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 641, in forward 2025-08-14T21:50:12.6365665Z hidden_states = self.decoder(hidden_states) 2025-08-14T21:50:12.6365690Z 2025-08-14T21:50:12.6365793Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:12.6365994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:12.6366077Z return mod(**inputs) 2025-08-14T21:50:12.6366388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1086, in forward 2025-08-14T21:50:12.6366474Z lm_loss = self.loss_function( 2025-08-14T21:50:12.6366729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-08-14T21:50:12.6366902Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-08-14T21:50:12.6367156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-08-14T21:50:12.6367360Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-08-14T21:50:12.6367365Z 2025-08-14T21:50:23.8429569Z Compilation time (from dynamo_timed): 23.528612828 2025-08-14T21:50:23.8470628Z pass 2025-08-14T21:50:23.8475968Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:50:23.8479268Z TIMING: _recursive_pre_grad_passes:0.0111 _recursive_joint_graph_passes:1.11296 _recursive_post_grad_passes:0.13396 async_compile.wait:0.87824 code_gen:9.86036 inductor_compile:11.97448 backend_compile:18.10656 gc:0.00079 entire_frame_compile:23.52861 total_wall_time:23.52861 2025-08-14T21:50:23.8480610Z STATS: call_* op count: 723 | FakeTensorMode.__torch_dispatch__:28473 | FakeTensor.__torch_dispatch__:8903 | ProxyTorchDispatchMode.__torch_dispatch__:10946 2025-08-14T21:50:23.8481135Z Dynamo produced 1 graphs covering 723 ops with 0 graph breaks (0 unique) 2025-08-14T21:50:29.6196832Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:50:29.6197895Z from pkg_resources import resource_filename 2025-08-14T21:50:30.2706353Z 2025-08-14T21:50:33.2970301Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:50:33.2971459Z loading model: 0it [00:03, ?it/s] 2025-08-14T21:50:33.3002103Z cpu eval MegatronBertForQuestionAnswering 2025-08-14T21:50:34.9829017Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:50:35.6310731Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:50:36.2323016Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:50:50.1796901Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1797668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1798169Z return mod(**inputs) 2025-08-14T21:50:50.1798741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1799321Z outputs = self.bert( 2025-08-14T21:50:50.1800443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1800999Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1801488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1801955Z layer_outputs = layer_module( 2025-08-14T21:50:50.1802348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1803143Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1803619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.1804136Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.1804694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.1805159Z self_outputs = self.self( 2025-08-14T21:50:50.1805756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.1806191Z return func(*args, **kwargs) 2025-08-14T21:50:50.1806664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.1807156Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.1807316Z 2025-08-14T21:50:50.1807450Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1807846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1808335Z return mod(**inputs) 2025-08-14T21:50:50.1808764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1809206Z outputs = self.bert( 2025-08-14T21:50:50.1809637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1810118Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1810565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1811022Z layer_outputs = layer_module( 2025-08-14T21:50:50.1811450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1811919Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1812381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.1812845Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.1813306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.1813743Z self_outputs = self.self( 2025-08-14T21:50:50.1814137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.1814540Z return func(*args, **kwargs) 2025-08-14T21:50:50.1814977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.1815439Z key_layer = self.key(current_states) 2025-08-14T21:50:50.1815592Z 2025-08-14T21:50:50.1815704Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1816099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1816448Z return mod(**inputs) 2025-08-14T21:50:50.1816870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1817305Z outputs = self.bert( 2025-08-14T21:50:50.1817734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1818180Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1818609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1819036Z layer_outputs = layer_module( 2025-08-14T21:50:50.1819436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1819811Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1820258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.1820718Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.1821184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.1821600Z self_outputs = self.self( 2025-08-14T21:50:50.1821961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.1822333Z return func(*args, **kwargs) 2025-08-14T21:50:50.1822735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.1823167Z value_layer = self.value(current_states) 2025-08-14T21:50:50.1823347Z 2025-08-14T21:50:50.1823455Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.1823690Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.1823932Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1824316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1824666Z return mod(**inputs) 2025-08-14T21:50:50.1825087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1825527Z outputs = self.bert( 2025-08-14T21:50:50.1825945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1826395Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1826856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1827304Z layer_outputs = layer_module( 2025-08-14T21:50:50.1827674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1828042Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1828498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.1828952Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.1829407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.1829909Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.1830421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.1830893Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.1831039Z 2025-08-14T21:50:50.1831159Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1831536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1831878Z return mod(**inputs) 2025-08-14T21:50:50.1832314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1832755Z outputs = self.bert( 2025-08-14T21:50:50.1833173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1833619Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1834076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1834573Z layer_outputs = layer_module( 2025-08-14T21:50:50.1834955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1835328Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1835770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.1836226Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.1836646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.1837140Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.1837827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.1838383Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.1838908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.1839409Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.1839564Z 2025-08-14T21:50:50.1839676Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1840061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1840396Z return mod(**inputs) 2025-08-14T21:50:50.1840822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1841264Z outputs = self.bert( 2025-08-14T21:50:50.1841671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1842201Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1842654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1843101Z layer_outputs = layer_module( 2025-08-14T21:50:50.1843458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1843844Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1844321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.1844774Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.1845198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.1845696Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.1846202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.1846737Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.1847223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.1847742Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.1848160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.1848520Z return self.act(input) 2025-08-14T21:50:50.1848646Z 2025-08-14T21:50:50.1848756Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1849142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1849490Z return mod(**inputs) 2025-08-14T21:50:50.1849911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1850402Z outputs = self.bert( 2025-08-14T21:50:50.1850834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1851300Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1851720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1852138Z layer_outputs = layer_module( 2025-08-14T21:50:50.1852480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1852836Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1853264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.1853714Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.1854166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.1854604Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.1855078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.1855628Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.1856106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.1856559Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.1856714Z 2025-08-14T21:50:50.1856825Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1857255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1857603Z return mod(**inputs) 2025-08-14T21:50:50.1858041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1858507Z outputs = self.bert( 2025-08-14T21:50:50.1858931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1859352Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1859779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1860206Z layer_outputs = layer_module( 2025-08-14T21:50:50.1860543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1860911Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1861356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.1861805Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.1862254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.1862701Z self_outputs = self.self( 2025-08-14T21:50:50.1863090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.1863499Z return func(*args, **kwargs) 2025-08-14T21:50:50.1863926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.1864386Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.1864528Z 2025-08-14T21:50:50.1864650Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1865048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1865392Z return mod(**inputs) 2025-08-14T21:50:50.1865822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1866263Z outputs = self.bert( 2025-08-14T21:50:50.1866680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1867125Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1867577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1868026Z layer_outputs = layer_module( 2025-08-14T21:50:50.1868396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1868789Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1869257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.1869727Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.1870178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.1870627Z self_outputs = self.self( 2025-08-14T21:50:50.1871012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.1871400Z return func(*args, **kwargs) 2025-08-14T21:50:50.1871853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.1872300Z key_layer = self.key(current_states) 2025-08-14T21:50:50.1872468Z 2025-08-14T21:50:50.1872593Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1872982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1873334Z return mod(**inputs) 2025-08-14T21:50:50.1873766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1874207Z outputs = self.bert( 2025-08-14T21:50:50.1874630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1875089Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1875533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1875979Z layer_outputs = layer_module( 2025-08-14T21:50:50.1876366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1876757Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1877200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.1877656Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.1878110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.1878551Z self_outputs = self.self( 2025-08-14T21:50:50.1878933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.1879333Z return func(*args, **kwargs) 2025-08-14T21:50:50.1879786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.1880285Z value_layer = self.value(current_states) 2025-08-14T21:50:50.1880433Z 2025-08-14T21:50:50.1880527Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.1880763Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.1881022Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1881408Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1881765Z return mod(**inputs) 2025-08-14T21:50:50.1882201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1882649Z outputs = self.bert( 2025-08-14T21:50:50.1883064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1883522Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1883980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1884464Z layer_outputs = layer_module( 2025-08-14T21:50:50.1884857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1885258Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1885824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.1886293Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.1886765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.1887291Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.1887840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.1888308Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.1888468Z 2025-08-14T21:50:50.1888583Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1888979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1889339Z return mod(**inputs) 2025-08-14T21:50:50.1889770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1890224Z outputs = self.bert( 2025-08-14T21:50:50.1890651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1891113Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1891573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1892028Z layer_outputs = layer_module( 2025-08-14T21:50:50.1892406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1892791Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1893250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.1893718Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.1894154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.1894576Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.1895062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.1895594Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.1896135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.1896600Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.1896752Z 2025-08-14T21:50:50.1896861Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1897239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1897574Z return mod(**inputs) 2025-08-14T21:50:50.1898000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1898442Z outputs = self.bert( 2025-08-14T21:50:50.1898867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1899319Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1899770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1900284Z layer_outputs = layer_module( 2025-08-14T21:50:50.1900650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1901042Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1901493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.1901947Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.1902366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.1902790Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.1903313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.1903829Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.1904313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.1904806Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.1905211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.1905584Z return self.act(input) 2025-08-14T21:50:50.1905708Z 2025-08-14T21:50:50.1905818Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1906203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1906548Z return mod(**inputs) 2025-08-14T21:50:50.1906970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1907419Z outputs = self.bert( 2025-08-14T21:50:50.1907843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1908294Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1908733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1909172Z layer_outputs = layer_module( 2025-08-14T21:50:50.1909539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1909913Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1910376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.1910834Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.1911289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.1911705Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.1912197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.1912756Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.1913284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.1913738Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.1913890Z 2025-08-14T21:50:50.1914003Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1914389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1914736Z return mod(**inputs) 2025-08-14T21:50:50.1915159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1915647Z outputs = self.bert( 2025-08-14T21:50:50.1916066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1916514Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1916962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1917413Z layer_outputs = layer_module( 2025-08-14T21:50:50.1917772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1918150Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1918620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.1919080Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.1919508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.1919936Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.1920414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.1920958Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.1921460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:50.1921919Z return input_tensor + hidden_states 2025-08-14T21:50:50.1922058Z 2025-08-14T21:50:50.1922178Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1922567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1922910Z return mod(**inputs) 2025-08-14T21:50:50.1923344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1923793Z outputs = self.bert( 2025-08-14T21:50:50.1924207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1924661Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1925105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1925661Z layer_outputs = layer_module( 2025-08-14T21:50:50.1926045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1926480Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1926943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.1927407Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.1927876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.1928337Z self_outputs = self.self( 2025-08-14T21:50:50.1928741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.1929145Z return func(*args, **kwargs) 2025-08-14T21:50:50.1929595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.1930068Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.1930221Z 2025-08-14T21:50:50.1930343Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1930774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1931130Z return mod(**inputs) 2025-08-14T21:50:50.1931562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1932007Z outputs = self.bert( 2025-08-14T21:50:50.1932437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1932892Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1933343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1933791Z layer_outputs = layer_module( 2025-08-14T21:50:50.1934191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1934587Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1935045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.1935506Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.1935966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.1936429Z self_outputs = self.self( 2025-08-14T21:50:50.1936807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.1937203Z return func(*args, **kwargs) 2025-08-14T21:50:50.1937820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.1938288Z key_layer = self.key(current_states) 2025-08-14T21:50:50.1938431Z 2025-08-14T21:50:50.1938544Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1938932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1939281Z return mod(**inputs) 2025-08-14T21:50:50.1939695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1940140Z outputs = self.bert( 2025-08-14T21:50:50.1940565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1941014Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1941448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1941955Z layer_outputs = layer_module( 2025-08-14T21:50:50.1942322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1942702Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1943135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.1943585Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.1944032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.1944456Z self_outputs = self.self( 2025-08-14T21:50:50.1944832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.1945206Z return func(*args, **kwargs) 2025-08-14T21:50:50.1945625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.1946078Z value_layer = self.value(current_states) 2025-08-14T21:50:50.1946217Z 2025-08-14T21:50:50.1946325Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.1946538Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.1946766Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1947129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1947455Z return mod(**inputs) 2025-08-14T21:50:50.1947868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1948263Z outputs = self.bert( 2025-08-14T21:50:50.1948648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1949092Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1949502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1949910Z layer_outputs = layer_module( 2025-08-14T21:50:50.1950250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1950595Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1950997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.1951415Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.1951829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.1952294Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.1952752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.1953171Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.1953303Z 2025-08-14T21:50:50.1953411Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1953758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1954065Z return mod(**inputs) 2025-08-14T21:50:50.1954451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1954856Z outputs = self.bert( 2025-08-14T21:50:50.1955231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1955643Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1956070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1956477Z layer_outputs = layer_module( 2025-08-14T21:50:50.1956806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1957153Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1957564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.1957984Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.1958369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.1959288Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.1959729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.1960190Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.1960704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.1961129Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.1961262Z 2025-08-14T21:50:50.1961371Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1961715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1962034Z return mod(**inputs) 2025-08-14T21:50:50.1962433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1962844Z outputs = self.bert( 2025-08-14T21:50:50.1963248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1963673Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1964099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1964535Z layer_outputs = layer_module( 2025-08-14T21:50:50.1964898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1965276Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1965806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.1966274Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.1966718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.1967137Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.1967611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.1968092Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.1968553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.1969015Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.1969389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.1969736Z return self.act(input) 2025-08-14T21:50:50.1969854Z 2025-08-14T21:50:50.1969960Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1970319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1970634Z return mod(**inputs) 2025-08-14T21:50:50.1971069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1971485Z outputs = self.bert( 2025-08-14T21:50:50.1971876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1972290Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1972701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1973115Z layer_outputs = layer_module( 2025-08-14T21:50:50.1973449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1973805Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1974226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.1974654Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.1975099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.1975496Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.1975943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.1976452Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.1976923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.1977355Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.1977493Z 2025-08-14T21:50:50.1977605Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1977977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1978305Z return mod(**inputs) 2025-08-14T21:50:50.1978702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1979117Z outputs = self.bert( 2025-08-14T21:50:50.1979500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1979902Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1980296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1980689Z layer_outputs = layer_module( 2025-08-14T21:50:50.1981007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1981346Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1981744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.1982140Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.1982547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.1982951Z self_outputs = self.self( 2025-08-14T21:50:50.1983301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.1983658Z return func(*args, **kwargs) 2025-08-14T21:50:50.1984065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.1984497Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.1984631Z 2025-08-14T21:50:50.1984766Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1985130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1985478Z return mod(**inputs) 2025-08-14T21:50:50.1985897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1986333Z outputs = self.bert( 2025-08-14T21:50:50.1986780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1987226Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1987677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1988120Z layer_outputs = layer_module( 2025-08-14T21:50:50.1988465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1988822Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1989328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.1989788Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.1990250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.1990694Z self_outputs = self.self( 2025-08-14T21:50:50.1991070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.1991479Z return func(*args, **kwargs) 2025-08-14T21:50:50.1991919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.1992392Z key_layer = self.key(current_states) 2025-08-14T21:50:50.1992527Z 2025-08-14T21:50:50.1992633Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.1993007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.1993323Z return mod(**inputs) 2025-08-14T21:50:50.1993701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.1994105Z outputs = self.bert( 2025-08-14T21:50:50.1994491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.1994907Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.1995316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.1995740Z layer_outputs = layer_module( 2025-08-14T21:50:50.1996079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.1996429Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.1996843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.1997269Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.1997696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.1998107Z self_outputs = self.self( 2025-08-14T21:50:50.1998471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.1998839Z return func(*args, **kwargs) 2025-08-14T21:50:50.1999252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.1999682Z value_layer = self.value(current_states) 2025-08-14T21:50:50.1999823Z 2025-08-14T21:50:50.1999902Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2000113Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2000334Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2000681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2000995Z return mod(**inputs) 2025-08-14T21:50:50.2001377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2001773Z outputs = self.bert( 2025-08-14T21:50:50.2002155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2002572Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2002973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2003401Z layer_outputs = layer_module( 2025-08-14T21:50:50.2003756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2004107Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2004508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2004928Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2005343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.2005913Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.2006454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.2006930Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2007079Z 2025-08-14T21:50:50.2007211Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2007570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2007884Z return mod(**inputs) 2025-08-14T21:50:50.2008270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2008678Z outputs = self.bert( 2025-08-14T21:50:50.2009058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2009474Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2009880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2010290Z layer_outputs = layer_module( 2025-08-14T21:50:50.2010623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2010973Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2011381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2011821Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2012216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2012598Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2013035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2013540Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2013987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.2014406Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2014539Z 2025-08-14T21:50:50.2014649Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2014995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2015317Z return mod(**inputs) 2025-08-14T21:50:50.2015709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2016108Z outputs = self.bert( 2025-08-14T21:50:50.2016493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2016915Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2017319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2017770Z layer_outputs = layer_module( 2025-08-14T21:50:50.2018111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2018457Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2018869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2019284Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2019676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2020060Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2020511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2020979Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2021417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.2021864Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.2022228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.2022569Z return self.act(input) 2025-08-14T21:50:50.2022680Z 2025-08-14T21:50:50.2022778Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2023120Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2023427Z return mod(**inputs) 2025-08-14T21:50:50.2023811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2024231Z outputs = self.bert( 2025-08-14T21:50:50.2024615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2025037Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2025449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2025871Z layer_outputs = layer_module( 2025-08-14T21:50:50.2026199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2026544Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2026952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2027405Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2027789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2028174Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2028607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2029087Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2029552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.2029965Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2030095Z 2025-08-14T21:50:50.2030200Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2030546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2030873Z return mod(**inputs) 2025-08-14T21:50:50.2031295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2031736Z outputs = self.bert( 2025-08-14T21:50:50.2032122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2032549Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2032949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2033353Z layer_outputs = layer_module( 2025-08-14T21:50:50.2033704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2034052Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2034480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2034893Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2035284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2035665Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2036099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2036585Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2037039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:50.2037452Z return input_tensor + hidden_states 2025-08-14T21:50:50.2037579Z 2025-08-14T21:50:50.2037816Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2038180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2038508Z return mod(**inputs) 2025-08-14T21:50:50.2038912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2039330Z outputs = self.bert( 2025-08-14T21:50:50.2039738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2040166Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2040583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2040999Z layer_outputs = layer_module( 2025-08-14T21:50:50.2041342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2041775Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2042199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2042634Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2043072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2043481Z self_outputs = self.self( 2025-08-14T21:50:50.2043862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2044261Z return func(*args, **kwargs) 2025-08-14T21:50:50.2044696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.2045156Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.2045302Z 2025-08-14T21:50:50.2045416Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2045941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2046291Z return mod(**inputs) 2025-08-14T21:50:50.2046710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2047133Z outputs = self.bert( 2025-08-14T21:50:50.2047528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2047949Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2048359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2048788Z layer_outputs = layer_module( 2025-08-14T21:50:50.2049166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2049514Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2049930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2050350Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2050769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2051224Z self_outputs = self.self( 2025-08-14T21:50:50.2051586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2051953Z return func(*args, **kwargs) 2025-08-14T21:50:50.2052356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.2052770Z key_layer = self.key(current_states) 2025-08-14T21:50:50.2052909Z 2025-08-14T21:50:50.2053022Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2053375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2053687Z return mod(**inputs) 2025-08-14T21:50:50.2054085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2054497Z outputs = self.bert( 2025-08-14T21:50:50.2054888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2055304Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2055721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2056159Z layer_outputs = layer_module( 2025-08-14T21:50:50.2056507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2056850Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2057263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2057680Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2058093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2058505Z self_outputs = self.self( 2025-08-14T21:50:50.2058859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2059219Z return func(*args, **kwargs) 2025-08-14T21:50:50.2059615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.2060058Z value_layer = self.value(current_states) 2025-08-14T21:50:50.2060208Z 2025-08-14T21:50:50.2060296Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2060499Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2060730Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2061082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2061397Z return mod(**inputs) 2025-08-14T21:50:50.2061780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2062193Z outputs = self.bert( 2025-08-14T21:50:50.2062620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2063036Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2063449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2063888Z layer_outputs = layer_module( 2025-08-14T21:50:50.2064251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2064601Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2065045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2065498Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2065920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.2066415Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.2066924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.2067381Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2067525Z 2025-08-14T21:50:50.2067641Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2068013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2068357Z return mod(**inputs) 2025-08-14T21:50:50.2068775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2069205Z outputs = self.bert( 2025-08-14T21:50:50.2069625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2070068Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2070542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2070984Z layer_outputs = layer_module( 2025-08-14T21:50:50.2071352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2071734Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2072176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2072636Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2073063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2073482Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2073954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2074477Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2075025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.2075462Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2075600Z 2025-08-14T21:50:50.2075706Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2076068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2076393Z return mod(**inputs) 2025-08-14T21:50:50.2076780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2077198Z outputs = self.bert( 2025-08-14T21:50:50.2077613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2078043Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2078454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2078878Z layer_outputs = layer_module( 2025-08-14T21:50:50.2079221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2079579Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2079988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2080419Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2080819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2081210Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2081662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2082153Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2082601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.2083059Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.2083441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.2083780Z return self.act(input) 2025-08-14T21:50:50.2083889Z 2025-08-14T21:50:50.2084007Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2084381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2084751Z return mod(**inputs) 2025-08-14T21:50:50.2085172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2085705Z outputs = self.bert( 2025-08-14T21:50:50.2086117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2086553Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2086998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2087444Z layer_outputs = layer_module( 2025-08-14T21:50:50.2087824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2088189Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2088617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2089045Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2089500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2089906Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2090345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2090851Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2091331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.2091765Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2091901Z 2025-08-14T21:50:50.2092006Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2092388Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2092715Z return mod(**inputs) 2025-08-14T21:50:50.2093115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2093525Z outputs = self.bert( 2025-08-14T21:50:50.2093919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2094343Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2094750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2095164Z layer_outputs = layer_module( 2025-08-14T21:50:50.2095509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2095875Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2096294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2096729Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2097160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2097575Z self_outputs = self.self( 2025-08-14T21:50:50.2097928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2098300Z return func(*args, **kwargs) 2025-08-14T21:50:50.2098706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.2099126Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.2099298Z 2025-08-14T21:50:50.2099404Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2099758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2100086Z return mod(**inputs) 2025-08-14T21:50:50.2100476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2100899Z outputs = self.bert( 2025-08-14T21:50:50.2101287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2101692Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2102091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2102500Z layer_outputs = layer_module( 2025-08-14T21:50:50.2102837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2103175Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2103635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2104067Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2104494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2104901Z self_outputs = self.self( 2025-08-14T21:50:50.2105264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2105639Z return func(*args, **kwargs) 2025-08-14T21:50:50.2106045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.2106481Z key_layer = self.key(current_states) 2025-08-14T21:50:50.2106619Z 2025-08-14T21:50:50.2106722Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2107083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2107388Z return mod(**inputs) 2025-08-14T21:50:50.2107777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2108181Z outputs = self.bert( 2025-08-14T21:50:50.2108566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2108973Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2109371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2109782Z layer_outputs = layer_module( 2025-08-14T21:50:50.2110113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2110475Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2110893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2111323Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2111740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2112155Z self_outputs = self.self( 2025-08-14T21:50:50.2112512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2112879Z return func(*args, **kwargs) 2025-08-14T21:50:50.2113278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.2113732Z value_layer = self.value(current_states) 2025-08-14T21:50:50.2113866Z 2025-08-14T21:50:50.2113959Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2114167Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2114406Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2114768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2115100Z return mod(**inputs) 2025-08-14T21:50:50.2115492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2115904Z outputs = self.bert( 2025-08-14T21:50:50.2116296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2116720Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2117142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2117612Z layer_outputs = layer_module( 2025-08-14T21:50:50.2117961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2118311Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2118737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2119167Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2119594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.2120073Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.2120567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.2121052Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2121191Z 2025-08-14T21:50:50.2121294Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2121652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2121978Z return mod(**inputs) 2025-08-14T21:50:50.2122376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2122788Z outputs = self.bert( 2025-08-14T21:50:50.2123181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2123602Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2124011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2124436Z layer_outputs = layer_module( 2025-08-14T21:50:50.2124797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2125184Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2125706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2126181Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2126619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2127030Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2127475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2127989Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2128442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.2128871Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2129007Z 2025-08-14T21:50:50.2129110Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2129466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2129792Z return mod(**inputs) 2025-08-14T21:50:50.2130179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2130597Z outputs = self.bert( 2025-08-14T21:50:50.2130993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2131407Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2131847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2132293Z layer_outputs = layer_module( 2025-08-14T21:50:50.2132642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2132982Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2133405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2133874Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2134297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2134705Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2135202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2135720Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2136168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.2136621Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.2137000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.2137341Z return self.act(input) 2025-08-14T21:50:50.2137452Z 2025-08-14T21:50:50.2137563Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2138092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2138422Z return mod(**inputs) 2025-08-14T21:50:50.2138827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2139242Z outputs = self.bert( 2025-08-14T21:50:50.2139646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2140060Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2140471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2140875Z layer_outputs = layer_module( 2025-08-14T21:50:50.2141215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2141570Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2141982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2142458Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2142851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2143232Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2143667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2144174Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2144722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.2145189Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2145341Z 2025-08-14T21:50:50.2145461Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2145820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2146158Z return mod(**inputs) 2025-08-14T21:50:50.2146626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2147037Z outputs = self.bert( 2025-08-14T21:50:50.2147428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2147841Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2148237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2148644Z layer_outputs = layer_module( 2025-08-14T21:50:50.2148979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2149361Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2149772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2150192Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2150583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2150960Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2151394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2151885Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2152356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:50.2152768Z return input_tensor + hidden_states 2025-08-14T21:50:50.2152909Z 2025-08-14T21:50:50.2153012Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2153372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2153704Z return mod(**inputs) 2025-08-14T21:50:50.2154093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2154508Z outputs = self.bert( 2025-08-14T21:50:50.2154900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2155315Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2155743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2156141Z layer_outputs = layer_module( 2025-08-14T21:50:50.2156476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2156859Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2157280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2157707Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2158128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2158529Z self_outputs = self.self( 2025-08-14T21:50:50.2158888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2159258Z return func(*args, **kwargs) 2025-08-14T21:50:50.2159653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.2160091Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.2160239Z 2025-08-14T21:50:50.2160373Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2160762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2161082Z return mod(**inputs) 2025-08-14T21:50:50.2161482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2161897Z outputs = self.bert( 2025-08-14T21:50:50.2162285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2162712Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2163133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2163590Z layer_outputs = layer_module( 2025-08-14T21:50:50.2163955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2164349Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2164807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2165274Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2165783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2166238Z self_outputs = self.self( 2025-08-14T21:50:50.2166625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2167019Z return func(*args, **kwargs) 2025-08-14T21:50:50.2167473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.2167922Z key_layer = self.key(current_states) 2025-08-14T21:50:50.2168056Z 2025-08-14T21:50:50.2168180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2168543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2168902Z return mod(**inputs) 2025-08-14T21:50:50.2169329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2169777Z outputs = self.bert( 2025-08-14T21:50:50.2170192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2170647Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2171092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2171561Z layer_outputs = layer_module( 2025-08-14T21:50:50.2171927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2172306Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2172747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2173185Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2173633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2174070Z self_outputs = self.self( 2025-08-14T21:50:50.2174451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2174831Z return func(*args, **kwargs) 2025-08-14T21:50:50.2175260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.2175776Z value_layer = self.value(current_states) 2025-08-14T21:50:50.2175921Z 2025-08-14T21:50:50.2176009Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2176233Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2176482Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2176855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2177192Z return mod(**inputs) 2025-08-14T21:50:50.2177589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2178003Z outputs = self.bert( 2025-08-14T21:50:50.2178410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2178833Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2179250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2179664Z layer_outputs = layer_module( 2025-08-14T21:50:50.2179999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2180354Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2180775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2181195Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2181618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.2182092Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.2182567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.2182990Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2183133Z 2025-08-14T21:50:50.2183235Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2183594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2183931Z return mod(**inputs) 2025-08-14T21:50:50.2184341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2184780Z outputs = self.bert( 2025-08-14T21:50:50.2185192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2185762Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2186172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2186595Z layer_outputs = layer_module( 2025-08-14T21:50:50.2186941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2187289Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2187714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2188147Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2188556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2188943Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2189396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2189916Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2190383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.2190808Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2190985Z 2025-08-14T21:50:50.2191089Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2191447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2191762Z return mod(**inputs) 2025-08-14T21:50:50.2192155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2192570Z outputs = self.bert( 2025-08-14T21:50:50.2192981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2193403Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2193822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2194266Z layer_outputs = layer_module( 2025-08-14T21:50:50.2194635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2194991Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2195434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2195896Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2196288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2196709Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2197185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2197697Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2198163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.2198652Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.2199061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.2199417Z return self.act(input) 2025-08-14T21:50:50.2199545Z 2025-08-14T21:50:50.2199657Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2200044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2200423Z return mod(**inputs) 2025-08-14T21:50:50.2200837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2201287Z outputs = self.bert( 2025-08-14T21:50:50.2201704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2202169Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2202608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2203064Z layer_outputs = layer_module( 2025-08-14T21:50:50.2203434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2203817Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2204280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2204771Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2205222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2205728Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2206221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2206781Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2207317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.2207778Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2207935Z 2025-08-14T21:50:50.2208082Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2208475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2208817Z return mod(**inputs) 2025-08-14T21:50:50.2209249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2209695Z outputs = self.bert( 2025-08-14T21:50:50.2210122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2210575Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2211016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2211458Z layer_outputs = layer_module( 2025-08-14T21:50:50.2211828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2212181Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2212608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2213042Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2213456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2213902Z self_outputs = self.self( 2025-08-14T21:50:50.2214293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2214692Z return func(*args, **kwargs) 2025-08-14T21:50:50.2215116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.2215577Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.2215738Z 2025-08-14T21:50:50.2215853Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2216216Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2216532Z return mod(**inputs) 2025-08-14T21:50:50.2216934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2217000Z outputs = self.bert( 2025-08-14T21:50:50.2217291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2217367Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2217651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2217732Z layer_outputs = layer_module( 2025-08-14T21:50:50.2217957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2218070Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2218379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2218463Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2218762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2218832Z self_outputs = self.self( 2025-08-14T21:50:50.2219083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2219157Z return func(*args, **kwargs) 2025-08-14T21:50:50.2219466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.2219559Z key_layer = self.key(current_states) 2025-08-14T21:50:50.2219564Z 2025-08-14T21:50:50.2219668Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2219869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2219943Z return mod(**inputs) 2025-08-14T21:50:50.2220228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2220304Z outputs = self.bert( 2025-08-14T21:50:50.2220588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2220662Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2220956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2221035Z layer_outputs = layer_module( 2025-08-14T21:50:50.2221267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2221349Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2221636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2221727Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2222010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2222080Z self_outputs = self.self( 2025-08-14T21:50:50.2222327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2222398Z return func(*args, **kwargs) 2025-08-14T21:50:50.2222693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.2222811Z value_layer = self.value(current_states) 2025-08-14T21:50:50.2222815Z 2025-08-14T21:50:50.2222896Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2222986Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2223090Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2223294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2223359Z return mod(**inputs) 2025-08-14T21:50:50.2223650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2223727Z outputs = self.bert( 2025-08-14T21:50:50.2224036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2224116Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2224427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2224549Z layer_outputs = layer_module( 2025-08-14T21:50:50.2224793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2224875Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2225187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2225280Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2225582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.2225731Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.2226042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.2226130Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2226134Z 2025-08-14T21:50:50.2226246Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2226453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2226519Z return mod(**inputs) 2025-08-14T21:50:50.2226813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2226880Z outputs = self.bert( 2025-08-14T21:50:50.2227172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2227245Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2227533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2227615Z layer_outputs = layer_module( 2025-08-14T21:50:50.2227836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2227922Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2228204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2228286Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2228552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2228629Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2228955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2229101Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2229396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.2229485Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2229489Z 2025-08-14T21:50:50.2229590Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2229792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2229867Z return mod(**inputs) 2025-08-14T21:50:50.2230155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2230229Z outputs = self.bert( 2025-08-14T21:50:50.2230517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2230592Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2230933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2231007Z layer_outputs = layer_module( 2025-08-14T21:50:50.2231229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2231317Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2231600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2231691Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2231950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2232048Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2232377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2232484Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2232783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.2232899Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.2233112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.2233192Z return self.act(input) 2025-08-14T21:50:50.2233195Z 2025-08-14T21:50:50.2233297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2233495Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2233568Z return mod(**inputs) 2025-08-14T21:50:50.2233873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2233953Z outputs = self.bert( 2025-08-14T21:50:50.2234269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2234347Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2234661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2234736Z layer_outputs = layer_module( 2025-08-14T21:50:50.2234980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2235062Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2235377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2236924Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2237198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2237275Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2237603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2237879Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2238186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.2238270Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2238274Z 2025-08-14T21:50:50.2238378Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2238593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2238721Z return mod(**inputs) 2025-08-14T21:50:50.2239072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2239145Z outputs = self.bert( 2025-08-14T21:50:50.2239459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2239548Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2239862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2239940Z layer_outputs = layer_module( 2025-08-14T21:50:50.2240186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2240311Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2240635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2240727Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2241001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2241096Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2241431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2241577Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2241892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:50.2241975Z return input_tensor + hidden_states 2025-08-14T21:50:50.2241980Z 2025-08-14T21:50:50.2242097Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2242312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2242392Z return mod(**inputs) 2025-08-14T21:50:50.2242697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2242767Z outputs = self.bert( 2025-08-14T21:50:50.2243079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2243156Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2243456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2243545Z layer_outputs = layer_module( 2025-08-14T21:50:50.2243820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2243913Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2244213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2244300Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2244612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2244689Z self_outputs = self.self( 2025-08-14T21:50:50.2244945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2245031Z return func(*args, **kwargs) 2025-08-14T21:50:50.2245332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.2245429Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.2245510Z 2025-08-14T21:50:50.2245631Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2245869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2245955Z return mod(**inputs) 2025-08-14T21:50:50.2246275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2246355Z outputs = self.bert( 2025-08-14T21:50:50.2246670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2246745Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2247042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2247154Z layer_outputs = layer_module( 2025-08-14T21:50:50.2247389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2247487Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2247799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2247898Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2248207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2248285Z self_outputs = self.self( 2025-08-14T21:50:50.2248550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2248624Z return func(*args, **kwargs) 2025-08-14T21:50:50.2248941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.2249028Z key_layer = self.key(current_states) 2025-08-14T21:50:50.2249032Z 2025-08-14T21:50:50.2249140Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2249359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2249429Z return mod(**inputs) 2025-08-14T21:50:50.2249731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2249806Z outputs = self.bert( 2025-08-14T21:50:50.2250106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2250189Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2250491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2250590Z layer_outputs = layer_module( 2025-08-14T21:50:50.2250834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2250915Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2251223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2251307Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2251607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2251688Z self_outputs = self.self( 2025-08-14T21:50:50.2251942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2252019Z return func(*args, **kwargs) 2025-08-14T21:50:50.2252330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.2252500Z value_layer = self.value(current_states) 2025-08-14T21:50:50.2252504Z 2025-08-14T21:50:50.2252601Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2252683Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2252792Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2253013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2253082Z return mod(**inputs) 2025-08-14T21:50:50.2253388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2253462Z outputs = self.bert( 2025-08-14T21:50:50.2253772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2253859Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2254165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2254241Z layer_outputs = layer_module( 2025-08-14T21:50:50.2254485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2254566Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2254876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2254964Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2255267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.2255413Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.2255718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.2255817Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2255821Z 2025-08-14T21:50:50.2255939Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2256137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2256211Z return mod(**inputs) 2025-08-14T21:50:50.2256500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2256565Z outputs = self.bert( 2025-08-14T21:50:50.2256855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2256960Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2257254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2257327Z layer_outputs = layer_module( 2025-08-14T21:50:50.2257544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2257626Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2257909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2257998Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2258258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2258333Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2258658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2258789Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2259097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.2259186Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2259190Z 2025-08-14T21:50:50.2259291Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2259493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2259558Z return mod(**inputs) 2025-08-14T21:50:50.2259845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2259920Z outputs = self.bert( 2025-08-14T21:50:50.2260230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2260315Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2260606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2260677Z layer_outputs = layer_module( 2025-08-14T21:50:50.2260906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2260982Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2261271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2261362Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2261627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2261712Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2262034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2262137Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2262433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.2262546Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.2262767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.2262838Z return self.act(input) 2025-08-14T21:50:50.2262841Z 2025-08-14T21:50:50.2262942Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2263147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2263237Z return mod(**inputs) 2025-08-14T21:50:50.2263525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2263599Z outputs = self.bert( 2025-08-14T21:50:50.2263883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2263965Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2264269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2264344Z layer_outputs = layer_module( 2025-08-14T21:50:50.2264584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2264665Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2264972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2265088Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2265380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2265464Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2265780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2265910Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2266205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.2266285Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2266291Z 2025-08-14T21:50:50.2266419Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2266627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2266699Z return mod(**inputs) 2025-08-14T21:50:50.2267010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2267080Z outputs = self.bert( 2025-08-14T21:50:50.2267398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2267476Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2267787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2267870Z layer_outputs = layer_module( 2025-08-14T21:50:50.2268106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2268190Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2268505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2268589Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2268886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2268954Z self_outputs = self.self( 2025-08-14T21:50:50.2269193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2269273Z return func(*args, **kwargs) 2025-08-14T21:50:50.2269558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.2269674Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.2269678Z 2025-08-14T21:50:50.2269780Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2269980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2270052Z return mod(**inputs) 2025-08-14T21:50:50.2270340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2270406Z outputs = self.bert( 2025-08-14T21:50:50.2270698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2270772Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2271060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2271133Z layer_outputs = layer_module( 2025-08-14T21:50:50.2271353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2271463Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2271781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2271872Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2272156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2272233Z self_outputs = self.self( 2025-08-14T21:50:50.2272489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2272575Z return func(*args, **kwargs) 2025-08-14T21:50:50.2272902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.2272998Z key_layer = self.key(current_states) 2025-08-14T21:50:50.2273003Z 2025-08-14T21:50:50.2273111Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2273330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2273398Z return mod(**inputs) 2025-08-14T21:50:50.2273703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2273783Z outputs = self.bert( 2025-08-14T21:50:50.2274091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2274175Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2274478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2274555Z layer_outputs = layer_module( 2025-08-14T21:50:50.2274797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2274878Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2275179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2275269Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2275552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2275631Z self_outputs = self.self( 2025-08-14T21:50:50.2275893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2275968Z return func(*args, **kwargs) 2025-08-14T21:50:50.2276302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.2276386Z value_layer = self.value(current_states) 2025-08-14T21:50:50.2276391Z 2025-08-14T21:50:50.2276485Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2276571Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2276679Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2276893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2276961Z return mod(**inputs) 2025-08-14T21:50:50.2277264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2277341Z outputs = self.bert( 2025-08-14T21:50:50.2277654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2277744Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2278078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2278172Z layer_outputs = layer_module( 2025-08-14T21:50:50.2278418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2278496Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2278811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2278903Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2279275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.2279439Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.2279755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.2279844Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2279847Z 2025-08-14T21:50:50.2279965Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2280175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2280252Z return mod(**inputs) 2025-08-14T21:50:50.2280554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2280621Z outputs = self.bert( 2025-08-14T21:50:50.2280932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2281007Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2281317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2281402Z layer_outputs = layer_module( 2025-08-14T21:50:50.2281633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2281721Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2282020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2282106Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2282386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2282467Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2282807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2282938Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2283241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.2283335Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2283339Z 2025-08-14T21:50:50.2283449Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2283666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2283739Z return mod(**inputs) 2025-08-14T21:50:50.2284044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2284124Z outputs = self.bert( 2025-08-14T21:50:50.2284436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2284518Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2284877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2284953Z layer_outputs = layer_module( 2025-08-14T21:50:50.2285191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2285272Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2285646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2285745Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2286025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2286164Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2286536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2286655Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2286975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.2287109Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.2287332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.2287416Z return self.act(input) 2025-08-14T21:50:50.2287420Z 2025-08-14T21:50:50.2287529Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2287750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2287825Z return mod(**inputs) 2025-08-14T21:50:50.2288130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2288214Z outputs = self.bert( 2025-08-14T21:50:50.2288512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2288600Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2288914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2288991Z layer_outputs = layer_module( 2025-08-14T21:50:50.2289234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2289318Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2289625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2289753Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2290027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2290114Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2290446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2290585Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2290894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.2290974Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2290978Z 2025-08-14T21:50:50.2291087Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2291290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2291382Z return mod(**inputs) 2025-08-14T21:50:50.2291719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2291790Z outputs = self.bert( 2025-08-14T21:50:50.2292097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2292179Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2292465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2292546Z layer_outputs = layer_module( 2025-08-14T21:50:50.2292778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2292881Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2293198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2293281Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2293543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2293618Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2293930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2294065Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2294345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:50.2294426Z return input_tensor + hidden_states 2025-08-14T21:50:50.2294437Z 2025-08-14T21:50:50.2294537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2294739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2294814Z return mod(**inputs) 2025-08-14T21:50:50.2295096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2295162Z outputs = self.bert( 2025-08-14T21:50:50.2295450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2295522Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2295814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2295885Z layer_outputs = layer_module( 2025-08-14T21:50:50.2296130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2296220Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2296503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2296586Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2296875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2296945Z self_outputs = self.self( 2025-08-14T21:50:50.2297192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2297265Z return func(*args, **kwargs) 2025-08-14T21:50:50.2297550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.2297640Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.2297673Z 2025-08-14T21:50:50.2297796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2298004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2298068Z return mod(**inputs) 2025-08-14T21:50:50.2298356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2298429Z outputs = self.bert( 2025-08-14T21:50:50.2298720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2298797Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2299111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2299185Z layer_outputs = layer_module( 2025-08-14T21:50:50.2299417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2299493Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2299777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2299865Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2300155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2300234Z self_outputs = self.self( 2025-08-14T21:50:50.2300476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2300547Z return func(*args, **kwargs) 2025-08-14T21:50:50.2300845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.2300924Z key_layer = self.key(current_states) 2025-08-14T21:50:50.2300930Z 2025-08-14T21:50:50.2301031Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2301232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2301294Z return mod(**inputs) 2025-08-14T21:50:50.2301595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2301660Z outputs = self.bert( 2025-08-14T21:50:50.2301949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2302032Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2302342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2302423Z layer_outputs = layer_module( 2025-08-14T21:50:50.2302642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2302718Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2303014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2303097Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2303391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2303473Z self_outputs = self.self( 2025-08-14T21:50:50.2303723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2303814Z return func(*args, **kwargs) 2025-08-14T21:50:50.2304137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.2304242Z value_layer = self.value(current_states) 2025-08-14T21:50:50.2304246Z 2025-08-14T21:50:50.2304339Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2304427Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2304536Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2304753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2304824Z return mod(**inputs) 2025-08-14T21:50:50.2305145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2305214Z outputs = self.bert( 2025-08-14T21:50:50.2305538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2305629Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2305933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2306014Z layer_outputs = layer_module( 2025-08-14T21:50:50.2306237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2306313Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2306608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2306689Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2306976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.2307113Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.2307405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.2307496Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2307499Z 2025-08-14T21:50:50.2307599Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2307789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2307870Z return mod(**inputs) 2025-08-14T21:50:50.2308155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2308230Z outputs = self.bert( 2025-08-14T21:50:50.2308518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2308612Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2308904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2308977Z layer_outputs = layer_module( 2025-08-14T21:50:50.2309194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2309277Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2309561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2309653Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2309910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2309986Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2310315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2310464Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2310755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.2310837Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2310840Z 2025-08-14T21:50:50.2310941Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2311146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2311212Z return mod(**inputs) 2025-08-14T21:50:50.2311497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2311569Z outputs = self.bert( 2025-08-14T21:50:50.2311872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2311955Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2312242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2312311Z layer_outputs = layer_module( 2025-08-14T21:50:50.2312537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2312615Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2312907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2312990Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2313251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2313339Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2313657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2313765Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2314047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.2314157Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.2314372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.2314445Z return self.act(input) 2025-08-14T21:50:50.2314449Z 2025-08-14T21:50:50.2314557Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2314775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2314874Z return mod(**inputs) 2025-08-14T21:50:50.2315206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2315274Z outputs = self.bert( 2025-08-14T21:50:50.2315574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2315669Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2315951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2316022Z layer_outputs = layer_module( 2025-08-14T21:50:50.2316250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2316332Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2316637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2316774Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2317048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2317133Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2317467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2317612Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2317914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.2318000Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2318030Z 2025-08-14T21:50:50.2318146Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2318353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2318431Z return mod(**inputs) 2025-08-14T21:50:50.2318733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2318802Z outputs = self.bert( 2025-08-14T21:50:50.2319106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2319181Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2319479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2319562Z layer_outputs = layer_module( 2025-08-14T21:50:50.2319797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2319887Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2320185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2320271Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2320577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2320650Z self_outputs = self.self( 2025-08-14T21:50:50.2320910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2320986Z return func(*args, **kwargs) 2025-08-14T21:50:50.2321283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.2321417Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.2321420Z 2025-08-14T21:50:50.2321528Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2321737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2321817Z return mod(**inputs) 2025-08-14T21:50:50.2322122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2322200Z outputs = self.bert( 2025-08-14T21:50:50.2322499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2322575Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2322882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2322959Z layer_outputs = layer_module( 2025-08-14T21:50:50.2323197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2323325Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2323624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2323717Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2324014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2324086Z self_outputs = self.self( 2025-08-14T21:50:50.2324344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2324417Z return func(*args, **kwargs) 2025-08-14T21:50:50.2324743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.2324830Z key_layer = self.key(current_states) 2025-08-14T21:50:50.2324834Z 2025-08-14T21:50:50.2324944Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2325164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2325233Z return mod(**inputs) 2025-08-14T21:50:50.2325626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2325710Z outputs = self.bert( 2025-08-14T21:50:50.2326031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2326115Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2326423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2326502Z layer_outputs = layer_module( 2025-08-14T21:50:50.2326743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2326824Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2327128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2327213Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2327511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2327592Z self_outputs = self.self( 2025-08-14T21:50:50.2327845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2327920Z return func(*args, **kwargs) 2025-08-14T21:50:50.2328252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.2328339Z value_layer = self.value(current_states) 2025-08-14T21:50:50.2328342Z 2025-08-14T21:50:50.2328435Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2328521Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2328629Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2328846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2328916Z return mod(**inputs) 2025-08-14T21:50:50.2329244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2329314Z outputs = self.bert( 2025-08-14T21:50:50.2329626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2329715Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2330058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2330135Z layer_outputs = layer_module( 2025-08-14T21:50:50.2330375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2330456Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2330764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2330845Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2331152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.2331313Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.2331617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.2331709Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2331713Z 2025-08-14T21:50:50.2331819Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2332024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2332098Z return mod(**inputs) 2025-08-14T21:50:50.2332395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2332463Z outputs = self.bert( 2025-08-14T21:50:50.2332777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2332857Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2333161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2333238Z layer_outputs = layer_module( 2025-08-14T21:50:50.2333468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2333555Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2333852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2333947Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2334218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2334299Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2334638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2334772Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2335079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.2335172Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2335176Z 2025-08-14T21:50:50.2335281Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2335496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2335564Z return mod(**inputs) 2025-08-14T21:50:50.2335882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2335959Z outputs = self.bert( 2025-08-14T21:50:50.2336262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2336372Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2336699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2336776Z layer_outputs = layer_module( 2025-08-14T21:50:50.2337017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2337096Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2337402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2337503Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2337977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2338078Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2338427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2338538Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2338855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.2338976Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.2339233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.2339313Z return self.act(input) 2025-08-14T21:50:50.2339317Z 2025-08-14T21:50:50.2339431Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2340886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2341076Z return mod(**inputs) 2025-08-14T21:50:50.2341467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2341561Z outputs = self.bert( 2025-08-14T21:50:50.2341902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2341995Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2342328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2342408Z layer_outputs = layer_module( 2025-08-14T21:50:50.2342668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2342760Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2343290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2343388Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2343681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2343778Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2344124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2344314Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2344751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.2344849Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2344856Z 2025-08-14T21:50:50.2344989Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2345209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2345342Z return mod(**inputs) 2025-08-14T21:50:50.2345718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2345792Z outputs = self.bert( 2025-08-14T21:50:50.2346103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2346185Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2346547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2346635Z layer_outputs = layer_module( 2025-08-14T21:50:50.2346922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2347014Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2347338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2347438Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2347721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2347807Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2348161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2348309Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2348612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:50.2348709Z return input_tensor + hidden_states 2025-08-14T21:50:50.2348714Z 2025-08-14T21:50:50.2348834Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2349061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2349145Z return mod(**inputs) 2025-08-14T21:50:50.2349460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2349542Z outputs = self.bert( 2025-08-14T21:50:50.2349875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2349961Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2350287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2350445Z layer_outputs = layer_module( 2025-08-14T21:50:50.2350699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2350794Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2351110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2351206Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2351518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2351592Z self_outputs = self.self( 2025-08-14T21:50:50.2351865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2351941Z return func(*args, **kwargs) 2025-08-14T21:50:50.2352243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.2352363Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.2352367Z 2025-08-14T21:50:50.2352497Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2352720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2352789Z return mod(**inputs) 2025-08-14T21:50:50.2353095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2353171Z outputs = self.bert( 2025-08-14T21:50:50.2353483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2353571Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2353898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2353973Z layer_outputs = layer_module( 2025-08-14T21:50:50.2354205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2354283Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2354588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2354682Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2354980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2355062Z self_outputs = self.self( 2025-08-14T21:50:50.2355325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2355404Z return func(*args, **kwargs) 2025-08-14T21:50:50.2355721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.2355807Z key_layer = self.key(current_states) 2025-08-14T21:50:50.2355811Z 2025-08-14T21:50:50.2355927Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2356137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2356205Z return mod(**inputs) 2025-08-14T21:50:50.2356521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2356585Z outputs = self.bert( 2025-08-14T21:50:50.2356870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2356949Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2357253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2357333Z layer_outputs = layer_module( 2025-08-14T21:50:50.2357554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2357631Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2357923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2358002Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2358293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2358361Z self_outputs = self.self( 2025-08-14T21:50:50.2358599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2358679Z return func(*args, **kwargs) 2025-08-14T21:50:50.2359002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.2359082Z value_layer = self.value(current_states) 2025-08-14T21:50:50.2359093Z 2025-08-14T21:50:50.2359175Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2359253Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2359363Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2359558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2359623Z return mod(**inputs) 2025-08-14T21:50:50.2359914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2359979Z outputs = self.bert( 2025-08-14T21:50:50.2360282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2360365Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2360652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2360730Z layer_outputs = layer_module( 2025-08-14T21:50:50.2360946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2361020Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2361312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2361394Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2361682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.2361812Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.2362100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.2362188Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2362192Z 2025-08-14T21:50:50.2362292Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2362496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2362562Z return mod(**inputs) 2025-08-14T21:50:50.2362848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2362922Z outputs = self.bert( 2025-08-14T21:50:50.2363203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2363296Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2363597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2363668Z layer_outputs = layer_module( 2025-08-14T21:50:50.2363893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2363969Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2364274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2364369Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2364645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2364736Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2365074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2365223Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2365817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.2365912Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2365916Z 2025-08-14T21:50:50.2366022Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2366261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2366333Z return mod(**inputs) 2025-08-14T21:50:50.2366663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2366763Z outputs = self.bert( 2025-08-14T21:50:50.2367071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2367156Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2367443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2367522Z layer_outputs = layer_module( 2025-08-14T21:50:50.2367741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2367818Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2368109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2368190Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2368448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2368533Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2368847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2368957Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2369243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.2369355Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.2369574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.2369646Z return self.act(input) 2025-08-14T21:50:50.2369650Z 2025-08-14T21:50:50.2369756Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2369975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2370045Z return mod(**inputs) 2025-08-14T21:50:50.2370354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2370420Z outputs = self.bert( 2025-08-14T21:50:50.2370692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2370773Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2371049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2371125Z layer_outputs = layer_module( 2025-08-14T21:50:50.2371335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2371413Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2371697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2371825Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2372081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2372156Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2372469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2372607Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2372894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.2372993Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2373006Z 2025-08-14T21:50:50.2373107Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2373311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2373385Z return mod(**inputs) 2025-08-14T21:50:50.2373689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2373759Z outputs = self.bert( 2025-08-14T21:50:50.2374078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2374157Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2374472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2374546Z layer_outputs = layer_module( 2025-08-14T21:50:50.2374777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2374870Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2375174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2375268Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2375553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2375621Z self_outputs = self.self( 2025-08-14T21:50:50.2378324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2378403Z return func(*args, **kwargs) 2025-08-14T21:50:50.2378686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.2379853Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.2379858Z 2025-08-14T21:50:50.2379973Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2380174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2380240Z return mod(**inputs) 2025-08-14T21:50:50.2380540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2380605Z outputs = self.bert( 2025-08-14T21:50:50.2380895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2381001Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2381288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2381369Z layer_outputs = layer_module( 2025-08-14T21:50:50.2381604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2381706Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2382000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2382081Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2382365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2382449Z self_outputs = self.self( 2025-08-14T21:50:50.2382691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2382771Z return func(*args, **kwargs) 2025-08-14T21:50:50.2383077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.2383171Z key_layer = self.key(current_states) 2025-08-14T21:50:50.2383176Z 2025-08-14T21:50:50.2383284Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2383476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2383547Z return mod(**inputs) 2025-08-14T21:50:50.2383828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2383893Z outputs = self.bert( 2025-08-14T21:50:50.2384186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2384258Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2384541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2384619Z layer_outputs = layer_module( 2025-08-14T21:50:50.2384839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2384921Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2385204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2385282Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2385574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2385710Z self_outputs = self.self( 2025-08-14T21:50:50.2385954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2386031Z return func(*args, **kwargs) 2025-08-14T21:50:50.2386341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.2386426Z value_layer = self.value(current_states) 2025-08-14T21:50:50.2386430Z 2025-08-14T21:50:50.2386509Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2386589Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2386698Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2386892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2386965Z return mod(**inputs) 2025-08-14T21:50:50.2387248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2387314Z outputs = self.bert( 2025-08-14T21:50:50.2387606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2387680Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2387980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2388062Z layer_outputs = layer_module( 2025-08-14T21:50:50.2388283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2388371Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2388661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2388746Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2389041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.2389193Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.2389489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.2389571Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2389575Z 2025-08-14T21:50:50.2389674Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2389876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2389941Z return mod(**inputs) 2025-08-14T21:50:50.2390227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2390303Z outputs = self.bert( 2025-08-14T21:50:50.2390586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2390668Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2390956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2391027Z layer_outputs = layer_module( 2025-08-14T21:50:50.2391253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2391330Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2391623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2391746Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2392007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2392092Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2392411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2392548Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2392840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.2392922Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2392926Z 2025-08-14T21:50:50.2393035Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2393230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2393297Z return mod(**inputs) 2025-08-14T21:50:50.2393590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2393654Z outputs = self.bert( 2025-08-14T21:50:50.2393948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2394021Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2394322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2394403Z layer_outputs = layer_module( 2025-08-14T21:50:50.2394737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2394827Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2395121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2395207Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2395495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2395576Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2395892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2396005Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2396290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.2396413Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.2396626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.2396700Z return self.act(input) 2025-08-14T21:50:50.2396704Z 2025-08-14T21:50:50.2396814Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2397017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2397085Z return mod(**inputs) 2025-08-14T21:50:50.2397392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2397460Z outputs = self.bert( 2025-08-14T21:50:50.2397762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2397836Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2398128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2398291Z layer_outputs = layer_module( 2025-08-14T21:50:50.2398518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2398601Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2398891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2398997Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2399262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2399339Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2399652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2399792Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2400076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.2400161Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2400165Z 2025-08-14T21:50:50.2400268Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2400463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2400556Z return mod(**inputs) 2025-08-14T21:50:50.2400843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2400915Z outputs = self.bert( 2025-08-14T21:50:50.2401192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2401266Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2401554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2401625Z layer_outputs = layer_module( 2025-08-14T21:50:50.2401870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2401952Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2402234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2402319Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2402574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2402648Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2402966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2403096Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2403384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:50.2403463Z return input_tensor + hidden_states 2025-08-14T21:50:50.2403466Z 2025-08-14T21:50:50.2403570Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2403777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2403841Z return mod(**inputs) 2025-08-14T21:50:50.2404148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2404219Z outputs = self.bert( 2025-08-14T21:50:50.2404524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2404634Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2404936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2405011Z layer_outputs = layer_module( 2025-08-14T21:50:50.2405273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2405357Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2405779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2405878Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2406180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2406269Z self_outputs = self.self( 2025-08-14T21:50:50.2406525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2406611Z return func(*args, **kwargs) 2025-08-14T21:50:50.2406916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.2407005Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.2407038Z 2025-08-14T21:50:50.2407156Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2407366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2407435Z return mod(**inputs) 2025-08-14T21:50:50.2407751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2407820Z outputs = self.bert( 2025-08-14T21:50:50.2408131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2408207Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2408528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2408615Z layer_outputs = layer_module( 2025-08-14T21:50:50.2408852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2408940Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2409241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2409326Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2409636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2409712Z self_outputs = self.self( 2025-08-14T21:50:50.2409967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2410052Z return func(*args, **kwargs) 2025-08-14T21:50:50.2410389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.2410481Z key_layer = self.key(current_states) 2025-08-14T21:50:50.2410485Z 2025-08-14T21:50:50.2410592Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2410801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2410879Z return mod(**inputs) 2025-08-14T21:50:50.2411189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2411289Z outputs = self.bert( 2025-08-14T21:50:50.2411595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2411673Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2411984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2412082Z layer_outputs = layer_module( 2025-08-14T21:50:50.2412314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2412403Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2412704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2412794Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2413096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2413168Z self_outputs = self.self( 2025-08-14T21:50:50.2413428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2413503Z return func(*args, **kwargs) 2025-08-14T21:50:50.2413823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.2413914Z value_layer = self.value(current_states) 2025-08-14T21:50:50.2413918Z 2025-08-14T21:50:50.2414004Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2414097Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2414204Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2414411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2414490Z return mod(**inputs) 2025-08-14T21:50:50.2414796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2414889Z outputs = self.bert( 2025-08-14T21:50:50.2415194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2415274Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2415589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2415667Z layer_outputs = layer_module( 2025-08-14T21:50:50.2415903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2415995Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2416302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2416396Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2416703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.2416845Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.2417159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.2417250Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2417254Z 2025-08-14T21:50:50.2417370Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2417582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2417655Z return mod(**inputs) 2025-08-14T21:50:50.2417990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2418054Z outputs = self.bert( 2025-08-14T21:50:50.2418338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2418443Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2418728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2418804Z layer_outputs = layer_module( 2025-08-14T21:50:50.2419023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2419098Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2419386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2419470Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2419734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2419812Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2420142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2420258Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2420541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.2420622Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2420633Z 2025-08-14T21:50:50.2420734Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2420933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2421005Z return mod(**inputs) 2025-08-14T21:50:50.2421320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2421387Z outputs = self.bert( 2025-08-14T21:50:50.2421682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2421754Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2422049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2422124Z layer_outputs = layer_module( 2025-08-14T21:50:50.2422355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2422446Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2422749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2422836Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2423116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2423207Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2423533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2423635Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2423925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.2424053Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.2424298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.2424380Z return self.act(input) 2025-08-14T21:50:50.2424384Z 2025-08-14T21:50:50.2424494Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2424699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2424800Z return mod(**inputs) 2025-08-14T21:50:50.2425109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2425180Z outputs = self.bert( 2025-08-14T21:50:50.2425486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2425564Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2425873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2425944Z layer_outputs = layer_module( 2025-08-14T21:50:50.2426165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2426250Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2426555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2426647Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2426906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2426980Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2427302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2427433Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2427717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.2427825Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2427831Z 2025-08-14T21:50:50.2427932Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2428137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2428202Z return mod(**inputs) 2025-08-14T21:50:50.2428487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2428560Z outputs = self.bert( 2025-08-14T21:50:50.2428843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2428922Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2429203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2429274Z layer_outputs = layer_module( 2025-08-14T21:50:50.2429502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2429581Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2429867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2429957Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2430253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2430333Z self_outputs = self.self( 2025-08-14T21:50:50.2430609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2430682Z return func(*args, **kwargs) 2025-08-14T21:50:50.2430996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.2431104Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.2431108Z 2025-08-14T21:50:50.2431226Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2431432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2431500Z return mod(**inputs) 2025-08-14T21:50:50.2431811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2431879Z outputs = self.bert( 2025-08-14T21:50:50.2432188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2432274Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2432576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2432659Z layer_outputs = layer_module( 2025-08-14T21:50:50.2432909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2432991Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2433297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2433381Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2433695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2433769Z self_outputs = self.self( 2025-08-14T21:50:50.2434019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2434100Z return func(*args, **kwargs) 2025-08-14T21:50:50.2434430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.2434516Z key_layer = self.key(current_states) 2025-08-14T21:50:50.2434527Z 2025-08-14T21:50:50.2434637Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2434853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2434930Z return mod(**inputs) 2025-08-14T21:50:50.2435250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2435326Z outputs = self.bert( 2025-08-14T21:50:50.2435649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2435726Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2436056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2436130Z layer_outputs = layer_module( 2025-08-14T21:50:50.2436364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2436451Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2436749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2436831Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2437138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2437235Z self_outputs = self.self( 2025-08-14T21:50:50.2437502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2437577Z return func(*args, **kwargs) 2025-08-14T21:50:50.2438309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.2438410Z value_layer = self.value(current_states) 2025-08-14T21:50:50.2438414Z 2025-08-14T21:50:50.2438501Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2438594Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2438701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2438912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2438995Z return mod(**inputs) 2025-08-14T21:50:50.2439306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2439378Z outputs = self.bert( 2025-08-14T21:50:50.2439707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2439790Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2440180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2440260Z layer_outputs = layer_module( 2025-08-14T21:50:50.2440501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2440593Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2440922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2441009Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2441352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.2441490Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.2441804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.2441895Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2441899Z 2025-08-14T21:50:50.2442012Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2442240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2442316Z return mod(**inputs) 2025-08-14T21:50:50.2442644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2442721Z outputs = self.bert( 2025-08-14T21:50:50.2443037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2443131Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2443447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2443536Z layer_outputs = layer_module( 2025-08-14T21:50:50.2443781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2443865Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2444181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2444311Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2444595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2444685Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2445030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2445183Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2445568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.2445668Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2445673Z 2025-08-14T21:50:50.2445791Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2446005Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2446087Z return mod(**inputs) 2025-08-14T21:50:50.2446405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2446488Z outputs = self.bert( 2025-08-14T21:50:50.2446800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2446917Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2447218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2447304Z layer_outputs = layer_module( 2025-08-14T21:50:50.2447540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2447632Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2447937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2448026Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2448336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2448421Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2448764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2448872Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2449177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.2449304Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.2449527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.2449603Z return self.act(input) 2025-08-14T21:50:50.2449615Z 2025-08-14T21:50:50.2449723Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2449933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2450010Z return mod(**inputs) 2025-08-14T21:50:50.2450315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2450386Z outputs = self.bert( 2025-08-14T21:50:50.2450696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2450773Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2451080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2451178Z layer_outputs = layer_module( 2025-08-14T21:50:50.2451412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2451503Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2451813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2451922Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2452205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2452283Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2452615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2452756Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2453040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.2453128Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2453133Z 2025-08-14T21:50:50.2453234Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2453461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2453528Z return mod(**inputs) 2025-08-14T21:50:50.2453815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2453887Z outputs = self.bert( 2025-08-14T21:50:50.2454170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2454242Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2454534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2454603Z layer_outputs = layer_module( 2025-08-14T21:50:50.2454851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2454930Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2455212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2455303Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2455557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2455637Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2455948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2456080Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2456372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:50.2456449Z return input_tensor + hidden_states 2025-08-14T21:50:50.2456453Z 2025-08-14T21:50:50.2456562Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2456762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2456827Z return mod(**inputs) 2025-08-14T21:50:50.2457126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2457191Z outputs = self.bert( 2025-08-14T21:50:50.2457478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2457581Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2457866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2457967Z layer_outputs = layer_module( 2025-08-14T21:50:50.2458195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2458275Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2458571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2458651Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2458941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2459020Z self_outputs = self.self( 2025-08-14T21:50:50.2459262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2459339Z return func(*args, **kwargs) 2025-08-14T21:50:50.2459628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.2459728Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.2459733Z 2025-08-14T21:50:50.2459841Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2460039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2460112Z return mod(**inputs) 2025-08-14T21:50:50.2460402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2460468Z outputs = self.bert( 2025-08-14T21:50:50.2460756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2460828Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2461132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2461214Z layer_outputs = layer_module( 2025-08-14T21:50:50.2461435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2461520Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2461806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2461885Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2462174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2462244Z self_outputs = self.self( 2025-08-14T21:50:50.2462491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2462562Z return func(*args, **kwargs) 2025-08-14T21:50:50.2462850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.2462933Z key_layer = self.key(current_states) 2025-08-14T21:50:50.2462936Z 2025-08-14T21:50:50.2463036Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2463232Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2463305Z return mod(**inputs) 2025-08-14T21:50:50.2463590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2463685Z outputs = self.bert( 2025-08-14T21:50:50.2463995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2464074Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2464414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2464488Z layer_outputs = layer_module( 2025-08-14T21:50:50.2464727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2464807Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2465116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2465207Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2465519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2465591Z self_outputs = self.self( 2025-08-14T21:50:50.2465852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2465924Z return func(*args, **kwargs) 2025-08-14T21:50:50.2466237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.2466317Z value_layer = self.value(current_states) 2025-08-14T21:50:50.2466321Z 2025-08-14T21:50:50.2466401Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2466489Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2466589Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2466792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2466859Z return mod(**inputs) 2025-08-14T21:50:50.2467146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2467236Z outputs = self.bert( 2025-08-14T21:50:50.2467523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2467595Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2467887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2467955Z layer_outputs = layer_module( 2025-08-14T21:50:50.2468180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2468256Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2468540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2468625Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2468910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.2469039Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.2469330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.2469412Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2469416Z 2025-08-14T21:50:50.2469522Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2469719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2469805Z return mod(**inputs) 2025-08-14T21:50:50.2470101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2470165Z outputs = self.bert( 2025-08-14T21:50:50.2470459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2470562Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2470857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2470934Z layer_outputs = layer_module( 2025-08-14T21:50:50.2471158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2471235Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2471530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2471612Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2471886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2471961Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2472303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2472414Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2472695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.2472783Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2472786Z 2025-08-14T21:50:50.2472887Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2473086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2473160Z return mod(**inputs) 2025-08-14T21:50:50.2473466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2473546Z outputs = self.bert( 2025-08-14T21:50:50.2473836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2473910Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2474203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2474277Z layer_outputs = layer_module( 2025-08-14T21:50:50.2474510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2474602Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2474901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2474997Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2475270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2475352Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2475695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2475802Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2476169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.2476281Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.2476512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.2476615Z return self.act(input) 2025-08-14T21:50:50.2476619Z 2025-08-14T21:50:50.2476720Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2476936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2477010Z return mod(**inputs) 2025-08-14T21:50:50.2477297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2477369Z outputs = self.bert( 2025-08-14T21:50:50.2477655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2477726Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2478020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2478094Z layer_outputs = layer_module( 2025-08-14T21:50:50.2478333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2478416Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2478744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2478840Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2479109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2479187Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2479526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2479664Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2480009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.2480096Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2480102Z 2025-08-14T21:50:50.2480209Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2480427Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2480496Z return mod(**inputs) 2025-08-14T21:50:50.2480805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2480874Z outputs = self.bert( 2025-08-14T21:50:50.2481185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2481270Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2481569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2481645Z layer_outputs = layer_module( 2025-08-14T21:50:50.2481886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2481971Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2482276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2482360Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2482669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2482772Z self_outputs = self.self( 2025-08-14T21:50:50.2483030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2483111Z return func(*args, **kwargs) 2025-08-14T21:50:50.2483413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.2483522Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.2483526Z 2025-08-14T21:50:50.2483642Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2483850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2483918Z return mod(**inputs) 2025-08-14T21:50:50.2484228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2484296Z outputs = self.bert( 2025-08-14T21:50:50.2484602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2484677Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2484980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2485063Z layer_outputs = layer_module( 2025-08-14T21:50:50.2485316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2485406Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2485865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2485961Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2486285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2486364Z self_outputs = self.self( 2025-08-14T21:50:50.2486635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2486747Z return func(*args, **kwargs) 2025-08-14T21:50:50.2487032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.2487125Z key_layer = self.key(current_states) 2025-08-14T21:50:50.2487129Z 2025-08-14T21:50:50.2487236Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2487446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2487524Z return mod(**inputs) 2025-08-14T21:50:50.2487838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2487914Z outputs = self.bert( 2025-08-14T21:50:50.2488197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2488270Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2488562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2488634Z layer_outputs = layer_module( 2025-08-14T21:50:50.2488853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2488939Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2489219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2489305Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2489644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2489715Z self_outputs = self.self( 2025-08-14T21:50:50.2489964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2490076Z return func(*args, **kwargs) 2025-08-14T21:50:50.2490377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.2490456Z value_layer = self.value(current_states) 2025-08-14T21:50:50.2490460Z 2025-08-14T21:50:50.2490541Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2490629Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2490730Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2490930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2491005Z return mod(**inputs) 2025-08-14T21:50:50.2491301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2491374Z outputs = self.bert( 2025-08-14T21:50:50.2491663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2491736Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2493039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2493116Z layer_outputs = layer_module( 2025-08-14T21:50:50.2493340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2493424Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2493708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2493798Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2494103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.2494234Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.2494529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.2494616Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2494619Z 2025-08-14T21:50:50.2494733Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2494931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2494999Z return mod(**inputs) 2025-08-14T21:50:50.2495296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2495363Z outputs = self.bert( 2025-08-14T21:50:50.2495659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2495744Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2496031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2496112Z layer_outputs = layer_module( 2025-08-14T21:50:50.2496334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2496413Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2496706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2496829Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2497094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2497170Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2497502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2497615Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2497904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.2497984Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2497996Z 2025-08-14T21:50:50.2498098Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2498292Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2498368Z return mod(**inputs) 2025-08-14T21:50:50.2498652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2498721Z outputs = self.bert( 2025-08-14T21:50:50.2499012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2499105Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2499399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2499491Z layer_outputs = layer_module( 2025-08-14T21:50:50.2499759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2499845Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2500131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2500218Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2500517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2500599Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2500947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2501056Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2501359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.2501485Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.2501709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.2501788Z return self.act(input) 2025-08-14T21:50:50.2501792Z 2025-08-14T21:50:50.2501909Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2502109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2502180Z return mod(**inputs) 2025-08-14T21:50:50.2502469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2502534Z outputs = self.bert( 2025-08-14T21:50:50.2502825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2502898Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2503190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2503285Z layer_outputs = layer_module( 2025-08-14T21:50:50.2503504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2503590Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2503893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2503981Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2504235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2504309Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2504626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2504756Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2505043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.2505125Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2505129Z 2025-08-14T21:50:50.2505232Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2505461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2505528Z return mod(**inputs) 2025-08-14T21:50:50.2505819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2505891Z outputs = self.bert( 2025-08-14T21:50:50.2506173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2506254Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2506539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2506609Z layer_outputs = layer_module( 2025-08-14T21:50:50.2506854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2506935Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2507225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2507308Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2507563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2507645Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2507964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2508094Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2508387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:50.2508467Z return input_tensor + hidden_states 2025-08-14T21:50:50.2508471Z 2025-08-14T21:50:50.2508579Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2508774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2508839Z return mod(**inputs) 2025-08-14T21:50:50.2509176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2509240Z outputs = self.bert( 2025-08-14T21:50:50.2509555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2509626Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2509939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2510037Z layer_outputs = layer_module( 2025-08-14T21:50:50.2510254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2510330Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2510620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2510697Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2511036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2511107Z self_outputs = self.self( 2025-08-14T21:50:50.2511349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2511429Z return func(*args, **kwargs) 2025-08-14T21:50:50.2511719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.2511828Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.2511832Z 2025-08-14T21:50:50.2511935Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2512135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2512206Z return mod(**inputs) 2025-08-14T21:50:50.2512495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2512563Z outputs = self.bert( 2025-08-14T21:50:50.2512857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2512928Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2513244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2513318Z layer_outputs = layer_module( 2025-08-14T21:50:50.2513548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2513631Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2513910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2513996Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2514274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2514342Z self_outputs = self.self( 2025-08-14T21:50:50.2514585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2514654Z return func(*args, **kwargs) 2025-08-14T21:50:50.2514934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.2515017Z key_layer = self.key(current_states) 2025-08-14T21:50:50.2515021Z 2025-08-14T21:50:50.2515119Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2515321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2515383Z return mod(**inputs) 2025-08-14T21:50:50.2515663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2515756Z outputs = self.bert( 2025-08-14T21:50:50.2516039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2516117Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2516419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2516488Z layer_outputs = layer_module( 2025-08-14T21:50:50.2516709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2516781Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2517057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2517143Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2517414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2517487Z self_outputs = self.self( 2025-08-14T21:50:50.2517718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2517787Z return func(*args, **kwargs) 2025-08-14T21:50:50.2518087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.2518163Z value_layer = self.value(current_states) 2025-08-14T21:50:50.2518167Z 2025-08-14T21:50:50.2518250Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2518327Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2518424Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2518622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2518684Z return mod(**inputs) 2025-08-14T21:50:50.2518990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2519060Z outputs = self.bert( 2025-08-14T21:50:50.2519343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2519421Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2519697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2519765Z layer_outputs = layer_module( 2025-08-14T21:50:50.2519988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2520065Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2520338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2520423Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2520705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.2520838Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.2521119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.2521197Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2521200Z 2025-08-14T21:50:50.2521308Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2521498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2521590Z return mod(**inputs) 2025-08-14T21:50:50.2521869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2521933Z outputs = self.bert( 2025-08-14T21:50:50.2522222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2522314Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2522591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2522668Z layer_outputs = layer_module( 2025-08-14T21:50:50.2522880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2522964Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2523242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2523322Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2523589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2523667Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2524004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2524109Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2524397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.2524487Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2524491Z 2025-08-14T21:50:50.2524595Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2524805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2524873Z return mod(**inputs) 2025-08-14T21:50:50.2525184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2525261Z outputs = self.bert( 2025-08-14T21:50:50.2525641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2525722Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2526028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2526105Z layer_outputs = layer_module( 2025-08-14T21:50:50.2526344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2526427Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2526726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2526825Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2527101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2527183Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2527525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2527627Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2527919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.2528058Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.2528270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.2528350Z return self.act(input) 2025-08-14T21:50:50.2528354Z 2025-08-14T21:50:50.2528456Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2528684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2528751Z return mod(**inputs) 2025-08-14T21:50:50.2529042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2529117Z outputs = self.bert( 2025-08-14T21:50:50.2529402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2529474Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2529770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2529843Z layer_outputs = layer_module( 2025-08-14T21:50:50.2530078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2530161Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2530468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2530559Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2530814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2530896Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2531206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2531334Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2531644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.2531727Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2531731Z 2025-08-14T21:50:50.2531841Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2532041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2532107Z return mod(**inputs) 2025-08-14T21:50:50.2532404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2532470Z outputs = self.bert( 2025-08-14T21:50:50.2532752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2532835Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2533117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2533195Z layer_outputs = layer_module( 2025-08-14T21:50:50.2533415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2533490Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2533784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2533864Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2534153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2534245Z self_outputs = self.self( 2025-08-14T21:50:50.2534487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2534566Z return func(*args, **kwargs) 2025-08-14T21:50:50.2534852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.2534954Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.2534959Z 2025-08-14T21:50:50.2535068Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2535267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2535339Z return mod(**inputs) 2025-08-14T21:50:50.2535623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2535690Z outputs = self.bert( 2025-08-14T21:50:50.2535976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2536048Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2536339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2536418Z layer_outputs = layer_module( 2025-08-14T21:50:50.2536656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2536743Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2537026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2537106Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2537402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2537472Z self_outputs = self.self( 2025-08-14T21:50:50.2537964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2538107Z return func(*args, **kwargs) 2025-08-14T21:50:50.2538504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.2538601Z key_layer = self.key(current_states) 2025-08-14T21:50:50.2538605Z 2025-08-14T21:50:50.2538716Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2538936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2539005Z return mod(**inputs) 2025-08-14T21:50:50.2539321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2539402Z outputs = self.bert( 2025-08-14T21:50:50.2539710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2539786Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2540084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2540159Z layer_outputs = layer_module( 2025-08-14T21:50:50.2540387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2540465Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2540750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2540837Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2541185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2541254Z self_outputs = self.self( 2025-08-14T21:50:50.2541505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2541606Z return func(*args, **kwargs) 2025-08-14T21:50:50.2541907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.2541988Z value_layer = self.value(current_states) 2025-08-14T21:50:50.2541992Z 2025-08-14T21:50:50.2542083Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2542172Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2542272Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2542500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2542569Z return mod(**inputs) 2025-08-14T21:50:50.2542851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2542923Z outputs = self.bert( 2025-08-14T21:50:50.2543200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2543310Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2543593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2543660Z layer_outputs = layer_module( 2025-08-14T21:50:50.2543882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2543957Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2544241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2544328Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2544643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.2544781Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.2545065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.2545145Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2545149Z 2025-08-14T21:50:50.2545255Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2545450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2545516Z return mod(**inputs) 2025-08-14T21:50:50.2545811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2545874Z outputs = self.bert( 2025-08-14T21:50:50.2546157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2546231Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2546506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2546584Z layer_outputs = layer_module( 2025-08-14T21:50:50.2546793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2546875Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2547149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2547247Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2547509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2547582Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2547906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2548015Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2548289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.2548375Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2548379Z 2025-08-14T21:50:50.2548476Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2548671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2548741Z return mod(**inputs) 2025-08-14T21:50:50.2549017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2549087Z outputs = self.bert( 2025-08-14T21:50:50.2549384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2549458Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2549741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2549808Z layer_outputs = layer_module( 2025-08-14T21:50:50.2550021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2550102Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2550377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2550463Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2550732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2550808Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2551122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2551221Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2551505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.2551616Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.2551820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.2551895Z return self.act(input) 2025-08-14T21:50:50.2551899Z 2025-08-14T21:50:50.2551999Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2552190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2552260Z return mod(**inputs) 2025-08-14T21:50:50.2552538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2552609Z outputs = self.bert( 2025-08-14T21:50:50.2552885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2552956Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2553244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2553337Z layer_outputs = layer_module( 2025-08-14T21:50:50.2553561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2553636Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2553936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2554026Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2554275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2554349Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2554662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2554789Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2555072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.2555151Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2555156Z 2025-08-14T21:50:50.2555254Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2555481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2555548Z return mod(**inputs) 2025-08-14T21:50:50.2555840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2555904Z outputs = self.bert( 2025-08-14T21:50:50.2556180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2556259Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2556538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2556626Z layer_outputs = layer_module( 2025-08-14T21:50:50.2556848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2556925Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2557209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2557287Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2557536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2557615Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2557923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2558054Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2558338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:50.2558412Z return input_tensor + hidden_states 2025-08-14T21:50:50.2558416Z 2025-08-14T21:50:50.2558518Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2558707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2558776Z return mod(**inputs) 2025-08-14T21:50:50.2559046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2559129Z outputs = self.bert( 2025-08-14T21:50:50.2559401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2559470Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2559738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2559833Z layer_outputs = layer_module( 2025-08-14T21:50:50.2560043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2560124Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2560392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2560471Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2560760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2560832Z self_outputs = self.self( 2025-08-14T21:50:50.2561075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2561148Z return func(*args, **kwargs) 2025-08-14T21:50:50.2561447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.2561537Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.2561541Z 2025-08-14T21:50:50.2561642Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2561837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2561912Z return mod(**inputs) 2025-08-14T21:50:50.2562197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2562274Z outputs = self.bert( 2025-08-14T21:50:50.2562555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2562649Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2562943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2563015Z layer_outputs = layer_module( 2025-08-14T21:50:50.2563230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2563314Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2580979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2581176Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2581553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2581635Z self_outputs = self.self( 2025-08-14T21:50:50.2581902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2581993Z return func(*args, **kwargs) 2025-08-14T21:50:50.2582296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.2582386Z key_layer = self.key(current_states) 2025-08-14T21:50:50.2582393Z 2025-08-14T21:50:50.2582508Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2582716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2582796Z return mod(**inputs) 2025-08-14T21:50:50.2583194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2583264Z outputs = self.bert( 2025-08-14T21:50:50.2583561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2583713Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2584013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2584093Z layer_outputs = layer_module( 2025-08-14T21:50:50.2584330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2584427Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2584730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2584823Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2585104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2585180Z self_outputs = self.self( 2025-08-14T21:50:50.2585452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2585559Z return func(*args, **kwargs) 2025-08-14T21:50:50.2585867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.2585962Z value_layer = self.value(current_states) 2025-08-14T21:50:50.2585966Z 2025-08-14T21:50:50.2586055Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2586148Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2586263Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2586480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2586560Z return mod(**inputs) 2025-08-14T21:50:50.2586898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2586981Z outputs = self.bert( 2025-08-14T21:50:50.2587285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2587362Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2587661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2587735Z layer_outputs = layer_module( 2025-08-14T21:50:50.2587962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2588052Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2588335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2588428Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2588714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.2588847Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.2589144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.2589228Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2589232Z 2025-08-14T21:50:50.2589341Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2589544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2589635Z return mod(**inputs) 2025-08-14T21:50:50.2589938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2590008Z outputs = self.bert( 2025-08-14T21:50:50.2590298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2590407Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2590688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2590767Z layer_outputs = layer_module( 2025-08-14T21:50:50.2590985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2591062Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2591357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2591451Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2591707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2591795Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2592166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2592276Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2592564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.2592645Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2592649Z 2025-08-14T21:50:50.2592761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2592962Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2593030Z return mod(**inputs) 2025-08-14T21:50:50.2593341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2593410Z outputs = self.bert( 2025-08-14T21:50:50.2593697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2593780Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2594062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2594155Z layer_outputs = layer_module( 2025-08-14T21:50:50.2594376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2594455Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2594747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2594830Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2595099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2595188Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2595522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2595651Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2595933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.2596068Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.2596289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.2596361Z return self.act(input) 2025-08-14T21:50:50.2596366Z 2025-08-14T21:50:50.2596477Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2596698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2596764Z return mod(**inputs) 2025-08-14T21:50:50.2597065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2597129Z outputs = self.bert( 2025-08-14T21:50:50.2597406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2597485Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2597760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2597835Z layer_outputs = layer_module( 2025-08-14T21:50:50.2598050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2598127Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2598431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2598512Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2598769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2598844Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2599147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2599286Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2599587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.2599675Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2599678Z 2025-08-14T21:50:50.2599781Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2599973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2600046Z return mod(**inputs) 2025-08-14T21:50:50.2600324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2600387Z outputs = self.bert( 2025-08-14T21:50:50.2600678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2600749Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2601040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2601110Z layer_outputs = layer_module( 2025-08-14T21:50:50.2601325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2601409Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2601686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2601773Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2602051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2602140Z self_outputs = self.self( 2025-08-14T21:50:50.2602382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2602453Z return func(*args, **kwargs) 2025-08-14T21:50:50.2602733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.2602846Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.2602850Z 2025-08-14T21:50:50.2602953Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2603156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2603221Z return mod(**inputs) 2025-08-14T21:50:50.2603509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2603586Z outputs = self.bert( 2025-08-14T21:50:50.2603871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2603954Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2604245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2604318Z layer_outputs = layer_module( 2025-08-14T21:50:50.2604564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2604643Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2604927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2605015Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2605300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2605380Z self_outputs = self.self( 2025-08-14T21:50:50.2605739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2605818Z return func(*args, **kwargs) 2025-08-14T21:50:50.2606144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.2606231Z key_layer = self.key(current_states) 2025-08-14T21:50:50.2606235Z 2025-08-14T21:50:50.2606356Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2606574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2606646Z return mod(**inputs) 2025-08-14T21:50:50.2606969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2607042Z outputs = self.bert( 2025-08-14T21:50:50.2607352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2607437Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2607728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2607811Z layer_outputs = layer_module( 2025-08-14T21:50:50.2608033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2608110Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2608406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2608539Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2608826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2608902Z self_outputs = self.self( 2025-08-14T21:50:50.2609142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2609251Z return func(*args, **kwargs) 2025-08-14T21:50:50.2609564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.2609646Z value_layer = self.value(current_states) 2025-08-14T21:50:50.2609650Z 2025-08-14T21:50:50.2609741Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2609826Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2609939Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2610148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2610220Z return mod(**inputs) 2025-08-14T21:50:50.2610533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2610602Z outputs = self.bert( 2025-08-14T21:50:50.2610933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2611020Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2611330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2611412Z layer_outputs = layer_module( 2025-08-14T21:50:50.2611642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2611723Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2612036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2612121Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2612460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.2612599Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.2612909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.2613002Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2613006Z 2025-08-14T21:50:50.2613114Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2613322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2613400Z return mod(**inputs) 2025-08-14T21:50:50.2613759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2613834Z outputs = self.bert( 2025-08-14T21:50:50.2614141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2614219Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2614527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2614599Z layer_outputs = layer_module( 2025-08-14T21:50:50.2614835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2614915Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2615223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2615340Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2615613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2615715Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2616057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2616164Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2616478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.2616563Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2616567Z 2025-08-14T21:50:50.2616670Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2616887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2616956Z return mod(**inputs) 2025-08-14T21:50:50.2617269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2617337Z outputs = self.bert( 2025-08-14T21:50:50.2617664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2617751Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2618051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2618127Z layer_outputs = layer_module( 2025-08-14T21:50:50.2618367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2618450Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2618754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2618859Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2619134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2619226Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2619557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2619672Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2619982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.2620103Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.2620332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.2620406Z return self.act(input) 2025-08-14T21:50:50.2620410Z 2025-08-14T21:50:50.2620517Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2620738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2620811Z return mod(**inputs) 2025-08-14T21:50:50.2621113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2621179Z outputs = self.bert( 2025-08-14T21:50:50.2621470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2621551Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2621854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2621932Z layer_outputs = layer_module( 2025-08-14T21:50:50.2622151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2622248Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2622539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2622621Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2622875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2622960Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2623270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2623412Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2623697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.2623778Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2623783Z 2025-08-14T21:50:50.2623894Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2624123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2624201Z return mod(**inputs) 2025-08-14T21:50:50.2624507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2624575Z outputs = self.bert( 2025-08-14T21:50:50.2624889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2624966Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2625286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2625369Z layer_outputs = layer_module( 2025-08-14T21:50:50.2625601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2625691Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2625987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2626068Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2626332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2626411Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2626729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2626862Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2627147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:50.2627235Z return input_tensor + hidden_states 2025-08-14T21:50:50.2627239Z 2025-08-14T21:50:50.2627340Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2627544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2627610Z return mod(**inputs) 2025-08-14T21:50:50.2627896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2627993Z outputs = self.bert( 2025-08-14T21:50:50.2628284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2628356Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2628659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2628751Z layer_outputs = layer_module( 2025-08-14T21:50:50.2628991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2629071Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2629373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2629466Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2629767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2629847Z self_outputs = self.self( 2025-08-14T21:50:50.2630102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2630176Z return func(*args, **kwargs) 2025-08-14T21:50:50.2630504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.2630602Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.2630606Z 2025-08-14T21:50:50.2630705Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2630909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2630973Z return mod(**inputs) 2025-08-14T21:50:50.2631267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2631332Z outputs = self.bert( 2025-08-14T21:50:50.2631641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2631725Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2632015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2632085Z layer_outputs = layer_module( 2025-08-14T21:50:50.2632309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2632386Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2632675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2632757Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2633041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2633119Z self_outputs = self.self( 2025-08-14T21:50:50.2633358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2633437Z return func(*args, **kwargs) 2025-08-14T21:50:50.2633723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.2633801Z key_layer = self.key(current_states) 2025-08-14T21:50:50.2633805Z 2025-08-14T21:50:50.2633913Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2634108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2634172Z return mod(**inputs) 2025-08-14T21:50:50.2634489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2634554Z outputs = self.bert( 2025-08-14T21:50:50.2634850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2634946Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2635232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2635315Z layer_outputs = layer_module( 2025-08-14T21:50:50.2635547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2635635Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2635936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2636028Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2636321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2636392Z self_outputs = self.self( 2025-08-14T21:50:50.2636630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2636726Z return func(*args, **kwargs) 2025-08-14T21:50:50.2637011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.2637097Z value_layer = self.value(current_states) 2025-08-14T21:50:50.2637101Z 2025-08-14T21:50:50.2637181Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2637261Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2637370Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2637565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2637860Z return mod(**inputs) 2025-08-14T21:50:50.2638513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2638605Z outputs = self.bert( 2025-08-14T21:50:50.2638921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2639000Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2639301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2639384Z layer_outputs = layer_module( 2025-08-14T21:50:50.2639621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2639716Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2640017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2640104Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2640421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.2640559Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.2640868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.2640957Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2640961Z 2025-08-14T21:50:50.2641069Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2641323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2641392Z return mod(**inputs) 2025-08-14T21:50:50.2641695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2641770Z outputs = self.bert( 2025-08-14T21:50:50.2642103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2642187Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2642491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2642564Z layer_outputs = layer_module( 2025-08-14T21:50:50.2642804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2642887Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2643197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2643283Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2643564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2643658Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2644032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2644150Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2644449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.2644533Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2644539Z 2025-08-14T21:50:50.2644652Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2644856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2644924Z return mod(**inputs) 2025-08-14T21:50:50.2645248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2645320Z outputs = self.bert( 2025-08-14T21:50:50.2645690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2645772Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2646073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2646155Z layer_outputs = layer_module( 2025-08-14T21:50:50.2646386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2646475Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2646778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2646866Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2647144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2647225Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2647557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2647671Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2647969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.2648122Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.2648346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.2648420Z return self.act(input) 2025-08-14T21:50:50.2648445Z 2025-08-14T21:50:50.2648565Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2648779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2648856Z return mod(**inputs) 2025-08-14T21:50:50.2649160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2649229Z outputs = self.bert( 2025-08-14T21:50:50.2649536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2649614Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2649912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2649994Z layer_outputs = layer_module( 2025-08-14T21:50:50.2650230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2650324Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2650611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2650690Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2650943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2651016Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2651330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2651457Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2651749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.2651839Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2651844Z 2025-08-14T21:50:50.2651944Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2652142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2652206Z return mod(**inputs) 2025-08-14T21:50:50.2652484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2652556Z outputs = self.bert( 2025-08-14T21:50:50.2652843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2652912Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2653190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2653257Z layer_outputs = layer_module( 2025-08-14T21:50:50.2653473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2653545Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2653814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2653897Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2654178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2654271Z self_outputs = self.self( 2025-08-14T21:50:50.2654506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2654575Z return func(*args, **kwargs) 2025-08-14T21:50:50.2654878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 223, in forward 2025-08-14T21:50:50.2654957Z query_layer = self.query(hidden_states) 2025-08-14T21:50:50.2654961Z 2025-08-14T21:50:50.2655059Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2655258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2655322Z return mod(**inputs) 2025-08-14T21:50:50.2655611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2655676Z outputs = self.bert( 2025-08-14T21:50:50.2655958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2656036Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2656305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2656398Z layer_outputs = layer_module( 2025-08-14T21:50:50.2656612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2656684Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2656957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2657033Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2657305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2657377Z self_outputs = self.self( 2025-08-14T21:50:50.2657620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2657697Z return func(*args, **kwargs) 2025-08-14T21:50:50.2657970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 252, in forward 2025-08-14T21:50:50.2658045Z key_layer = self.key(current_states) 2025-08-14T21:50:50.2658049Z 2025-08-14T21:50:50.2658152Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2658340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2658402Z return mod(**inputs) 2025-08-14T21:50:50.2658683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2658746Z outputs = self.bert( 2025-08-14T21:50:50.2659023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2659091Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2659366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2659440Z layer_outputs = layer_module( 2025-08-14T21:50:50.2659647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2659728Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2660000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2660094Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2660370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 372, in forward 2025-08-14T21:50:50.2660434Z self_outputs = self.self( 2025-08-14T21:50:50.2660658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:50:50.2660751Z return func(*args, **kwargs) 2025-08-14T21:50:50.2661023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 256, in forward 2025-08-14T21:50:50.2661105Z value_layer = self.value(current_states) 2025-08-14T21:50:50.2661108Z 2025-08-14T21:50:50.2661187Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2661261Z cudagraph partition due to non gpu ops 2025-08-14T21:50:50.2661364Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2661553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2661621Z return mod(**inputs) 2025-08-14T21:50:50.2661897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2661964Z outputs = self.bert( 2025-08-14T21:50:50.2662277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2662351Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2662637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2662713Z layer_outputs = layer_module( 2025-08-14T21:50:50.2662932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2663016Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2663297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 444, in forward 2025-08-14T21:50:50.2663375Z self_attention_outputs = self.attention( 2025-08-14T21:50:50.2663685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 381, in forward 2025-08-14T21:50:50.2663817Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:50:50.2664115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 329, in forward 2025-08-14T21:50:50.2664196Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2664199Z 2025-08-14T21:50:50.2664297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2664496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2664562Z return mod(**inputs) 2025-08-14T21:50:50.2664841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2664915Z outputs = self.bert( 2025-08-14T21:50:50.2665194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2665277Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2665554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2665624Z layer_outputs = layer_module( 2025-08-14T21:50:50.2665844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2665918Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2666230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2666309Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2666554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2666657Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2666963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2667062Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2667345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 397, in forward 2025-08-14T21:50:50.2667423Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2667426Z 2025-08-14T21:50:50.2667534Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2667730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2667794Z return mod(**inputs) 2025-08-14T21:50:50.2668082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2668147Z outputs = self.bert( 2025-08-14T21:50:50.2668450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2668523Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2668803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2668879Z layer_outputs = layer_module( 2025-08-14T21:50:50.2669092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2669168Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2669458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2669555Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2669816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2669890Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2670195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 481, in feed_forward_chunk 2025-08-14T21:50:50.2670302Z intermediate_output = self.intermediate(ln_output) 2025-08-14T21:50:50.2670582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 398, in forward 2025-08-14T21:50:50.2670701Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:50:50.2670906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:50:50.2670974Z return self.act(input) 2025-08-14T21:50:50.2670979Z 2025-08-14T21:50:50.2671087Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2671280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2671345Z return mod(**inputs) 2025-08-14T21:50:50.2671634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2671697Z outputs = self.bert( 2025-08-14T21:50:50.2671983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2672056Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2672397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2672477Z layer_outputs = layer_module( 2025-08-14T21:50:50.2672696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2672806Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2673098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2673177Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2673432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2673503Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2673815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2673954Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2674237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 410, in forward 2025-08-14T21:50:50.2674324Z hidden_states = self.dense(hidden_states) 2025-08-14T21:50:50.2674327Z 2025-08-14T21:50:50.2674452Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2674650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2674723Z return mod(**inputs) 2025-08-14T21:50:50.2675009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1597, in forward 2025-08-14T21:50:50.2675083Z outputs = self.bert( 2025-08-14T21:50:50.2675368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 856, in forward 2025-08-14T21:50:50.2675442Z encoder_outputs = self.encoder( 2025-08-14T21:50:50.2675755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 537, in forward 2025-08-14T21:50:50.2675826Z layer_outputs = layer_module( 2025-08-14T21:50:50.2676036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:50:50.2676118Z return super().__call__(*args, **kwargs) 2025-08-14T21:50:50.2676394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 474, in forward 2025-08-14T21:50:50.2676480Z layer_output = apply_chunking_to_forward( 2025-08-14T21:50:50.2676727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:50:50.2676804Z return forward_fn(*input_tensors) 2025-08-14T21:50:50.2677123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 482, in feed_forward_chunk 2025-08-14T21:50:50.2677253Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:50:50.2677546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 412, in forward 2025-08-14T21:50:50.2677624Z return input_tensor + hidden_states 2025-08-14T21:50:50.2677628Z 2025-08-14T21:50:50.2677728Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2677933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2677997Z return mod(**inputs) 2025-08-14T21:50:50.2678286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1611, in forward 2025-08-14T21:50:50.2678390Z logits = self.qa_outputs(sequence_output) 2025-08-14T21:50:50.2678393Z 2025-08-14T21:50:50.2678496Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2678701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2678797Z return mod(**inputs) 2025-08-14T21:50:50.2679087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1629, in forward 2025-08-14T21:50:50.2679200Z start_loss = loss_fct(start_logits, start_positions) 2025-08-14T21:50:50.2679204Z 2025-08-14T21:50:50.2679304Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:50:50.2679506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:50:50.2679571Z return mod(**inputs) 2025-08-14T21:50:50.2679859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/megatron_bert/modeling_megatron_bert.py", line 1630, in forward 2025-08-14T21:50:50.2679962Z end_loss = loss_fct(end_logits, end_positions) 2025-08-14T21:50:50.2679966Z 2025-08-14T21:50:59.9905455Z Compilation time (from dynamo_timed): 22.114874447 2025-08-14T21:50:59.9911640Z pass 2025-08-14T21:50:59.9914185Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:50:59.9915397Z TIMING: _recursive_pre_grad_passes:0.01089 _recursive_joint_graph_passes:1.11484 _recursive_post_grad_passes:0.13461 async_compile.wait:0.00314 code_gen:8.5036 inductor_compile:10.62007 backend_compile:16.78768 gc:0.00111 entire_frame_compile:22.11487 total_wall_time:22.11487 2025-08-14T21:50:59.9916292Z STATS: call_* op count: 724 | FakeTensorMode.__torch_dispatch__:28476 | FakeTensor.__torch_dispatch__:8921 | ProxyTorchDispatchMode.__torch_dispatch__:10973 2025-08-14T21:50:59.9921130Z Dynamo produced 1 graphs covering 724 ops with 0 graph breaks (0 unique) 2025-08-14T21:51:05.5828513Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:51:05.5829421Z from pkg_resources import resource_filename 2025-08-14T21:51:06.1507194Z 2025-08-14T21:51:06.8843665Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:51:06.8847364Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:51:06.8917630Z cpu eval MobileBertForMaskedLM 2025-08-14T21:51:07.1618281Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:51:07.3299271Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:51:07.4923519Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:51:33.1221713Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.1222264Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.1222647Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1223115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1223523Z return mod(**inputs) 2025-08-14T21:51:33.1224000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1224463Z outputs = self.mobilebert( 2025-08-14T21:51:33.1224905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-08-14T21:51:33.1225356Z embedding_output = self.embeddings( 2025-08-14T21:51:33.1225812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 199, in forward 2025-08-14T21:51:33.1226591Z inputs_embeds = torch.cat( 2025-08-14T21:51:33.1226721Z 2025-08-14T21:51:33.1226843Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1227238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1227653Z return mod(**inputs) 2025-08-14T21:51:33.1228084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-08-14T21:51:33.1228564Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:51:33.1229039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-08-14T21:51:33.1229522Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:51:33.1230013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 632, in forward 2025-08-14T21:51:33.1230596Z hidden_states = hidden_states.matmul(torch.cat([self.decoder.weight.t(), self.dense.weight], dim=0)) 2025-08-14T21:51:33.1230891Z 2025-08-14T21:51:33.1231004Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1231428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1231785Z return mod(**inputs) 2025-08-14T21:51:33.1232326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1232774Z outputs = self.mobilebert( 2025-08-14T21:51:33.1233205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-08-14T21:51:33.1233661Z embedding_output = self.embeddings( 2025-08-14T21:51:33.1234105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 208, in forward 2025-08-14T21:51:33.1234625Z inputs_embeds = self.embedding_transformation(inputs_embeds) 2025-08-14T21:51:33.1234823Z 2025-08-14T21:51:33.1234937Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1235381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1235778Z return mod(**inputs) 2025-08-14T21:51:33.1236215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1236665Z outputs = self.mobilebert( 2025-08-14T21:51:33.1237104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-08-14T21:51:33.1237559Z embedding_output = self.embeddings( 2025-08-14T21:51:33.1238191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 215, in forward 2025-08-14T21:51:33.1238667Z embeddings = self.LayerNorm(embeddings) 2025-08-14T21:51:33.1239128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1239616Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1239791Z 2025-08-14T21:51:33.1239907Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1240365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1240722Z return mod(**inputs) 2025-08-14T21:51:33.1241145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1241601Z outputs = self.mobilebert( 2025-08-14T21:51:33.1242039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1242538Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1242965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1243429Z layer_outputs = layer_module( 2025-08-14T21:51:33.1243859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.1244448Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.1245006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.1245639Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.1246125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.1246602Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.1246758Z 2025-08-14T21:51:33.1246870Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1247257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1247618Z return mod(**inputs) 2025-08-14T21:51:33.1248047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1248544Z outputs = self.mobilebert( 2025-08-14T21:51:33.1248988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1249446Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1249896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1250349Z layer_outputs = layer_module( 2025-08-14T21:51:33.1250784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1251266Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1251762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.1252499Z self_outputs = self.self( 2025-08-14T21:51:33.1252933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.1253372Z self.value(value_tensor) 2025-08-14T21:51:33.1253503Z 2025-08-14T21:51:33.1253613Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1254003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1254365Z return mod(**inputs) 2025-08-14T21:51:33.1254791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1255240Z outputs = self.mobilebert( 2025-08-14T21:51:33.1255668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1256115Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1256550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1256998Z layer_outputs = layer_module( 2025-08-14T21:51:33.1257430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.1257950Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.1258487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.1259016Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.1259495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.1259952Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.1260124Z 2025-08-14T21:51:33.1260245Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1260620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1260951Z return mod(**inputs) 2025-08-14T21:51:33.1261356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1261789Z outputs = self.mobilebert( 2025-08-14T21:51:33.1262202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1262640Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1263063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1263494Z layer_outputs = layer_module( 2025-08-14T21:51:33.1263915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.1264458Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.1264979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.1265447Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.1265902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.1266347Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.1266788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1267264Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1267424Z 2025-08-14T21:51:33.1267533Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1267911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1268255Z return mod(**inputs) 2025-08-14T21:51:33.1268655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1269092Z outputs = self.mobilebert( 2025-08-14T21:51:33.1269510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1269946Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1270366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1270798Z layer_outputs = layer_module( 2025-08-14T21:51:33.1271235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1271688Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1272120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.1272550Z self_outputs = self.self( 2025-08-14T21:51:33.1272968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.1273390Z self.query(query_tensor) 2025-08-14T21:51:33.1273576Z 2025-08-14T21:51:33.1273684Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1274057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1274393Z return mod(**inputs) 2025-08-14T21:51:33.1274789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1275277Z outputs = self.mobilebert( 2025-08-14T21:51:33.1275691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1276102Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1276498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1276907Z layer_outputs = layer_module( 2025-08-14T21:51:33.1277306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1277719Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1278137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.1278542Z self_outputs = self.self( 2025-08-14T21:51:33.1278958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.1279358Z self.key(key_tensor) 2025-08-14T21:51:33.1279470Z 2025-08-14T21:51:33.1279557Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.1279781Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.1280021Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1280395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1280733Z return mod(**inputs) 2025-08-14T21:51:33.1281139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1281561Z outputs = self.mobilebert( 2025-08-14T21:51:33.1281997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1282438Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1282859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1283296Z layer_outputs = layer_module( 2025-08-14T21:51:33.1283723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1284170Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1284607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.1285093Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.1285679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.1286147Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1286306Z 2025-08-14T21:51:33.1286418Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1286815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1287162Z return mod(**inputs) 2025-08-14T21:51:33.1287574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1288002Z outputs = self.mobilebert( 2025-08-14T21:51:33.1288421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1288918Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1289333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1289764Z layer_outputs = layer_module( 2025-08-14T21:51:33.1290211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1290665Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1291125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.1291617Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.1292104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.1292596Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1293080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1293552Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1293710Z 2025-08-14T21:51:33.1293831Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1294229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1294581Z return mod(**inputs) 2025-08-14T21:51:33.1294998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1295440Z outputs = self.mobilebert( 2025-08-14T21:51:33.1295858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1296305Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1296741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1297185Z layer_outputs = layer_module( 2025-08-14T21:51:33.1297632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1298106Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1298572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1299055Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1299539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1299994Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1300149Z 2025-08-14T21:51:33.1300277Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1300659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1301014Z return mod(**inputs) 2025-08-14T21:51:33.1301437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1301882Z outputs = self.mobilebert( 2025-08-14T21:51:33.1302303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1302748Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1303189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1303622Z layer_outputs = layer_module( 2025-08-14T21:51:33.1304057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1304550Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1305024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1305520Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1306006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1306494Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1306673Z 2025-08-14T21:51:33.1306792Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1307170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1307590Z return mod(**inputs) 2025-08-14T21:51:33.1308019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1308454Z outputs = self.mobilebert( 2025-08-14T21:51:33.1308882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1309327Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1309786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1310229Z layer_outputs = layer_module( 2025-08-14T21:51:33.1310667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1311139Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1311604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1312098Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1312619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.1313082Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1313234Z 2025-08-14T21:51:33.1313345Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1313729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1314076Z return mod(**inputs) 2025-08-14T21:51:33.1314502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1314943Z outputs = self.mobilebert( 2025-08-14T21:51:33.1315368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1315816Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1316254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1316693Z layer_outputs = layer_module( 2025-08-14T21:51:33.1317129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1317594Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1318034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1318527Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1319022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.1319540Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1320027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1320517Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1320692Z 2025-08-14T21:51:33.1320822Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1321211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1321550Z return mod(**inputs) 2025-08-14T21:51:33.1321968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1322412Z outputs = self.mobilebert( 2025-08-14T21:51:33.1322832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1323279Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1323713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1324153Z layer_outputs = layer_module( 2025-08-14T21:51:33.1324576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1325057Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1325632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1326140Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1326612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1327072Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1327221Z 2025-08-14T21:51:33.1327337Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1327735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1328101Z return mod(**inputs) 2025-08-14T21:51:33.1328537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1329009Z outputs = self.mobilebert( 2025-08-14T21:51:33.1329432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1329895Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1330335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1330791Z layer_outputs = layer_module( 2025-08-14T21:51:33.1331217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1331706Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1332174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1332672Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1333161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1333655Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1333832Z 2025-08-14T21:51:33.1333951Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1334341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1334705Z return mod(**inputs) 2025-08-14T21:51:33.1335168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1335617Z outputs = self.mobilebert( 2025-08-14T21:51:33.1336044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1336517Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1336961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1337394Z layer_outputs = layer_module( 2025-08-14T21:51:33.1338060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1338540Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1339005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1339498Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1340000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.1340463Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1340617Z 2025-08-14T21:51:33.1340739Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1341195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1341552Z return mod(**inputs) 2025-08-14T21:51:33.1341968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1342405Z outputs = self.mobilebert( 2025-08-14T21:51:33.1342835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1343285Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1343717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1344189Z layer_outputs = layer_module( 2025-08-14T21:51:33.1344638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1345113Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1345585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1346082Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1346591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.1347086Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1347579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1348042Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1348206Z 2025-08-14T21:51:33.1348317Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1348700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1349040Z return mod(**inputs) 2025-08-14T21:51:33.1349457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1349897Z outputs = self.mobilebert( 2025-08-14T21:51:33.1350322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1350802Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1351237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1351677Z layer_outputs = layer_module( 2025-08-14T21:51:33.1352110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1352616Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1353088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1353585Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1354069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1354543Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1354706Z 2025-08-14T21:51:33.1354821Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1355214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1355563Z return mod(**inputs) 2025-08-14T21:51:33.1355992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1356451Z outputs = self.mobilebert( 2025-08-14T21:51:33.1356898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1357347Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1357784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1358226Z layer_outputs = layer_module( 2025-08-14T21:51:33.1358647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1359118Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1359600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1360092Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1360572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1361070Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1361259Z 2025-08-14T21:51:33.1361398Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1361797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1362149Z return mod(**inputs) 2025-08-14T21:51:33.1362577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1363035Z outputs = self.mobilebert( 2025-08-14T21:51:33.1363463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1363920Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1364375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1364829Z layer_outputs = layer_module( 2025-08-14T21:51:33.1365261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1365813Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1366280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1366813Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1367295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.1367759Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1367932Z 2025-08-14T21:51:33.1368052Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1368443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1368803Z return mod(**inputs) 2025-08-14T21:51:33.1369220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1369673Z outputs = self.mobilebert( 2025-08-14T21:51:33.1370104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1370571Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1371008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1371473Z layer_outputs = layer_module( 2025-08-14T21:51:33.1371906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1372405Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1372877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1373393Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1373890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.1374400Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1374905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1375388Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1375570Z 2025-08-14T21:51:33.1375683Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1376091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1376432Z return mod(**inputs) 2025-08-14T21:51:33.1376832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1377260Z outputs = self.mobilebert( 2025-08-14T21:51:33.1377675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1378104Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1378532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1378973Z layer_outputs = layer_module( 2025-08-14T21:51:33.1379412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.1379904Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.1380400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1380866Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1381018Z 2025-08-14T21:51:33.1381136Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1381511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1381883Z return mod(**inputs) 2025-08-14T21:51:33.1382306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1382745Z outputs = self.mobilebert( 2025-08-14T21:51:33.1383177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1383644Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1384083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1384519Z layer_outputs = layer_module( 2025-08-14T21:51:33.1384955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.1385444Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.1385925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1386413Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1386594Z 2025-08-14T21:51:33.1386704Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1387086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1387430Z return mod(**inputs) 2025-08-14T21:51:33.1387864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1388313Z outputs = self.mobilebert( 2025-08-14T21:51:33.1388739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1389182Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1389627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1390085Z layer_outputs = layer_module( 2025-08-14T21:51:33.1390548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1391094Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1391633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.1392107Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.1392269Z 2025-08-14T21:51:33.1392380Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1392769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1393120Z return mod(**inputs) 2025-08-14T21:51:33.1393542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1393978Z outputs = self.mobilebert( 2025-08-14T21:51:33.1394407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1394855Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1395284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1395724Z layer_outputs = layer_module( 2025-08-14T21:51:33.1396157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1396692Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1397220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.1397741Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.1398239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1398705Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1398886Z 2025-08-14T21:51:33.1398998Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1399384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1399732Z return mod(**inputs) 2025-08-14T21:51:33.1400150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1400591Z outputs = self.mobilebert( 2025-08-14T21:51:33.1401032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1401482Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1401913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1402357Z layer_outputs = layer_module( 2025-08-14T21:51:33.1402794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1403354Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1403891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.1404390Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.1404891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.1405355Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1405587Z 2025-08-14T21:51:33.1405704Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1406115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1406494Z return mod(**inputs) 2025-08-14T21:51:33.1406908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1407367Z outputs = self.mobilebert( 2025-08-14T21:51:33.1407808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1408287Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1408723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1409191Z layer_outputs = layer_module( 2025-08-14T21:51:33.1409634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1410188Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1410729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.1411254Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.1411759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.1412293Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1412788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1413280Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1413437Z 2025-08-14T21:51:33.1413557Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1413955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1414318Z return mod(**inputs) 2025-08-14T21:51:33.1414753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1415213Z outputs = self.mobilebert( 2025-08-14T21:51:33.1415634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1416091Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1416532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1416993Z layer_outputs = layer_module( 2025-08-14T21:51:33.1417422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.1417959Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.1418501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.1419005Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.1419481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.1419937Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.1420084Z 2025-08-14T21:51:33.1420201Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1420576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1420935Z return mod(**inputs) 2025-08-14T21:51:33.1421351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1421789Z outputs = self.mobilebert( 2025-08-14T21:51:33.1422230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1422684Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1423122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1423561Z layer_outputs = layer_module( 2025-08-14T21:51:33.1423996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1424459Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1424919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.1425356Z self_outputs = self.self( 2025-08-14T21:51:33.1425799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.1426249Z self.value(value_tensor) 2025-08-14T21:51:33.1426375Z 2025-08-14T21:51:33.1426495Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1426875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1427233Z return mod(**inputs) 2025-08-14T21:51:33.1427654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1428092Z outputs = self.mobilebert( 2025-08-14T21:51:33.1428520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1428987Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1429424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1429860Z layer_outputs = layer_module( 2025-08-14T21:51:33.1430313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.1430857Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.1431401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.1431885Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.1432367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.1432819Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.1432966Z 2025-08-14T21:51:33.1433086Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1433468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1433818Z return mod(**inputs) 2025-08-14T21:51:33.1434248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1434688Z outputs = self.mobilebert( 2025-08-14T21:51:33.1435123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1435562Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1436009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1436444Z layer_outputs = layer_module( 2025-08-14T21:51:33.1436882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.1437443Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.1438130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.1438622Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.1439111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.1439573Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.1440028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1440503Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1440669Z 2025-08-14T21:51:33.1440783Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1441170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1441517Z return mod(**inputs) 2025-08-14T21:51:33.1441947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1442393Z outputs = self.mobilebert( 2025-08-14T21:51:33.1442826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1443275Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1443717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1444165Z layer_outputs = layer_module( 2025-08-14T21:51:33.1444665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1445131Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1445664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.1446152Z self_outputs = self.self( 2025-08-14T21:51:33.1446584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.1447025Z self.query(query_tensor) 2025-08-14T21:51:33.1447149Z 2025-08-14T21:51:33.1447268Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1447663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1448036Z return mod(**inputs) 2025-08-14T21:51:33.1448469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1448919Z outputs = self.mobilebert( 2025-08-14T21:51:33.1449350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1449816Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1450301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1450763Z layer_outputs = layer_module( 2025-08-14T21:51:33.1451191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1451670Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1452129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.1452586Z self_outputs = self.self( 2025-08-14T21:51:33.1453011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.1453457Z self.key(key_tensor) 2025-08-14T21:51:33.1453597Z 2025-08-14T21:51:33.1453695Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.1453928Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.1454184Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1454572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1454938Z return mod(**inputs) 2025-08-14T21:51:33.1455355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1455808Z outputs = self.mobilebert( 2025-08-14T21:51:33.1456237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1456696Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1457139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1457603Z layer_outputs = layer_module( 2025-08-14T21:51:33.1458044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1458507Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1458967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.1459469Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.1459964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.1460449Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1460609Z 2025-08-14T21:51:33.1460721Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1461109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1461476Z return mod(**inputs) 2025-08-14T21:51:33.1461897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1462346Z outputs = self.mobilebert( 2025-08-14T21:51:33.1462774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1463218Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1463655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1464104Z layer_outputs = layer_module( 2025-08-14T21:51:33.1464529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1464985Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1465440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.1465956Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.1466443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.1466955Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1467452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1467920Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1468078Z 2025-08-14T21:51:33.1468190Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1468576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1468949Z return mod(**inputs) 2025-08-14T21:51:33.1469372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1469818Z outputs = self.mobilebert( 2025-08-14T21:51:33.1470246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1470698Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1471133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1471585Z layer_outputs = layer_module( 2025-08-14T21:51:33.1472027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1472503Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1472973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1473463Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1473953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1474415Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1474567Z 2025-08-14T21:51:33.1474678Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1475063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1475415Z return mod(**inputs) 2025-08-14T21:51:33.1475862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1476313Z outputs = self.mobilebert( 2025-08-14T21:51:33.1476760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1477237Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1477672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1478122Z layer_outputs = layer_module( 2025-08-14T21:51:33.1478568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1479033Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1479489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1479977Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1480463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1480957Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1481137Z 2025-08-14T21:51:33.1481252Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1481655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1482007Z return mod(**inputs) 2025-08-14T21:51:33.1482417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1482862Z outputs = self.mobilebert( 2025-08-14T21:51:33.1483295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1483738Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1484166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1484627Z layer_outputs = layer_module( 2025-08-14T21:51:33.1485071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1485626Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1486085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1486583Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1487083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.1487538Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1487699Z 2025-08-14T21:51:33.1487811Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1488199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1488548Z return mod(**inputs) 2025-08-14T21:51:33.1488962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1489410Z outputs = self.mobilebert( 2025-08-14T21:51:33.1489841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1490288Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1490719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1491196Z layer_outputs = layer_module( 2025-08-14T21:51:33.1491631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1492105Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1492571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1493086Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1493569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.1494053Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1494536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1495002Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1495159Z 2025-08-14T21:51:33.1495279Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1495659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1496019Z return mod(**inputs) 2025-08-14T21:51:33.1496437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1496910Z outputs = self.mobilebert( 2025-08-14T21:51:33.1497347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1497805Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1498229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1498676Z layer_outputs = layer_module( 2025-08-14T21:51:33.1499116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1499609Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1500078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1500550Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1501021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1501470Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1501622Z 2025-08-14T21:51:33.1501734Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1502118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1502465Z return mod(**inputs) 2025-08-14T21:51:33.1502885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1503328Z outputs = self.mobilebert( 2025-08-14T21:51:33.1503771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1504206Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1504633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1505076Z layer_outputs = layer_module( 2025-08-14T21:51:33.1505508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1505986Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1506441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1506944Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1507421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1507898Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1508109Z 2025-08-14T21:51:33.1508220Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1508612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1508971Z return mod(**inputs) 2025-08-14T21:51:33.1509388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1509842Z outputs = self.mobilebert( 2025-08-14T21:51:33.1510278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1510732Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1511168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1511622Z layer_outputs = layer_module( 2025-08-14T21:51:33.1512066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1512559Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1513020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1513526Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1514024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.1514489Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1514640Z 2025-08-14T21:51:33.1514750Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1515195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1515547Z return mod(**inputs) 2025-08-14T21:51:33.1515960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1516404Z outputs = self.mobilebert( 2025-08-14T21:51:33.1516833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1517280Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1517705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1518150Z layer_outputs = layer_module( 2025-08-14T21:51:33.1518583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1519037Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1519506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1520006Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1520501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.1520985Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1521475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1521946Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1522126Z 2025-08-14T21:51:33.1522246Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1522627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1522982Z return mod(**inputs) 2025-08-14T21:51:33.1523402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1523870Z outputs = self.mobilebert( 2025-08-14T21:51:33.1524291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1524733Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1525167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1525690Z layer_outputs = layer_module( 2025-08-14T21:51:33.1526142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1526614Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1527082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1527568Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1528079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1528543Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1528691Z 2025-08-14T21:51:33.1528811Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1529186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1529535Z return mod(**inputs) 2025-08-14T21:51:33.1529947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1530378Z outputs = self.mobilebert( 2025-08-14T21:51:33.1530828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1531287Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1531725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1532172Z layer_outputs = layer_module( 2025-08-14T21:51:33.1532609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1533093Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1533551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1534051Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1534531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1535025Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1535203Z 2025-08-14T21:51:33.1535314Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1535710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1536077Z return mod(**inputs) 2025-08-14T21:51:33.1536495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1536942Z outputs = self.mobilebert( 2025-08-14T21:51:33.1537368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1538114Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1538546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1539011Z layer_outputs = layer_module( 2025-08-14T21:51:33.1539510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1539993Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1540453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1540959Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1541459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.1541922Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1542074Z 2025-08-14T21:51:33.1542187Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1542578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1542953Z return mod(**inputs) 2025-08-14T21:51:33.1543366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1543843Z outputs = self.mobilebert( 2025-08-14T21:51:33.1544277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1544719Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1545148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1545588Z layer_outputs = layer_module( 2025-08-14T21:51:33.1546021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1546481Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1546966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1547478Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1547978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.1548475Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1548967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1549432Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1549593Z 2025-08-14T21:51:33.1549714Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1550096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1550448Z return mod(**inputs) 2025-08-14T21:51:33.1550870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1551322Z outputs = self.mobilebert( 2025-08-14T21:51:33.1551750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1552198Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1552640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1553086Z layer_outputs = layer_module( 2025-08-14T21:51:33.1553566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.1554067Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.1554573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1556101Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1556259Z 2025-08-14T21:51:33.1556373Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1556762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1557110Z return mod(**inputs) 2025-08-14T21:51:33.1557521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1557964Z outputs = self.mobilebert( 2025-08-14T21:51:33.1558393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1558840Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1559282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1559735Z layer_outputs = layer_module( 2025-08-14T21:51:33.1560194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.1560688Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.1561182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1561671Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1561847Z 2025-08-14T21:51:33.1561965Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1562348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1562699Z return mod(**inputs) 2025-08-14T21:51:33.1563142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1563582Z outputs = self.mobilebert( 2025-08-14T21:51:33.1564022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1564464Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1564904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1565350Z layer_outputs = layer_module( 2025-08-14T21:51:33.1565878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1566433Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1566986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.1567462Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.1567638Z 2025-08-14T21:51:33.1567751Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1568145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1568491Z return mod(**inputs) 2025-08-14T21:51:33.1568918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1569369Z outputs = self.mobilebert( 2025-08-14T21:51:33.1569801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1570276Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1570712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1571161Z layer_outputs = layer_module( 2025-08-14T21:51:33.1571590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1572140Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1572675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.1573180Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.1573664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1574143Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1574332Z 2025-08-14T21:51:33.1574446Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1574842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1575201Z return mod(**inputs) 2025-08-14T21:51:33.1575642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1576102Z outputs = self.mobilebert( 2025-08-14T21:51:33.1576530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1576981Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1577423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1577884Z layer_outputs = layer_module( 2025-08-14T21:51:33.1578310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1578876Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1579421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.1579927Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.1580414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.1580888Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1581045Z 2025-08-14T21:51:33.1581155Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1581545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1581893Z return mod(**inputs) 2025-08-14T21:51:33.1582311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1582752Z outputs = self.mobilebert( 2025-08-14T21:51:33.1583168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1583618Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1584053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1584491Z layer_outputs = layer_module( 2025-08-14T21:51:33.1584916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1585455Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1586016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.1586513Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.1587004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.1587522Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1588013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1588485Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1588640Z 2025-08-14T21:51:33.1588751Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1589135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1589486Z return mod(**inputs) 2025-08-14T21:51:33.1589898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1590342Z outputs = self.mobilebert( 2025-08-14T21:51:33.1590768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1591211Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1591660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1592114Z layer_outputs = layer_module( 2025-08-14T21:51:33.1592553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.1593094Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.1593636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.1594125Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.1594632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.1595083Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.1595237Z 2025-08-14T21:51:33.1595354Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1595742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1596092Z return mod(**inputs) 2025-08-14T21:51:33.1596502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1596945Z outputs = self.mobilebert( 2025-08-14T21:51:33.1597373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1597816Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1598247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1598691Z layer_outputs = layer_module( 2025-08-14T21:51:33.1599127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1599582Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1600055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.1600497Z self_outputs = self.self( 2025-08-14T21:51:33.1600924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.1601388Z self.value(value_tensor) 2025-08-14T21:51:33.1601520Z 2025-08-14T21:51:33.1601630Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1602023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1602400Z return mod(**inputs) 2025-08-14T21:51:33.1602813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1603262Z outputs = self.mobilebert( 2025-08-14T21:51:33.1603694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1604141Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1604579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1605026Z layer_outputs = layer_module( 2025-08-14T21:51:33.1605544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.1606101Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.1606657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.1607183Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.1607678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.1608129Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.1608287Z 2025-08-14T21:51:33.1608400Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1608791Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1609141Z return mod(**inputs) 2025-08-14T21:51:33.1609559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1610028Z outputs = self.mobilebert( 2025-08-14T21:51:33.1610469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1610898Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1611321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1611753Z layer_outputs = layer_module( 2025-08-14T21:51:33.1612169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.1612695Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.1613233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.1613728Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.1614222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.1614676Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.1615174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1615669Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1615821Z 2025-08-14T21:51:33.1615928Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1616325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1616724Z return mod(**inputs) 2025-08-14T21:51:33.1617147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1617613Z outputs = self.mobilebert( 2025-08-14T21:51:33.1618044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1618540Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1618975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1619446Z layer_outputs = layer_module( 2025-08-14T21:51:33.1619880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1620363Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1620809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.1621277Z self_outputs = self.self( 2025-08-14T21:51:33.1621744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.1622208Z self.query(query_tensor) 2025-08-14T21:51:33.1622336Z 2025-08-14T21:51:33.1622449Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1622870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1623240Z return mod(**inputs) 2025-08-14T21:51:33.1623638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1624068Z outputs = self.mobilebert( 2025-08-14T21:51:33.1624482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1624917Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1625336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1625804Z layer_outputs = layer_module( 2025-08-14T21:51:33.1626234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1626681Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1627131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.1627559Z self_outputs = self.self( 2025-08-14T21:51:33.1627975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.1628393Z self.key(key_tensor) 2025-08-14T21:51:33.1628512Z 2025-08-14T21:51:33.1628598Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.1628826Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.1629067Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1629458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1629807Z return mod(**inputs) 2025-08-14T21:51:33.1630228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1630651Z outputs = self.mobilebert( 2025-08-14T21:51:33.1631068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1631505Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1631922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1632386Z layer_outputs = layer_module( 2025-08-14T21:51:33.1632807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1633251Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1633680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.1634192Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.1634673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.1635123Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1635267Z 2025-08-14T21:51:33.1635375Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1635751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1636091Z return mod(**inputs) 2025-08-14T21:51:33.1636486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1636920Z outputs = self.mobilebert( 2025-08-14T21:51:33.1637335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1638051Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1638481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1638927Z layer_outputs = layer_module( 2025-08-14T21:51:33.1639363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1639821Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1640270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.1640772Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.1641310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.1641819Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1642317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1642784Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1642941Z 2025-08-14T21:51:33.1643062Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1643441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1643796Z return mod(**inputs) 2025-08-14T21:51:33.1644216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1644665Z outputs = self.mobilebert( 2025-08-14T21:51:33.1645090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1645603Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1646054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1646496Z layer_outputs = layer_module( 2025-08-14T21:51:33.1646935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1647411Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1647881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1648406Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1648894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1649386Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1649537Z 2025-08-14T21:51:33.1649658Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1650037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1650387Z return mod(**inputs) 2025-08-14T21:51:33.1650807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1651244Z outputs = self.mobilebert( 2025-08-14T21:51:33.1651672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1652120Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1652559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1652994Z layer_outputs = layer_module( 2025-08-14T21:51:33.1653449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1653921Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1654387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1654866Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1655334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1655805Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1655976Z 2025-08-14T21:51:33.1656093Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1656482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1656859Z return mod(**inputs) 2025-08-14T21:51:33.1657275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1657732Z outputs = self.mobilebert( 2025-08-14T21:51:33.1658161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1658620Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1659056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1659506Z layer_outputs = layer_module( 2025-08-14T21:51:33.1659936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1660426Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1660883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1661404Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1661897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.1662379Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1662524Z 2025-08-14T21:51:33.1662633Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1663030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1663403Z return mod(**inputs) 2025-08-14T21:51:33.1663821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1664270Z outputs = self.mobilebert( 2025-08-14T21:51:33.1664704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1665175Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1665604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1666048Z layer_outputs = layer_module( 2025-08-14T21:51:33.1666479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1666946Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1667403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1667903Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1668401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.1668900Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1669408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1669878Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1670040Z 2025-08-14T21:51:33.1670161Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1670381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1670465Z return mod(**inputs) 2025-08-14T21:51:33.1670769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1670850Z outputs = self.mobilebert( 2025-08-14T21:51:33.1671174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1671255Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1671558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1671643Z layer_outputs = layer_module( 2025-08-14T21:51:33.1671943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1672053Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1672350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1672472Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1672778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1672870Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1672875Z 2025-08-14T21:51:33.1672992Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1673203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1673276Z return mod(**inputs) 2025-08-14T21:51:33.1673580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1673657Z outputs = self.mobilebert( 2025-08-14T21:51:33.1673954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1674065Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1674369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1674454Z layer_outputs = layer_module( 2025-08-14T21:51:33.1674778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1674878Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1675182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1675301Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1675607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1675729Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1675734Z 2025-08-14T21:51:33.1675844Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1676067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1676140Z return mod(**inputs) 2025-08-14T21:51:33.1676466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1676554Z outputs = self.mobilebert( 2025-08-14T21:51:33.1676858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1676947Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1677249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1677330Z layer_outputs = layer_module( 2025-08-14T21:51:33.1677645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1677745Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1678075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1678214Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1678517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.1678619Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1678623Z 2025-08-14T21:51:33.1678734Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1678952Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1679026Z return mod(**inputs) 2025-08-14T21:51:33.1679328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1679412Z outputs = self.mobilebert( 2025-08-14T21:51:33.1679713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1679793Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1680104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1680181Z layer_outputs = layer_module( 2025-08-14T21:51:33.1680486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1680585Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1680911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1681049Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1681352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.1681505Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1681807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1681906Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1681910Z 2025-08-14T21:51:33.1682027Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1682241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1682311Z return mod(**inputs) 2025-08-14T21:51:33.1682623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1682697Z outputs = self.mobilebert( 2025-08-14T21:51:33.1683006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1683085Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1683403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1683490Z layer_outputs = layer_module( 2025-08-14T21:51:33.1683793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1683901Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1684199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1684320Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1684644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1684736Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1684742Z 2025-08-14T21:51:33.1684852Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1685073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1685143Z return mod(**inputs) 2025-08-14T21:51:33.1685528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1685617Z outputs = self.mobilebert( 2025-08-14T21:51:33.1685920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1686012Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1686312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1686399Z layer_outputs = layer_module( 2025-08-14T21:51:33.1686703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1686803Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1687112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1687229Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1687527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1687683Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1687687Z 2025-08-14T21:51:33.1687797Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1688019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1688090Z return mod(**inputs) 2025-08-14T21:51:33.1688414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1688499Z outputs = self.mobilebert( 2025-08-14T21:51:33.1688796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1688880Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1689174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1689252Z layer_outputs = layer_module( 2025-08-14T21:51:33.1689555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1689653Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1689948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1690109Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1690407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.1690504Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1690508Z 2025-08-14T21:51:33.1690616Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1690827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1690909Z return mod(**inputs) 2025-08-14T21:51:33.1691205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1691288Z outputs = self.mobilebert( 2025-08-14T21:51:33.1691604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1691687Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1691998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1692077Z layer_outputs = layer_module( 2025-08-14T21:51:33.1692378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1692484Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1692786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1692925Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1693229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.1693360Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1693671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1693771Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1693775Z 2025-08-14T21:51:33.1693895Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1694109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1694181Z return mod(**inputs) 2025-08-14T21:51:33.1694521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1694598Z outputs = self.mobilebert( 2025-08-14T21:51:33.1694906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1695003Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1695309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1695393Z layer_outputs = layer_module( 2025-08-14T21:51:33.1695696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.1695826Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.1696138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1696227Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1696231Z 2025-08-14T21:51:33.1696347Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1696573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1696647Z return mod(**inputs) 2025-08-14T21:51:33.1696976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1697055Z outputs = self.mobilebert( 2025-08-14T21:51:33.1697364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1697441Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1697740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1697825Z layer_outputs = layer_module( 2025-08-14T21:51:33.1698123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.1698265Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.1698574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1698691Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1698695Z 2025-08-14T21:51:33.1698811Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1699031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1699104Z return mod(**inputs) 2025-08-14T21:51:33.1699410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1699489Z outputs = self.mobilebert( 2025-08-14T21:51:33.1699794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1699873Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1700175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1700261Z layer_outputs = layer_module( 2025-08-14T21:51:33.1700558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1700730Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1701034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.1701157Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.1701161Z 2025-08-14T21:51:33.1701276Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1701503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1701574Z return mod(**inputs) 2025-08-14T21:51:33.1701913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1701990Z outputs = self.mobilebert( 2025-08-14T21:51:33.1702298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1702375Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1702674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1702762Z layer_outputs = layer_module( 2025-08-14T21:51:33.1703059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1703226Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1703533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.1703683Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.1703998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1704097Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1704101Z 2025-08-14T21:51:33.1704213Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1704445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1704518Z return mod(**inputs) 2025-08-14T21:51:33.1704830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1704905Z outputs = self.mobilebert( 2025-08-14T21:51:33.1705228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1705320Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1705619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1705696Z layer_outputs = layer_module( 2025-08-14T21:51:33.1706002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1706167Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1706471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.1706599Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.1706898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.1706997Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1707002Z 2025-08-14T21:51:33.1707109Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1707327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1707397Z return mod(**inputs) 2025-08-14T21:51:33.1707694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1707777Z outputs = self.mobilebert( 2025-08-14T21:51:33.1708102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1708188Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1708489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1708585Z layer_outputs = layer_module( 2025-08-14T21:51:33.1708893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1709059Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1709354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.1709491Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.1709789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.1709922Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1710222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1710322Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1710326Z 2025-08-14T21:51:33.1710497Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1710713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1710792Z return mod(**inputs) 2025-08-14T21:51:33.1711092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1711168Z outputs = self.mobilebert( 2025-08-14T21:51:33.1711479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1711558Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1711901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1711991Z layer_outputs = layer_module( 2025-08-14T21:51:33.1712291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.1712472Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.1712773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.1712894Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.1713205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.1713296Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.1713300Z 2025-08-14T21:51:33.1713417Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1713630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1713704Z return mod(**inputs) 2025-08-14T21:51:33.1714013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1714089Z outputs = self.mobilebert( 2025-08-14T21:51:33.1714393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1714472Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1714770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1714876Z layer_outputs = layer_module( 2025-08-14T21:51:33.1715181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1715275Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1715611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.1715701Z self_outputs = self.self( 2025-08-14T21:51:33.1716009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.1716086Z self.value(value_tensor) 2025-08-14T21:51:33.1716090Z 2025-08-14T21:51:33.1716200Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1716417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1716490Z return mod(**inputs) 2025-08-14T21:51:33.1716790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1716874Z outputs = self.mobilebert( 2025-08-14T21:51:33.1717174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1717280Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1717580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1717656Z layer_outputs = layer_module( 2025-08-14T21:51:33.1717966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.1718137Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.1718451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.1718570Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.1718887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.1718989Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.1718993Z 2025-08-14T21:51:33.1719101Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1719319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1719390Z return mod(**inputs) 2025-08-14T21:51:33.1719693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1719779Z outputs = self.mobilebert( 2025-08-14T21:51:33.1720078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1720156Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1720468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1720545Z layer_outputs = layer_module( 2025-08-14T21:51:33.1720853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.1721021Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.1721319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.1721442Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.1721779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.1721877Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.1722178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1722296Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1722300Z 2025-08-14T21:51:33.1722420Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1722631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1722701Z return mod(**inputs) 2025-08-14T21:51:33.1723009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1723085Z outputs = self.mobilebert( 2025-08-14T21:51:33.1723394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1723471Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1723774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1723860Z layer_outputs = layer_module( 2025-08-14T21:51:33.1724175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1724275Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1724575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.1724652Z self_outputs = self.self( 2025-08-14T21:51:33.1724957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.1725034Z self.query(query_tensor) 2025-08-14T21:51:33.1725038Z 2025-08-14T21:51:33.1725146Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1725403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1725548Z return mod(**inputs) 2025-08-14T21:51:33.1725867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1725943Z outputs = self.mobilebert( 2025-08-14T21:51:33.1726242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1726331Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1726633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1726722Z layer_outputs = layer_module( 2025-08-14T21:51:33.1727020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1727111Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1727421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.1727499Z self_outputs = self.self( 2025-08-14T21:51:33.1727798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.1727879Z self.key(key_tensor) 2025-08-14T21:51:33.1727883Z 2025-08-14T21:51:33.1727975Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.1728070Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.1728180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1728393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1728507Z return mod(**inputs) 2025-08-14T21:51:33.1728810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1728889Z outputs = self.mobilebert( 2025-08-14T21:51:33.1729219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1729299Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1729609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1729687Z layer_outputs = layer_module( 2025-08-14T21:51:33.1729983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1730083Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1730382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.1730521Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.1730817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.1730927Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1730931Z 2025-08-14T21:51:33.1731049Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1731259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1731332Z return mod(**inputs) 2025-08-14T21:51:33.1731639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1731718Z outputs = self.mobilebert( 2025-08-14T21:51:33.1732027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1732106Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1732426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1732517Z layer_outputs = layer_module( 2025-08-14T21:51:33.1732818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1732914Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1733212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.1733341Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.1733644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.1733779Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1734077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1734185Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1734189Z 2025-08-14T21:51:33.1734298Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1734527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1734598Z return mod(**inputs) 2025-08-14T21:51:33.1734894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1734979Z outputs = self.mobilebert( 2025-08-14T21:51:33.1735274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1735380Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1735684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1735780Z layer_outputs = layer_module( 2025-08-14T21:51:33.1736093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1736196Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1736493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1736622Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1736923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1737021Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1737025Z 2025-08-14T21:51:33.1737135Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1737347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1737430Z return mod(**inputs) 2025-08-14T21:51:33.1737965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1738061Z outputs = self.mobilebert( 2025-08-14T21:51:33.1738360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1738438Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1738745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1738824Z layer_outputs = layer_module( 2025-08-14T21:51:33.1739123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1739263Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1739564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1739695Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1739999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1740119Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1740123Z 2025-08-14T21:51:33.1740241Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1740456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1740537Z return mod(**inputs) 2025-08-14T21:51:33.1740843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1740923Z outputs = self.mobilebert( 2025-08-14T21:51:33.1741234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1741315Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1741618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1741700Z layer_outputs = layer_module( 2025-08-14T21:51:33.1742003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1742110Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1742446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1742579Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1742888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.1743008Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1743014Z 2025-08-14T21:51:33.1743129Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1743351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1743420Z return mod(**inputs) 2025-08-14T21:51:33.1743726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1743801Z outputs = self.mobilebert( 2025-08-14T21:51:33.1744111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1744189Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1744491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1744576Z layer_outputs = layer_module( 2025-08-14T21:51:33.1744896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1744999Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1745306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1745438Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1745746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.1745877Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1746191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1746300Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1746304Z 2025-08-14T21:51:33.1746418Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1746641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1746712Z return mod(**inputs) 2025-08-14T21:51:33.1747014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1747099Z outputs = self.mobilebert( 2025-08-14T21:51:33.1747402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1747483Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1747796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1747873Z layer_outputs = layer_module( 2025-08-14T21:51:33.1748183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1748283Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1748585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1748710Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1749011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1749131Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1749135Z 2025-08-14T21:51:33.1749244Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1749454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1749551Z return mod(**inputs) 2025-08-14T21:51:33.1749857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1749934Z outputs = self.mobilebert( 2025-08-14T21:51:33.1750243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1750320Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1750626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1750706Z layer_outputs = layer_module( 2025-08-14T21:51:33.1751006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1751112Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1751412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1751561Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1751865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1751985Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1751989Z 2025-08-14T21:51:33.1752105Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1752317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1752391Z return mod(**inputs) 2025-08-14T21:51:33.1752696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1752801Z outputs = self.mobilebert( 2025-08-14T21:51:33.1753113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1753196Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1753495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1753579Z layer_outputs = layer_module( 2025-08-14T21:51:33.1753878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1753985Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1754289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1754420Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1754731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.1754821Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1754827Z 2025-08-14T21:51:33.1754942Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1755156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1755226Z return mod(**inputs) 2025-08-14T21:51:33.1755535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1755610Z outputs = self.mobilebert( 2025-08-14T21:51:33.1755930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1756017Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1756317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1756419Z layer_outputs = layer_module( 2025-08-14T21:51:33.1756723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1756821Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1757131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1757261Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1757569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.1757698Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1757997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1758102Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1758106Z 2025-08-14T21:51:33.1758233Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1758447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1758523Z return mod(**inputs) 2025-08-14T21:51:33.1758821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1758905Z outputs = self.mobilebert( 2025-08-14T21:51:33.1759205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1759285Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1759608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1759686Z layer_outputs = layer_module( 2025-08-14T21:51:33.1759997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1760097Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1760398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1760522Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1760823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1760914Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1760925Z 2025-08-14T21:51:33.1761033Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1761258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1761335Z return mod(**inputs) 2025-08-14T21:51:33.1761635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1761709Z outputs = self.mobilebert( 2025-08-14T21:51:33.1762015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1762093Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1762402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1762498Z layer_outputs = layer_module( 2025-08-14T21:51:33.1762800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1762908Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1763208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1763345Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1763650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1763767Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1763771Z 2025-08-14T21:51:33.1763889Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1764110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1764182Z return mod(**inputs) 2025-08-14T21:51:33.1764489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1764567Z outputs = self.mobilebert( 2025-08-14T21:51:33.1764872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1764971Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1765275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1765362Z layer_outputs = layer_module( 2025-08-14T21:51:33.1765728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1765834Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1766146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1766276Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1766599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.1766693Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1766697Z 2025-08-14T21:51:33.1766808Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1767029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1767099Z return mod(**inputs) 2025-08-14T21:51:33.1767406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1767483Z outputs = self.mobilebert( 2025-08-14T21:51:33.1767785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1767872Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1768172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1768253Z layer_outputs = layer_module( 2025-08-14T21:51:33.1768563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1768664Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1768972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1769103Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1769400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.1769560Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1769864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1769989Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1769992Z 2025-08-14T21:51:33.1770102Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1770314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1770394Z return mod(**inputs) 2025-08-14T21:51:33.1770697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1770772Z outputs = self.mobilebert( 2025-08-14T21:51:33.1771081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1771162Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1771471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1771550Z layer_outputs = layer_module( 2025-08-14T21:51:33.1771873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.1772011Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.1772312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1772408Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1772412Z 2025-08-14T21:51:33.1772520Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1772732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1772811Z return mod(**inputs) 2025-08-14T21:51:33.1773129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1773216Z outputs = self.mobilebert( 2025-08-14T21:51:33.1773520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1773599Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1773906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1773983Z layer_outputs = layer_module( 2025-08-14T21:51:33.1774282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.1774415Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.1774712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1774837Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1774841Z 2025-08-14T21:51:33.1774951Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1775169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1775247Z return mod(**inputs) 2025-08-14T21:51:33.1775548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1775630Z outputs = self.mobilebert( 2025-08-14T21:51:33.1775930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1776035Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1776342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1776419Z layer_outputs = layer_module( 2025-08-14T21:51:33.1776722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1776919Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1777218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.1777323Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.1777328Z 2025-08-14T21:51:33.1777436Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1777647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1777726Z return mod(**inputs) 2025-08-14T21:51:33.1778026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1778109Z outputs = self.mobilebert( 2025-08-14T21:51:33.1778409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1778488Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1778825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1778907Z layer_outputs = layer_module( 2025-08-14T21:51:33.1779211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1779386Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1779694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.1779831Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.1780155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1780256Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1780262Z 2025-08-14T21:51:33.1780379Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1780609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1780686Z return mod(**inputs) 2025-08-14T21:51:33.1780989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1781063Z outputs = self.mobilebert( 2025-08-14T21:51:33.1781373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1781452Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1781752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1781839Z layer_outputs = layer_module( 2025-08-14T21:51:33.1782143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1782319Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1782619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.1782749Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.1783076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.1783166Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1783169Z 2025-08-14T21:51:33.1783290Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1783513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1783604Z return mod(**inputs) 2025-08-14T21:51:33.1783924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1783999Z outputs = self.mobilebert( 2025-08-14T21:51:33.1784313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1784392Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1784696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1784783Z layer_outputs = layer_module( 2025-08-14T21:51:33.1785089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1785257Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1785588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.1785722Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.1786035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.1786179Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1786486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1786597Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1786601Z 2025-08-14T21:51:33.1786715Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1786969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1787042Z return mod(**inputs) 2025-08-14T21:51:33.1787343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1787430Z outputs = self.mobilebert( 2025-08-14T21:51:33.1787728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1787807Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1788113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1788192Z layer_outputs = layer_module( 2025-08-14T21:51:33.1788501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.1788674Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.1788977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.1789103Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.1789404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.1789499Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.1789503Z 2025-08-14T21:51:33.1789612Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1789844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1789924Z return mod(**inputs) 2025-08-14T21:51:33.1790226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1790320Z outputs = self.mobilebert( 2025-08-14T21:51:33.1790626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1790704Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1791008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1791086Z layer_outputs = layer_module( 2025-08-14T21:51:33.1791382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1791482Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1791782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.1791868Z self_outputs = self.self( 2025-08-14T21:51:33.1792166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.1792263Z self.value(value_tensor) 2025-08-14T21:51:33.1792267Z 2025-08-14T21:51:33.1792386Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1792595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1792665Z return mod(**inputs) 2025-08-14T21:51:33.1792970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1793046Z outputs = self.mobilebert( 2025-08-14T21:51:33.1793352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1793430Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1793746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1793833Z layer_outputs = layer_module( 2025-08-14T21:51:33.1794143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.1794321Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.1794623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.1794737Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.1795048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.1795136Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.1795140Z 2025-08-14T21:51:33.1795256Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1795466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1795538Z return mod(**inputs) 2025-08-14T21:51:33.1795847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1795922Z outputs = self.mobilebert( 2025-08-14T21:51:33.1796224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1796310Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1796633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1796717Z layer_outputs = layer_module( 2025-08-14T21:51:33.1797015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.1797203Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.1797510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.1797628Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.1797932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.1798024Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.1798323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1798431Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1798435Z 2025-08-14T21:51:33.1798544Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1798768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1798849Z return mod(**inputs) 2025-08-14T21:51:33.1799168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1799252Z outputs = self.mobilebert( 2025-08-14T21:51:33.1799551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1799631Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1799937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1800014Z layer_outputs = layer_module( 2025-08-14T21:51:33.1800321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1800432Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1800736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.1800820Z self_outputs = self.self( 2025-08-14T21:51:33.1801118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.1801195Z self.query(query_tensor) 2025-08-14T21:51:33.1801199Z 2025-08-14T21:51:33.1801316Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1801526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1801606Z return mod(**inputs) 2025-08-14T21:51:33.1801905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1801982Z outputs = self.mobilebert( 2025-08-14T21:51:33.1802287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1802369Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1802673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1802750Z layer_outputs = layer_module( 2025-08-14T21:51:33.1803050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1803147Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1803479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.1803554Z self_outputs = self.self( 2025-08-14T21:51:33.1803860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.1803960Z self.key(key_tensor) 2025-08-14T21:51:33.1803963Z 2025-08-14T21:51:33.1805746Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.1805838Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.1805951Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1806174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1806246Z return mod(**inputs) 2025-08-14T21:51:33.1806546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1806631Z outputs = self.mobilebert( 2025-08-14T21:51:33.1806928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1807014Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1807316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1807395Z layer_outputs = layer_module( 2025-08-14T21:51:33.1807731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1807821Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1808128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.1808259Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.1808560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.1808659Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1808663Z 2025-08-14T21:51:33.1808791Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1809012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1809083Z return mod(**inputs) 2025-08-14T21:51:33.1809386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1809471Z outputs = self.mobilebert( 2025-08-14T21:51:33.1809772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1809850Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1810157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1810235Z layer_outputs = layer_module( 2025-08-14T21:51:33.1810543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1810633Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1810935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.1811071Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.1811370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.1811508Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1811807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1811924Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1811929Z 2025-08-14T21:51:33.1812044Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1812258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1812347Z return mod(**inputs) 2025-08-14T21:51:33.1812655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1812731Z outputs = self.mobilebert( 2025-08-14T21:51:33.1813033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1813121Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1813410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1813495Z layer_outputs = layer_module( 2025-08-14T21:51:33.1813785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1813890Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1814183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1814342Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1814646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1814734Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1814738Z 2025-08-14T21:51:33.1814842Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1815055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1815125Z return mod(**inputs) 2025-08-14T21:51:33.1815428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1815520Z outputs = self.mobilebert( 2025-08-14T21:51:33.1815811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1815897Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1816187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1816272Z layer_outputs = layer_module( 2025-08-14T21:51:33.1816572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1816673Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1816981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1817100Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1817401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1817531Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1817535Z 2025-08-14T21:51:33.1817645Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1817865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1817936Z return mod(**inputs) 2025-08-14T21:51:33.1818233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1818338Z outputs = self.mobilebert( 2025-08-14T21:51:33.1818643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1818729Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1819035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1819173Z layer_outputs = layer_module( 2025-08-14T21:51:33.1819483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1819583Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1819883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1820023Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1820323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.1820422Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1820425Z 2025-08-14T21:51:33.1820543Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1820757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1820834Z return mod(**inputs) 2025-08-14T21:51:33.1821142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1821226Z outputs = self.mobilebert( 2025-08-14T21:51:33.1821527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1821607Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1821913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1821992Z layer_outputs = layer_module( 2025-08-14T21:51:33.1822307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1822417Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1822722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1822860Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1823160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.1823289Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1823601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1823697Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1823701Z 2025-08-14T21:51:33.1823814Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1824026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1824099Z return mod(**inputs) 2025-08-14T21:51:33.1824409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1824483Z outputs = self.mobilebert( 2025-08-14T21:51:33.1824785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1824867Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1825160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1825261Z layer_outputs = layer_module( 2025-08-14T21:51:33.1825554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1825652Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1825969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1826085Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1826387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1826475Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1826479Z 2025-08-14T21:51:33.1826588Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1826810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1826880Z return mod(**inputs) 2025-08-14T21:51:33.1827187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1827265Z outputs = self.mobilebert( 2025-08-14T21:51:33.1827568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1827675Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1827975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1828052Z layer_outputs = layer_module( 2025-08-14T21:51:33.1828356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1828453Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1828764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1828879Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1829191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1829324Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1829329Z 2025-08-14T21:51:33.1829437Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1829656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1829727Z return mod(**inputs) 2025-08-14T21:51:33.1830028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1830113Z outputs = self.mobilebert( 2025-08-14T21:51:33.1830413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1830491Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1830799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1830877Z layer_outputs = layer_module( 2025-08-14T21:51:33.1831182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1831281Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1831576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1831714Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1832060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.1832156Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1832160Z 2025-08-14T21:51:33.1832270Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1832486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1832584Z return mod(**inputs) 2025-08-14T21:51:33.1832894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1832970Z outputs = self.mobilebert( 2025-08-14T21:51:33.1833287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1833365Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1833681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1833759Z layer_outputs = layer_module( 2025-08-14T21:51:33.1834070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1834178Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1834510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1834652Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1834951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.1835079Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1835386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1835484Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1835488Z 2025-08-14T21:51:33.1835603Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1835834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1835908Z return mod(**inputs) 2025-08-14T21:51:33.1836223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1836299Z outputs = self.mobilebert( 2025-08-14T21:51:33.1836601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1836690Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1836996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1837082Z layer_outputs = layer_module( 2025-08-14T21:51:33.1837381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1837481Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1837989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1838117Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1838421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1838521Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1838525Z 2025-08-14T21:51:33.1838635Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1838856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1838990Z return mod(**inputs) 2025-08-14T21:51:33.1839292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1839377Z outputs = self.mobilebert( 2025-08-14T21:51:33.1839677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1839794Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1840103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1840179Z layer_outputs = layer_module( 2025-08-14T21:51:33.1840492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1840590Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1840902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1841026Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1841335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1841463Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1841497Z 2025-08-14T21:51:33.1841611Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1841835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1841917Z return mod(**inputs) 2025-08-14T21:51:33.1842215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1842301Z outputs = self.mobilebert( 2025-08-14T21:51:33.1842600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1842679Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1843009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1843090Z layer_outputs = layer_module( 2025-08-14T21:51:33.1843391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1843501Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1843802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1843941Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1844240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.1844329Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1844333Z 2025-08-14T21:51:33.1844453Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1844676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1844755Z return mod(**inputs) 2025-08-14T21:51:33.1845055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1845131Z outputs = self.mobilebert( 2025-08-14T21:51:33.1845487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1845575Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1845888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1845990Z layer_outputs = layer_module( 2025-08-14T21:51:33.1846293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1846428Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1846754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1846884Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1847193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.1847320Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1847630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1847730Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1847735Z 2025-08-14T21:51:33.1847846Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1848075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1848147Z return mod(**inputs) 2025-08-14T21:51:33.1848474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1848553Z outputs = self.mobilebert( 2025-08-14T21:51:33.1848855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1848941Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1849241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1849320Z layer_outputs = layer_module( 2025-08-14T21:51:33.1849627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.1849773Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.1850086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1850176Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1850180Z 2025-08-14T21:51:33.1850289Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1850511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1850582Z return mod(**inputs) 2025-08-14T21:51:33.1850893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1850970Z outputs = self.mobilebert( 2025-08-14T21:51:33.1851271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1851357Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1851661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1851742Z layer_outputs = layer_module( 2025-08-14T21:51:33.1852054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.1852180Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.1852491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1852609Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1852635Z 2025-08-14T21:51:33.1852747Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1852973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1853046Z return mod(**inputs) 2025-08-14T21:51:33.1853353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1853450Z outputs = self.mobilebert( 2025-08-14T21:51:33.1853756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1853843Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1854147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1854226Z layer_outputs = layer_module( 2025-08-14T21:51:33.1854538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1854711Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1855022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.1855126Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.1855132Z 2025-08-14T21:51:33.1855262Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1855486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1855557Z return mod(**inputs) 2025-08-14T21:51:33.1855873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1855949Z outputs = self.mobilebert( 2025-08-14T21:51:33.1856250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1856337Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1856668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1856748Z layer_outputs = layer_module( 2025-08-14T21:51:33.1857062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1857228Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1857539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.1857668Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.1857972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1858078Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1858082Z 2025-08-14T21:51:33.1858193Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1858418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1858493Z return mod(**inputs) 2025-08-14T21:51:33.1858796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1858880Z outputs = self.mobilebert( 2025-08-14T21:51:33.1859183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1859260Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1859570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1859673Z layer_outputs = layer_module( 2025-08-14T21:51:33.1859983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1860151Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1860475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.1860615Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.1860918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.1861014Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1861018Z 2025-08-14T21:51:33.1861127Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1861343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1861423Z return mod(**inputs) 2025-08-14T21:51:33.1861724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1861807Z outputs = self.mobilebert( 2025-08-14T21:51:33.1862125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1862206Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1862519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1862596Z layer_outputs = layer_module( 2025-08-14T21:51:33.1862898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1863076Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1863391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.1863529Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.1863833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.1863963Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1864272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1864369Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1864373Z 2025-08-14T21:51:33.1864488Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1864700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1864770Z return mod(**inputs) 2025-08-14T21:51:33.1865075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1865150Z outputs = self.mobilebert( 2025-08-14T21:51:33.1865457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1865543Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1865843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1865927Z layer_outputs = layer_module( 2025-08-14T21:51:33.1866222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.1866417Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.1866723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.1866844Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.1867187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.1867275Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.1867279Z 2025-08-14T21:51:33.1867389Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1867611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1867681Z return mod(**inputs) 2025-08-14T21:51:33.1867993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1868070Z outputs = self.mobilebert( 2025-08-14T21:51:33.1868375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1868460Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1868764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1868856Z layer_outputs = layer_module( 2025-08-14T21:51:33.1869183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1869274Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1869581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.1869656Z self_outputs = self.self( 2025-08-14T21:51:33.1869958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.1870043Z self.value(value_tensor) 2025-08-14T21:51:33.1870046Z 2025-08-14T21:51:33.1870174Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1870399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1870468Z return mod(**inputs) 2025-08-14T21:51:33.1870769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1870852Z outputs = self.mobilebert( 2025-08-14T21:51:33.1871153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1871232Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1871537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1871616Z layer_outputs = layer_module( 2025-08-14T21:51:33.1871922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.1872094Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.1872394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.1872519Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.1872816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.1872910Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.1872914Z 2025-08-14T21:51:33.1873052Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1873265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1873343Z return mod(**inputs) 2025-08-14T21:51:33.1873641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1873733Z outputs = self.mobilebert( 2025-08-14T21:51:33.1874048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1874126Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1874437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1874512Z layer_outputs = layer_module( 2025-08-14T21:51:33.1874815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.1874991Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.1875296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.1875418Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.1875742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.1875839Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.1876149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1876246Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1876250Z 2025-08-14T21:51:33.1876359Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1876579Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1876649Z return mod(**inputs) 2025-08-14T21:51:33.1876974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1877053Z outputs = self.mobilebert( 2025-08-14T21:51:33.1877358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1877445Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1877746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1877833Z layer_outputs = layer_module( 2025-08-14T21:51:33.1878135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1878230Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1878538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.1878616Z self_outputs = self.self( 2025-08-14T21:51:33.1878917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.1879003Z self.query(query_tensor) 2025-08-14T21:51:33.1879009Z 2025-08-14T21:51:33.1879118Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1879339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1879409Z return mod(**inputs) 2025-08-14T21:51:33.1879706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1879790Z outputs = self.mobilebert( 2025-08-14T21:51:33.1880115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1880200Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1880499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1880597Z layer_outputs = layer_module( 2025-08-14T21:51:33.1880911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1881001Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1881304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.1881387Z self_outputs = self.self( 2025-08-14T21:51:33.1881689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.1881769Z self.key(key_tensor) 2025-08-14T21:51:33.1881772Z 2025-08-14T21:51:33.1881864Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.1881951Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.1882069Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1882283Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1882381Z return mod(**inputs) 2025-08-14T21:51:33.1882690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1882763Z outputs = self.mobilebert( 2025-08-14T21:51:33.1883072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1883150Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1883448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1883533Z layer_outputs = layer_module( 2025-08-14T21:51:33.1883845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1883944Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1884244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.1884374Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.1884677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.1884766Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1884770Z 2025-08-14T21:51:33.1884881Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1885117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1885186Z return mod(**inputs) 2025-08-14T21:51:33.1885565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1885650Z outputs = self.mobilebert( 2025-08-14T21:51:33.1885956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1886045Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1886347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1886434Z layer_outputs = layer_module( 2025-08-14T21:51:33.1886735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1886865Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1887171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.1887303Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.1887629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.1887774Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1888075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1888182Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1888186Z 2025-08-14T21:51:33.1888297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1888512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1888591Z return mod(**inputs) 2025-08-14T21:51:33.1888905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1888987Z outputs = self.mobilebert( 2025-08-14T21:51:33.1889306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1889387Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1889698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1889776Z layer_outputs = layer_module( 2025-08-14T21:51:33.1890074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1890185Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1890487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1890634Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1890936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1891029Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1891033Z 2025-08-14T21:51:33.1891149Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1891360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1891437Z return mod(**inputs) 2025-08-14T21:51:33.1891738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1891814Z outputs = self.mobilebert( 2025-08-14T21:51:33.1892121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1892198Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1892503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1892588Z layer_outputs = layer_module( 2025-08-14T21:51:33.1892892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1893002Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1893301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1893419Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1893759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1893878Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1893882Z 2025-08-14T21:51:33.1894000Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1894237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1894310Z return mod(**inputs) 2025-08-14T21:51:33.1894616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1894693Z outputs = self.mobilebert( 2025-08-14T21:51:33.1894997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1895083Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1895387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1895470Z layer_outputs = layer_module( 2025-08-14T21:51:33.1895772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1895875Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1896205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1896343Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1896651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.1896742Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1896745Z 2025-08-14T21:51:33.1896854Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1897075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1897144Z return mod(**inputs) 2025-08-14T21:51:33.1897471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1897551Z outputs = self.mobilebert( 2025-08-14T21:51:33.1897849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1897937Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1898235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1898311Z layer_outputs = layer_module( 2025-08-14T21:51:33.1898618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1898718Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1899019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1899151Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1899451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.1899589Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1899887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1899992Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1899996Z 2025-08-14T21:51:33.1900106Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1900342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1900418Z return mod(**inputs) 2025-08-14T21:51:33.1900719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1900797Z outputs = self.mobilebert( 2025-08-14T21:51:33.1901128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1901206Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1901516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1901593Z layer_outputs = layer_module( 2025-08-14T21:51:33.1901890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1901997Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1902296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1902422Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1902723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1902832Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1902836Z 2025-08-14T21:51:33.1902956Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1903180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1903251Z return mod(**inputs) 2025-08-14T21:51:33.1903559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1903638Z outputs = self.mobilebert( 2025-08-14T21:51:33.1903945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1904023Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1904346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1904436Z layer_outputs = layer_module( 2025-08-14T21:51:33.1904739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1904846Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1905147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1905263Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1905573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1905691Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1905695Z 2025-08-14T21:51:33.1905807Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1906030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1906105Z return mod(**inputs) 2025-08-14T21:51:33.1906412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1906489Z outputs = self.mobilebert( 2025-08-14T21:51:33.1906790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1906876Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1907179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1907281Z layer_outputs = layer_module( 2025-08-14T21:51:33.1907584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1907712Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1908018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1908149Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1908446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.1908543Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1908547Z 2025-08-14T21:51:33.1908654Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1908872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1908942Z return mod(**inputs) 2025-08-14T21:51:33.1909242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1909327Z outputs = self.mobilebert( 2025-08-14T21:51:33.1909647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1909735Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1910038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1910114Z layer_outputs = layer_module( 2025-08-14T21:51:33.1910420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1910521Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1910818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1910981Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1911282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.1911419Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1911723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1911822Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1911826Z 2025-08-14T21:51:33.1911943Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1912158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1912236Z return mod(**inputs) 2025-08-14T21:51:33.1912534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1912612Z outputs = self.mobilebert( 2025-08-14T21:51:33.1912919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1912998Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1913301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1913377Z layer_outputs = layer_module( 2025-08-14T21:51:33.1913676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1913805Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1914102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1914218Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1914527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1914635Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1914639Z 2025-08-14T21:51:33.1914757Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1914970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1915042Z return mod(**inputs) 2025-08-14T21:51:33.1915347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1915425Z outputs = self.mobilebert( 2025-08-14T21:51:33.1915732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1915808Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1916107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1916190Z layer_outputs = layer_module( 2025-08-14T21:51:33.1916503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1916604Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1916916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1917032Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1917341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1917456Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1917460Z 2025-08-14T21:51:33.1917588Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1917808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1917879Z return mod(**inputs) 2025-08-14T21:51:33.1918183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1918269Z outputs = self.mobilebert( 2025-08-14T21:51:33.1918558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1918639Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1918927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1919005Z layer_outputs = layer_module( 2025-08-14T21:51:33.1919299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1919394Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1919694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1919823Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1920112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.1920208Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1920211Z 2025-08-14T21:51:33.1920317Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1920552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1920621Z return mod(**inputs) 2025-08-14T21:51:33.1920914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1921016Z outputs = self.mobilebert( 2025-08-14T21:51:33.1921314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1921391Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1921692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1921767Z layer_outputs = layer_module( 2025-08-14T21:51:33.1922066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1922164Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1922457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1922594Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1922887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.1923043Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1923344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1923441Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1923445Z 2025-08-14T21:51:33.1923562Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1923775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1923848Z return mod(**inputs) 2025-08-14T21:51:33.1924155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1924247Z outputs = self.mobilebert( 2025-08-14T21:51:33.1924560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1924640Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1924939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1925023Z layer_outputs = layer_module( 2025-08-14T21:51:33.1925322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.1925530Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.1925850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1925946Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1925951Z 2025-08-14T21:51:33.1926073Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1926287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1926370Z return mod(**inputs) 2025-08-14T21:51:33.1926683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1926757Z outputs = self.mobilebert( 2025-08-14T21:51:33.1927061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1927137Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1927502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1927582Z layer_outputs = layer_module( 2025-08-14T21:51:33.1927873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.1928022Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.1928316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1928429Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1928433Z 2025-08-14T21:51:33.1928549Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1928758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1928836Z return mod(**inputs) 2025-08-14T21:51:33.1929126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1929198Z outputs = self.mobilebert( 2025-08-14T21:51:33.1929497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1929573Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1929882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1929968Z layer_outputs = layer_module( 2025-08-14T21:51:33.1930259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1930434Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1930724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.1930826Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.1930830Z 2025-08-14T21:51:33.1930963Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1931173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1931253Z return mod(**inputs) 2025-08-14T21:51:33.1931546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1931620Z outputs = self.mobilebert( 2025-08-14T21:51:33.1931918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1931993Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1932285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1932370Z layer_outputs = layer_module( 2025-08-14T21:51:33.1932661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1932832Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1933129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.1933257Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.1933559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1933654Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1933658Z 2025-08-14T21:51:33.1933772Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1934008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1934076Z return mod(**inputs) 2025-08-14T21:51:33.1934374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1934465Z outputs = self.mobilebert( 2025-08-14T21:51:33.1934757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1934840Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1935133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1935214Z layer_outputs = layer_module( 2025-08-14T21:51:33.1935506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1935669Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1935966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.1936096Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.1936412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.1936502Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1936505Z 2025-08-14T21:51:33.1936613Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1936828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1936896Z return mod(**inputs) 2025-08-14T21:51:33.1937196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1937272Z outputs = self.mobilebert( 2025-08-14T21:51:33.1937563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1937902Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1938207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1938283Z layer_outputs = layer_module( 2025-08-14T21:51:33.1938582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.1938744Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.1939041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.1939174Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.1939473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.1939612Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1939915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1940021Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1940025Z 2025-08-14T21:51:33.1940137Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1940349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1940430Z return mod(**inputs) 2025-08-14T21:51:33.1940732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1940861Z outputs = self.mobilebert( 2025-08-14T21:51:33.1941173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1941253Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1941588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1941668Z layer_outputs = layer_module( 2025-08-14T21:51:33.1941967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.1942148Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.1942448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.1942575Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.1942873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.1942964Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.1942968Z 2025-08-14T21:51:33.1943086Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1943328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1943410Z return mod(**inputs) 2025-08-14T21:51:33.1943711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1943788Z outputs = self.mobilebert( 2025-08-14T21:51:33.1944097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1944176Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1944476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1944563Z layer_outputs = layer_module( 2025-08-14T21:51:33.1944880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1944983Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1945281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.1945360Z self_outputs = self.self( 2025-08-14T21:51:33.1945666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.1945743Z self.value(value_tensor) 2025-08-14T21:51:33.1945746Z 2025-08-14T21:51:33.1945864Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1946072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1946144Z return mod(**inputs) 2025-08-14T21:51:33.1946449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1946525Z outputs = self.mobilebert( 2025-08-14T21:51:33.1946822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1946908Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1947206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1947292Z layer_outputs = layer_module( 2025-08-14T21:51:33.1947590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.1947784Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.1948095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.1948211Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.1948534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.1948630Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.1948634Z 2025-08-14T21:51:33.1948742Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1948961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1949033Z return mod(**inputs) 2025-08-14T21:51:33.1949334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1949419Z outputs = self.mobilebert( 2025-08-14T21:51:33.1949721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1949806Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1950121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1950200Z layer_outputs = layer_module( 2025-08-14T21:51:33.1950508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.1950674Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.1950983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.1951102Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.1951402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.1951520Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.1951827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1951924Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1951937Z 2025-08-14T21:51:33.1952048Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1952260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1952336Z return mod(**inputs) 2025-08-14T21:51:33.1952630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1952710Z outputs = self.mobilebert( 2025-08-14T21:51:33.1953013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1953093Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1953397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1953478Z layer_outputs = layer_module( 2025-08-14T21:51:33.1953777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1953876Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1954172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.1954246Z self_outputs = self.self( 2025-08-14T21:51:33.1954577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.1954655Z self.query(query_tensor) 2025-08-14T21:51:33.1954659Z 2025-08-14T21:51:33.1954777Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1954989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1955080Z return mod(**inputs) 2025-08-14T21:51:33.1955398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1955475Z outputs = self.mobilebert( 2025-08-14T21:51:33.1955791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1955869Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1956179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1956264Z layer_outputs = layer_module( 2025-08-14T21:51:33.1956573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1956663Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1957004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.1957080Z self_outputs = self.self( 2025-08-14T21:51:33.1957383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.1957456Z self.key(key_tensor) 2025-08-14T21:51:33.1957460Z 2025-08-14T21:51:33.1957549Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.1957642Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.1957753Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1957964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1958044Z return mod(**inputs) 2025-08-14T21:51:33.1958370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1958477Z outputs = self.mobilebert( 2025-08-14T21:51:33.1958780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1958859Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1959169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1959246Z layer_outputs = layer_module( 2025-08-14T21:51:33.1959554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1959646Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1959949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.1960093Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.1960397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.1960489Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1960492Z 2025-08-14T21:51:33.1960611Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1960824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1960903Z return mod(**inputs) 2025-08-14T21:51:33.1961204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1961301Z outputs = self.mobilebert( 2025-08-14T21:51:33.1961609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1961686Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1962015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1962092Z layer_outputs = layer_module( 2025-08-14T21:51:33.1962390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.1962486Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.1962784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.1962914Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.1963221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.1963357Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1963671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1963788Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1963798Z 2025-08-14T21:51:33.1963910Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1964143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1964215Z return mod(**inputs) 2025-08-14T21:51:33.1964523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1964600Z outputs = self.mobilebert( 2025-08-14T21:51:33.1964899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1965008Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1965314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1965396Z layer_outputs = layer_module( 2025-08-14T21:51:33.1965796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1965903Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1966210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1966327Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1966628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1966730Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1966734Z 2025-08-14T21:51:33.1966848Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1967067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1967141Z return mod(**inputs) 2025-08-14T21:51:33.1967438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1967522Z outputs = self.mobilebert( 2025-08-14T21:51:33.1967816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1967895Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1968236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1968314Z layer_outputs = layer_module( 2025-08-14T21:51:33.1968628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1968750Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1969053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1969180Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1969482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1969608Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1969612Z 2025-08-14T21:51:33.1969721Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1969944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1970023Z return mod(**inputs) 2025-08-14T21:51:33.1970326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1970404Z outputs = self.mobilebert( 2025-08-14T21:51:33.1970730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1970810Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1971118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1971194Z layer_outputs = layer_module( 2025-08-14T21:51:33.1971490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1971601Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1971898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1972060Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1972361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.1972454Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1972459Z 2025-08-14T21:51:33.1972577Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1972798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1972878Z return mod(**inputs) 2025-08-14T21:51:33.1973177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1973254Z outputs = self.mobilebert( 2025-08-14T21:51:33.1973559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1973637Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1973937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1974025Z layer_outputs = layer_module( 2025-08-14T21:51:33.1974319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1974427Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1974723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1974880Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1975194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.1975326Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1975639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1975763Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1975766Z 2025-08-14T21:51:33.1975880Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1976108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1976180Z return mod(**inputs) 2025-08-14T21:51:33.1976486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1976575Z outputs = self.mobilebert( 2025-08-14T21:51:33.1976880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1976969Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1977278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1977383Z layer_outputs = layer_module( 2025-08-14T21:51:33.1977696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1977799Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1978110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1978229Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1978537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1978634Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1978638Z 2025-08-14T21:51:33.1978771Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1978987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1979069Z return mod(**inputs) 2025-08-14T21:51:33.1979371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1979454Z outputs = self.mobilebert( 2025-08-14T21:51:33.1979754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1979834Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1980146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1980224Z layer_outputs = layer_module( 2025-08-14T21:51:33.1980532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1980635Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1980935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1981066Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1981371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1981491Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1981504Z 2025-08-14T21:51:33.1981640Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1981851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1981930Z return mod(**inputs) 2025-08-14T21:51:33.1982230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1982328Z outputs = self.mobilebert( 2025-08-14T21:51:33.1982641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1982720Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1983025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1983102Z layer_outputs = layer_module( 2025-08-14T21:51:33.1983404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1983514Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1983812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1983944Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1984281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.1984373Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1984377Z 2025-08-14T21:51:33.1984496Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1984709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1984779Z return mod(**inputs) 2025-08-14T21:51:33.1985087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1985164Z outputs = self.mobilebert( 2025-08-14T21:51:33.1985472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1985571Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1985874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1985960Z layer_outputs = layer_module( 2025-08-14T21:51:33.1986259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1986357Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1986663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1986798Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1987105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.1987232Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1987530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1987640Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1987644Z 2025-08-14T21:51:33.1987755Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1987985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1988056Z return mod(**inputs) 2025-08-14T21:51:33.1988353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1988461Z outputs = self.mobilebert( 2025-08-14T21:51:33.1988762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1988853Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1989158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1989255Z layer_outputs = layer_module( 2025-08-14T21:51:33.1989566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1989665Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1989965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1990090Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1990394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.1990492Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.1990496Z 2025-08-14T21:51:33.1990609Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1990826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1990927Z return mod(**inputs) 2025-08-14T21:51:33.1991231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1991318Z outputs = self.mobilebert( 2025-08-14T21:51:33.1991619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1991697Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1992010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1992091Z layer_outputs = layer_module( 2025-08-14T21:51:33.1992421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1992541Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1992848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.1992980Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.1993281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.1993399Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.1993403Z 2025-08-14T21:51:33.1993523Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1993739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1993821Z return mod(**inputs) 2025-08-14T21:51:33.1994123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1994200Z outputs = self.mobilebert( 2025-08-14T21:51:33.1994514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1994595Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1994898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1994985Z layer_outputs = layer_module( 2025-08-14T21:51:33.1995285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1995418Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1995718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1995852Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1996183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.1996284Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.1996288Z 2025-08-14T21:51:33.1996405Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1996615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1996685Z return mod(**inputs) 2025-08-14T21:51:33.1996990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.1997066Z outputs = self.mobilebert( 2025-08-14T21:51:33.1997364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.1997455Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.1997758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.1997862Z layer_outputs = layer_module( 2025-08-14T21:51:33.1998162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.1998260Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.1998571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.1998701Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.1999014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.1999165Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.1999468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.1999578Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.1999582Z 2025-08-14T21:51:33.1999691Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.1999914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.1999986Z return mod(**inputs) 2025-08-14T21:51:33.2000284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2000371Z outputs = self.mobilebert( 2025-08-14T21:51:33.2000676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2000754Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2001066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2001146Z layer_outputs = layer_module( 2025-08-14T21:51:33.2001453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2001580Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2001882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2001977Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2002000Z 2025-08-14T21:51:33.2002109Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2002330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2002400Z return mod(**inputs) 2025-08-14T21:51:33.2002700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2002801Z outputs = self.mobilebert( 2025-08-14T21:51:33.2003101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2003179Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2003487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2003563Z layer_outputs = layer_module( 2025-08-14T21:51:33.2003873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2003998Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2004299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2004426Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2004430Z 2025-08-14T21:51:33.2004557Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2004780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2004849Z return mod(**inputs) 2025-08-14T21:51:33.2005146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2005229Z outputs = self.mobilebert( 2025-08-14T21:51:33.2005630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2005717Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2006058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2006138Z layer_outputs = layer_module( 2025-08-14T21:51:33.2006460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2006630Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2006931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.2007042Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.2007046Z 2025-08-14T21:51:33.2007157Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2007379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2007451Z return mod(**inputs) 2025-08-14T21:51:33.2007750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2007836Z outputs = self.mobilebert( 2025-08-14T21:51:33.2008133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2008212Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2008519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2008595Z layer_outputs = layer_module( 2025-08-14T21:51:33.2008902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2009093Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2009390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.2009529Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.2009866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2009971Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2009975Z 2025-08-14T21:51:33.2010087Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2010297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2010378Z return mod(**inputs) 2025-08-14T21:51:33.2010674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2010751Z outputs = self.mobilebert( 2025-08-14T21:51:33.2011058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2011137Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2011442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2011537Z layer_outputs = layer_module( 2025-08-14T21:51:33.2011837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2012011Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2012306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2012445Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2012741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.2012852Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2012859Z 2025-08-14T21:51:33.2012976Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2013188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2013264Z return mod(**inputs) 2025-08-14T21:51:33.2013560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2013634Z outputs = self.mobilebert( 2025-08-14T21:51:33.2013939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2014020Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2014315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2014398Z layer_outputs = layer_module( 2025-08-14T21:51:33.2014697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2014873Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2015169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2015298Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2015601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.2015750Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2016058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2016156Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2016160Z 2025-08-14T21:51:33.2016288Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2016512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2016583Z return mod(**inputs) 2025-08-14T21:51:33.2016879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2016962Z outputs = self.mobilebert( 2025-08-14T21:51:33.2017262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2017348Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2017645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2017723Z layer_outputs = layer_module( 2025-08-14T21:51:33.2018033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2018224Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2018533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2018651Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2018959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2019053Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2019059Z 2025-08-14T21:51:33.2019165Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2019378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2019472Z return mod(**inputs) 2025-08-14T21:51:33.2019768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2019851Z outputs = self.mobilebert( 2025-08-14T21:51:33.2020146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2020221Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2051667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2051953Z layer_outputs = layer_module( 2025-08-14T21:51:33.2052374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2052479Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2052816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2052907Z self_outputs = self.self( 2025-08-14T21:51:33.2053217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.2053306Z self.value(value_tensor) 2025-08-14T21:51:33.2053314Z 2025-08-14T21:51:33.2053440Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2053668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2053752Z return mod(**inputs) 2025-08-14T21:51:33.2054059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2054373Z outputs = self.mobilebert( 2025-08-14T21:51:33.2054690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2054776Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2055164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2055246Z layer_outputs = layer_module( 2025-08-14T21:51:33.2055555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2055737Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2056040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.2056177Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.2056480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2056572Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2056586Z 2025-08-14T21:51:33.2056703Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2056965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2057051Z return mod(**inputs) 2025-08-14T21:51:33.2057361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2057441Z outputs = self.mobilebert( 2025-08-14T21:51:33.2057752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2057838Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2058148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2058268Z layer_outputs = layer_module( 2025-08-14T21:51:33.2058569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2058752Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2059051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2059179Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2059477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.2059575Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.2059880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2059987Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2059992Z 2025-08-14T21:51:33.2060105Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2060335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2060407Z return mod(**inputs) 2025-08-14T21:51:33.2060713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2060791Z outputs = self.mobilebert( 2025-08-14T21:51:33.2061086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2061195Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2061505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2061592Z layer_outputs = layer_module( 2025-08-14T21:51:33.2061899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2062012Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2062323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2062400Z self_outputs = self.self( 2025-08-14T21:51:33.2062699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.2062782Z self.query(query_tensor) 2025-08-14T21:51:33.2062786Z 2025-08-14T21:51:33.2062897Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2063121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2063197Z return mod(**inputs) 2025-08-14T21:51:33.2063507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2063587Z outputs = self.mobilebert( 2025-08-14T21:51:33.2063915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2063998Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2064296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2064381Z layer_outputs = layer_module( 2025-08-14T21:51:33.2064681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2064782Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2065086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2065177Z self_outputs = self.self( 2025-08-14T21:51:33.2065488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.2065563Z self.key(key_tensor) 2025-08-14T21:51:33.2065567Z 2025-08-14T21:51:33.2065664Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2065749Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2065860Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2066077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2066149Z return mod(**inputs) 2025-08-14T21:51:33.2066448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2066531Z outputs = self.mobilebert( 2025-08-14T21:51:33.2066834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2066924Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2067223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2067299Z layer_outputs = layer_module( 2025-08-14T21:51:33.2067605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2067694Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2067997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2068150Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2068449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.2068547Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2068570Z 2025-08-14T21:51:33.2068682Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2068908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2068987Z return mod(**inputs) 2025-08-14T21:51:33.2069285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2069367Z outputs = self.mobilebert( 2025-08-14T21:51:33.2069665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2069745Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2070050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2070127Z layer_outputs = layer_module( 2025-08-14T21:51:33.2070432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2070539Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2070841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2070980Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2071281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.2071418Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2071730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2071830Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2071852Z 2025-08-14T21:51:33.2071971Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2072187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2072259Z return mod(**inputs) 2025-08-14T21:51:33.2072565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2072641Z outputs = self.mobilebert( 2025-08-14T21:51:33.2072951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2073030Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2073330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2073413Z layer_outputs = layer_module( 2025-08-14T21:51:33.2073718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2073821Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2074132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2074253Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2074562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2074653Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2074674Z 2025-08-14T21:51:33.2074785Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2075015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2075087Z return mod(**inputs) 2025-08-14T21:51:33.2075397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2075493Z outputs = self.mobilebert( 2025-08-14T21:51:33.2075797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2075885Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2076187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2076265Z layer_outputs = layer_module( 2025-08-14T21:51:33.2076575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2076678Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2076986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2077103Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2077424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2077557Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2077561Z 2025-08-14T21:51:33.2077681Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2077905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2077975Z return mod(**inputs) 2025-08-14T21:51:33.2078266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2078351Z outputs = self.mobilebert( 2025-08-14T21:51:33.2078658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2078736Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2079041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2079116Z layer_outputs = layer_module( 2025-08-14T21:51:33.2079421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2079518Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2079818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2079965Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2080267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2080368Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2080372Z 2025-08-14T21:51:33.2080485Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2080700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2080781Z return mod(**inputs) 2025-08-14T21:51:33.2081084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2081160Z outputs = self.mobilebert( 2025-08-14T21:51:33.2081471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2081634Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2081948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2082025Z layer_outputs = layer_module( 2025-08-14T21:51:33.2082336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2082476Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2082781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2082921Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2083222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2083350Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2083661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2083758Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2083764Z 2025-08-14T21:51:33.2083879Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2084094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2084182Z return mod(**inputs) 2025-08-14T21:51:33.2084491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2084567Z outputs = self.mobilebert( 2025-08-14T21:51:33.2084868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2084955Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2085256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2085338Z layer_outputs = layer_module( 2025-08-14T21:51:33.2085761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2085872Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2086179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2086300Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2086606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2086695Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2086699Z 2025-08-14T21:51:33.2086810Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2087032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2087104Z return mod(**inputs) 2025-08-14T21:51:33.2087400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2087485Z outputs = self.mobilebert( 2025-08-14T21:51:33.2087792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2087879Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2088180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2088258Z layer_outputs = layer_module( 2025-08-14T21:51:33.2088560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2088679Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2088990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2089106Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2089431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2089560Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2089564Z 2025-08-14T21:51:33.2089676Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2089888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2089966Z return mod(**inputs) 2025-08-14T21:51:33.2090266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2090352Z outputs = self.mobilebert( 2025-08-14T21:51:33.2090653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2090732Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2091066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2091144Z layer_outputs = layer_module( 2025-08-14T21:51:33.2091455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2091554Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2091851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2091992Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2092294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2092401Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2092413Z 2025-08-14T21:51:33.2092527Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2092744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2092822Z return mod(**inputs) 2025-08-14T21:51:33.2093120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2093195Z outputs = self.mobilebert( 2025-08-14T21:51:33.2093502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2093581Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2093892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2093966Z layer_outputs = layer_module( 2025-08-14T21:51:33.2094258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2094364Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2094667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2094797Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2095106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2095235Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2095561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2095659Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2095663Z 2025-08-14T21:51:33.2095776Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2096023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2096095Z return mod(**inputs) 2025-08-14T21:51:33.2096399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2096476Z outputs = self.mobilebert( 2025-08-14T21:51:33.2096771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2096864Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2097167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2097249Z layer_outputs = layer_module( 2025-08-14T21:51:33.2097550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2097648Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2097970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2098092Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2098399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2098489Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2098493Z 2025-08-14T21:51:33.2098608Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2098828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2098901Z return mod(**inputs) 2025-08-14T21:51:33.2099215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2099302Z outputs = self.mobilebert( 2025-08-14T21:51:33.2099602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2099685Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2099984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2100062Z layer_outputs = layer_module( 2025-08-14T21:51:33.2100366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2100468Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2100772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2100890Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2101193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2101322Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2101327Z 2025-08-14T21:51:33.2101438Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2101651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2101732Z return mod(**inputs) 2025-08-14T21:51:33.2102029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2102130Z outputs = self.mobilebert( 2025-08-14T21:51:33.2102431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2102511Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2102843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2102920Z layer_outputs = layer_module( 2025-08-14T21:51:33.2103230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2103328Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2103629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2103770Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2104073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2104164Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2104176Z 2025-08-14T21:51:33.2104289Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2104523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2104604Z return mod(**inputs) 2025-08-14T21:51:33.2104909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2104986Z outputs = self.mobilebert( 2025-08-14T21:51:33.2105297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2105377Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2105688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2105765Z layer_outputs = layer_module( 2025-08-14T21:51:33.2106103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2106212Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2106521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2106651Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2106959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2107089Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2107398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2107497Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2107501Z 2025-08-14T21:51:33.2107612Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2107833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2107907Z return mod(**inputs) 2025-08-14T21:51:33.2108209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2108286Z outputs = self.mobilebert( 2025-08-14T21:51:33.2108585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2108671Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2108991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2109065Z layer_outputs = layer_module( 2025-08-14T21:51:33.2109361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2109510Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2109821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2109912Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2109917Z 2025-08-14T21:51:33.2110026Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2110247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2110317Z return mod(**inputs) 2025-08-14T21:51:33.2110623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2110699Z outputs = self.mobilebert( 2025-08-14T21:51:33.2110999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2111086Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2111411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2111489Z layer_outputs = layer_module( 2025-08-14T21:51:33.2111794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2111919Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2112222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2112341Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2112345Z 2025-08-14T21:51:33.2112452Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2112681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2112755Z return mod(**inputs) 2025-08-14T21:51:33.2113053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2113129Z outputs = self.mobilebert( 2025-08-14T21:51:33.2113419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2113502Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2113793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2113876Z layer_outputs = layer_module( 2025-08-14T21:51:33.2114165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2114333Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2114634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.2114732Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.2114736Z 2025-08-14T21:51:33.2114841Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2115053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2115121Z return mod(**inputs) 2025-08-14T21:51:33.2115419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2115515Z outputs = self.mobilebert( 2025-08-14T21:51:33.2115802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2115887Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2116201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2116282Z layer_outputs = layer_module( 2025-08-14T21:51:33.2116576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2116740Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2117042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.2117171Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.2117462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2117569Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2117573Z 2025-08-14T21:51:33.2117682Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2117912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2117983Z return mod(**inputs) 2025-08-14T21:51:33.2118275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2118355Z outputs = self.mobilebert( 2025-08-14T21:51:33.2118645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2118729Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2119016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2119090Z layer_outputs = layer_module( 2025-08-14T21:51:33.2119407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2119575Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2119867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2120000Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2120295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.2120394Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2120398Z 2025-08-14T21:51:33.2120505Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2120715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2120793Z return mod(**inputs) 2025-08-14T21:51:33.2121088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2121175Z outputs = self.mobilebert( 2025-08-14T21:51:33.2121473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2121549Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2121852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2121929Z layer_outputs = layer_module( 2025-08-14T21:51:33.2122246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2122407Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2122697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2122852Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2123145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.2123271Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2123569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2123667Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2123673Z 2025-08-14T21:51:33.2123787Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2123995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2124066Z return mod(**inputs) 2025-08-14T21:51:33.2124366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2124458Z outputs = self.mobilebert( 2025-08-14T21:51:33.2124759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2124835Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2125123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2125204Z layer_outputs = layer_module( 2025-08-14T21:51:33.2125599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2125781Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2126123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2126245Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2126555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2126646Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2126651Z 2025-08-14T21:51:33.2126761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2126986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2127061Z return mod(**inputs) 2025-08-14T21:51:33.2127371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2127448Z outputs = self.mobilebert( 2025-08-14T21:51:33.2127747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2127838Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2128145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2128220Z layer_outputs = layer_module( 2025-08-14T21:51:33.2128519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2128613Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2128924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2129024Z self_outputs = self.self( 2025-08-14T21:51:33.2129328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.2129418Z self.value(value_tensor) 2025-08-14T21:51:33.2129441Z 2025-08-14T21:51:33.2129551Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2129773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2129843Z return mod(**inputs) 2025-08-14T21:51:33.2130143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2130225Z outputs = self.mobilebert( 2025-08-14T21:51:33.2130526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2130605Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2130911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2130988Z layer_outputs = layer_module( 2025-08-14T21:51:33.2131295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2131497Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2131804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.2131928Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.2132226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2132321Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2132325Z 2025-08-14T21:51:33.2132434Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2132643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2132736Z return mod(**inputs) 2025-08-14T21:51:33.2133038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2133120Z outputs = self.mobilebert( 2025-08-14T21:51:33.2133417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2133497Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2133798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2133872Z layer_outputs = layer_module( 2025-08-14T21:51:33.2134169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2134345Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2134645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2134769Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2135069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.2135162Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.2135466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2135566Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2135591Z 2025-08-14T21:51:33.2135706Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2135921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2135994Z return mod(**inputs) 2025-08-14T21:51:33.2136301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2136400Z outputs = self.mobilebert( 2025-08-14T21:51:33.2136701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2136787Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2137088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2137171Z layer_outputs = layer_module( 2025-08-14T21:51:33.2137470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2137561Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2138069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2138156Z self_outputs = self.self( 2025-08-14T21:51:33.2138532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.2138612Z self.query(query_tensor) 2025-08-14T21:51:33.2138617Z 2025-08-14T21:51:33.2138730Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2138949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2139022Z return mod(**inputs) 2025-08-14T21:51:33.2139318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2139404Z outputs = self.mobilebert( 2025-08-14T21:51:33.2139706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2139827Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2140131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2140208Z layer_outputs = layer_module( 2025-08-14T21:51:33.2140519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2140610Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2140917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2140996Z self_outputs = self.self( 2025-08-14T21:51:33.2141297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.2141378Z self.key(key_tensor) 2025-08-14T21:51:33.2141382Z 2025-08-14T21:51:33.2141474Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2141563Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2141682Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2141892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2141970Z return mod(**inputs) 2025-08-14T21:51:33.2142266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2142341Z outputs = self.mobilebert( 2025-08-14T21:51:33.2142642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2142768Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2143068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2143155Z layer_outputs = layer_module( 2025-08-14T21:51:33.2143490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2143587Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2143883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2144014Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2144322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.2144414Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2144418Z 2025-08-14T21:51:33.2144533Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2144739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2144811Z return mod(**inputs) 2025-08-14T21:51:33.2145113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2145212Z outputs = self.mobilebert( 2025-08-14T21:51:33.2145511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2145599Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2145896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2145977Z layer_outputs = layer_module( 2025-08-14T21:51:33.2146274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2146362Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2146690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2146827Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2147139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.2147273Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2147572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2147681Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2147686Z 2025-08-14T21:51:33.2147796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2148006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2148085Z return mod(**inputs) 2025-08-14T21:51:33.2148387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2148472Z outputs = self.mobilebert( 2025-08-14T21:51:33.2148771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2148847Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2149154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2149231Z layer_outputs = layer_module( 2025-08-14T21:51:33.2149538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2149678Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2149981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2150142Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2150443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2150532Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2150544Z 2025-08-14T21:51:33.2150652Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2150861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2150937Z return mod(**inputs) 2025-08-14T21:51:33.2151232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2151307Z outputs = self.mobilebert( 2025-08-14T21:51:33.2151611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2151688Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2152015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2152093Z layer_outputs = layer_module( 2025-08-14T21:51:33.2152394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2152500Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2152800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2152920Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2153225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2153362Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2153367Z 2025-08-14T21:51:33.2153485Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2153699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2153770Z return mod(**inputs) 2025-08-14T21:51:33.2154077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2154153Z outputs = self.mobilebert( 2025-08-14T21:51:33.2154470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2154549Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2154855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2154939Z layer_outputs = layer_module( 2025-08-14T21:51:33.2155248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2155372Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2155680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2155821Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2156128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2156238Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2156242Z 2025-08-14T21:51:33.2156360Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2156572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2156652Z return mod(**inputs) 2025-08-14T21:51:33.2156953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2157060Z outputs = self.mobilebert( 2025-08-14T21:51:33.2157370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2157447Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2157753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2157832Z layer_outputs = layer_module( 2025-08-14T21:51:33.2158134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2158240Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2158542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2158676Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2159001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2159136Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2159441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2159540Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2159546Z 2025-08-14T21:51:33.2159655Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2159881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2159951Z return mod(**inputs) 2025-08-14T21:51:33.2160322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2160401Z outputs = self.mobilebert( 2025-08-14T21:51:33.2160703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2160789Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2161090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2161167Z layer_outputs = layer_module( 2025-08-14T21:51:33.2161476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2161579Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2161890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2162012Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2162313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2162409Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2162413Z 2025-08-14T21:51:33.2162522Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2162742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2162811Z return mod(**inputs) 2025-08-14T21:51:33.2163110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2163213Z outputs = self.mobilebert( 2025-08-14T21:51:33.2163514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2163591Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2163918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2163993Z layer_outputs = layer_module( 2025-08-14T21:51:33.2164293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2164391Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2164687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2164814Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2165111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2165238Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2165243Z 2025-08-14T21:51:33.2165352Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2165678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2165765Z return mod(**inputs) 2025-08-14T21:51:33.2166067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2166144Z outputs = self.mobilebert( 2025-08-14T21:51:33.2166456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2166538Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2166842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2166918Z layer_outputs = layer_module( 2025-08-14T21:51:33.2167243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2167353Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2167647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2167785Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2168078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2168168Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2168174Z 2025-08-14T21:51:33.2168291Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2168498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2168571Z return mod(**inputs) 2025-08-14T21:51:33.2168882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2168961Z outputs = self.mobilebert( 2025-08-14T21:51:33.2169278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2169352Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2169642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2169723Z layer_outputs = layer_module( 2025-08-14T21:51:33.2170037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2170140Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2170437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2170584Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2170887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2171014Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2171317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2171416Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2171422Z 2025-08-14T21:51:33.2171533Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2171767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2171840Z return mod(**inputs) 2025-08-14T21:51:33.2172147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2172233Z outputs = self.mobilebert( 2025-08-14T21:51:33.2172554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2172639Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2172943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2173017Z layer_outputs = layer_module( 2025-08-14T21:51:33.2173330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2173428Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2173759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2173874Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2174169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2174264Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2174268Z 2025-08-14T21:51:33.2174374Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2174615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2174692Z return mod(**inputs) 2025-08-14T21:51:33.2174998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2175082Z outputs = self.mobilebert( 2025-08-14T21:51:33.2175392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2175468Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2175773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2175848Z layer_outputs = layer_module( 2025-08-14T21:51:33.2176163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2176258Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2176563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2176706Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2177009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2177125Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2177155Z 2025-08-14T21:51:33.2177262Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2177494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2177570Z return mod(**inputs) 2025-08-14T21:51:33.2177871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2177945Z outputs = self.mobilebert( 2025-08-14T21:51:33.2178245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2178324Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2178634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2178708Z layer_outputs = layer_module( 2025-08-14T21:51:33.2179012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2179135Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2179428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2179552Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2179864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2179951Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2179956Z 2025-08-14T21:51:33.2180069Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2180274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2180827Z return mod(**inputs) 2025-08-14T21:51:33.2181141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2181218Z outputs = self.mobilebert( 2025-08-14T21:51:33.2181512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2181590Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2181878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2181961Z layer_outputs = layer_module( 2025-08-14T21:51:33.2182255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2182350Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2182646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2182773Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2183067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2183194Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2183483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2183588Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2183625Z 2025-08-14T21:51:33.2183733Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2183947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2184014Z return mod(**inputs) 2025-08-14T21:51:33.2184307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2184406Z outputs = self.mobilebert( 2025-08-14T21:51:33.2184697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2184775Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2185075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2185150Z layer_outputs = layer_module( 2025-08-14T21:51:33.2185448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2185576Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2185872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2185968Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2185974Z 2025-08-14T21:51:33.2186082Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2186312Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2186383Z return mod(**inputs) 2025-08-14T21:51:33.2186678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2186760Z outputs = self.mobilebert( 2025-08-14T21:51:33.2187054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2187133Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2187436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2187531Z layer_outputs = layer_module( 2025-08-14T21:51:33.2187837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2187963Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2188263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2188394Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2188398Z 2025-08-14T21:51:33.2188509Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2188740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2188813Z return mod(**inputs) 2025-08-14T21:51:33.2189115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2189204Z outputs = self.mobilebert( 2025-08-14T21:51:33.2189518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2189609Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2189912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2189992Z layer_outputs = layer_module( 2025-08-14T21:51:33.2190300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2190489Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2190782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.2190894Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.2190898Z 2025-08-14T21:51:33.2191028Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2191249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2191320Z return mod(**inputs) 2025-08-14T21:51:33.2191617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2191701Z outputs = self.mobilebert( 2025-08-14T21:51:33.2192002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2192090Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2192388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2192465Z layer_outputs = layer_module( 2025-08-14T21:51:33.2192774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2192965Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2193269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.2193412Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.2193712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2193821Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2193825Z 2025-08-14T21:51:33.2193936Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2194145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2194243Z return mod(**inputs) 2025-08-14T21:51:33.2194545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2194631Z outputs = self.mobilebert( 2025-08-14T21:51:33.2194934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2195014Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2195318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2195396Z layer_outputs = layer_module( 2025-08-14T21:51:33.2195694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2195867Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2196165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2196306Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2196607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.2196699Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2196703Z 2025-08-14T21:51:33.2196819Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2197029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2197125Z return mod(**inputs) 2025-08-14T21:51:33.2197433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2197509Z outputs = self.mobilebert( 2025-08-14T21:51:33.2197821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2197919Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2198224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2198307Z layer_outputs = layer_module( 2025-08-14T21:51:33.2198608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2198782Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2199082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2199210Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2199523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.2199651Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2199973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2200071Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2200075Z 2025-08-14T21:51:33.2200185Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2200403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2200475Z return mod(**inputs) 2025-08-14T21:51:33.2200779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2200855Z outputs = self.mobilebert( 2025-08-14T21:51:33.2201167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2201257Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2201559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2201637Z layer_outputs = layer_module( 2025-08-14T21:51:33.2201942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2202109Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2202417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2202538Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2202840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2202937Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2202941Z 2025-08-14T21:51:33.2203052Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2203269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2203339Z return mod(**inputs) 2025-08-14T21:51:33.2203639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2203721Z outputs = self.mobilebert( 2025-08-14T21:51:33.2204018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2204118Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2204428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2204505Z layer_outputs = layer_module( 2025-08-14T21:51:33.2204837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2204930Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2205231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2205318Z self_outputs = self.self( 2025-08-14T21:51:33.2205709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.2205810Z self.value(value_tensor) 2025-08-14T21:51:33.2205815Z 2025-08-14T21:51:33.2205928Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2206143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2206222Z return mod(**inputs) 2025-08-14T21:51:33.2206525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2206628Z outputs = self.mobilebert( 2025-08-14T21:51:33.2206941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2207019Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2207330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2207407Z layer_outputs = layer_module( 2025-08-14T21:51:33.2207707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2207885Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2208211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.2208341Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.2208642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2208730Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2208734Z 2025-08-14T21:51:33.2208851Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2209060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2209149Z return mod(**inputs) 2025-08-14T21:51:33.2209439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2209510Z outputs = self.mobilebert( 2025-08-14T21:51:33.2209810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2209886Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2210176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2210257Z layer_outputs = layer_module( 2025-08-14T21:51:33.2210552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2210713Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2211014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2211121Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2211409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.2211512Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.2211802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2211897Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2211901Z 2025-08-14T21:51:33.2212009Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2212224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2212294Z return mod(**inputs) 2025-08-14T21:51:33.2212586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2212666Z outputs = self.mobilebert( 2025-08-14T21:51:33.2212961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2213044Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2213357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2213436Z layer_outputs = layer_module( 2025-08-14T21:51:33.2213735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2213825Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2214121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2214197Z self_outputs = self.self( 2025-08-14T21:51:33.2214489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.2214586Z self.query(query_tensor) 2025-08-14T21:51:33.2214592Z 2025-08-14T21:51:33.2214701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2214910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2214987Z return mod(**inputs) 2025-08-14T21:51:33.2215279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2215358Z outputs = self.mobilebert( 2025-08-14T21:51:33.2215646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2215723Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2216022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2216099Z layer_outputs = layer_module( 2025-08-14T21:51:33.2216388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2216485Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2216775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2216855Z self_outputs = self.self( 2025-08-14T21:51:33.2217143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.2217212Z self.key(key_tensor) 2025-08-14T21:51:33.2217240Z 2025-08-14T21:51:33.2217335Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2217418Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2217531Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2217737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2217804Z return mod(**inputs) 2025-08-14T21:51:33.2218123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2218196Z outputs = self.mobilebert( 2025-08-14T21:51:33.2218489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2218575Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2218865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2218948Z layer_outputs = layer_module( 2025-08-14T21:51:33.2219243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2219331Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2219633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2219778Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2220082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.2220169Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2220174Z 2025-08-14T21:51:33.2220283Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2220497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2220570Z return mod(**inputs) 2025-08-14T21:51:33.2220867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2220951Z outputs = self.mobilebert( 2025-08-14T21:51:33.2221264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2221351Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2221648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2221722Z layer_outputs = layer_module( 2025-08-14T21:51:33.2222019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2222104Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2222401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2222529Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2222820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.2222960Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2223250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2223346Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2223359Z 2025-08-14T21:51:33.2223466Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2223910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2223997Z return mod(**inputs) 2025-08-14T21:51:33.2224327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2224404Z outputs = self.mobilebert( 2025-08-14T21:51:33.2224717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2224818Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2225131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2225209Z layer_outputs = layer_module( 2025-08-14T21:51:33.2225507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2225623Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2225926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2226046Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2226355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2226443Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2226449Z 2025-08-14T21:51:33.2226567Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2226798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2226873Z return mod(**inputs) 2025-08-14T21:51:33.2227181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2227256Z outputs = self.mobilebert( 2025-08-14T21:51:33.2227558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2227637Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2227936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2228037Z layer_outputs = layer_module( 2025-08-14T21:51:33.2228339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2228442Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2228752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2228869Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2229177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2229297Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2229302Z 2025-08-14T21:51:33.2229411Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2229631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2229704Z return mod(**inputs) 2025-08-14T21:51:33.2230013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2230090Z outputs = self.mobilebert( 2025-08-14T21:51:33.2230387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2230473Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2230775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2230851Z layer_outputs = layer_module( 2025-08-14T21:51:33.2231175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2231274Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2231580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2231738Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2232042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2232145Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2232149Z 2025-08-14T21:51:33.2232264Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2232487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2232561Z return mod(**inputs) 2025-08-14T21:51:33.2232863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2232949Z outputs = self.mobilebert( 2025-08-14T21:51:33.2233254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2233335Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2233682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2233761Z layer_outputs = layer_module( 2025-08-14T21:51:33.2234071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2234171Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2234473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2234613Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2234932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2235073Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2235374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2235471Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2235475Z 2025-08-14T21:51:33.2235591Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2235812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2235889Z return mod(**inputs) 2025-08-14T21:51:33.2236186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2236261Z outputs = self.mobilebert( 2025-08-14T21:51:33.2236565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2236644Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2236943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2237027Z layer_outputs = layer_module( 2025-08-14T21:51:33.2237324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2237429Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2237895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2238080Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2238390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2238483Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2238518Z 2025-08-14T21:51:33.2238639Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2238866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2238939Z return mod(**inputs) 2025-08-14T21:51:33.2239252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2239329Z outputs = self.mobilebert( 2025-08-14T21:51:33.2239635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2239727Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2240029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2240116Z layer_outputs = layer_module( 2025-08-14T21:51:33.2240416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2240547Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2240861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2240977Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2241283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2241405Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2241409Z 2025-08-14T21:51:33.2241516Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2241738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2241834Z return mod(**inputs) 2025-08-14T21:51:33.2242139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2242224Z outputs = self.mobilebert( 2025-08-14T21:51:33.2242526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2242611Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2242907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2242982Z layer_outputs = layer_module( 2025-08-14T21:51:33.2243290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2243388Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2243693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2243827Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2244128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2244225Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2244229Z 2025-08-14T21:51:33.2244336Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2244546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2244642Z return mod(**inputs) 2025-08-14T21:51:33.2244947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2245029Z outputs = self.mobilebert( 2025-08-14T21:51:33.2245335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2245486Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2245811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2245888Z layer_outputs = layer_module( 2025-08-14T21:51:33.2246194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2246294Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2246588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2246728Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2247031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2247160Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2247487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2247590Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2247594Z 2025-08-14T21:51:33.2247715Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2247932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2248003Z return mod(**inputs) 2025-08-14T21:51:33.2248317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2248394Z outputs = self.mobilebert( 2025-08-14T21:51:33.2248721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2248804Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2249109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2249193Z layer_outputs = layer_module( 2025-08-14T21:51:33.2249508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2249605Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2249909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2250023Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2250326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2250415Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2250419Z 2025-08-14T21:51:33.2250526Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2250742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2250809Z return mod(**inputs) 2025-08-14T21:51:33.2251110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2251183Z outputs = self.mobilebert( 2025-08-14T21:51:33.2251479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2251583Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2251879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2251961Z layer_outputs = layer_module( 2025-08-14T21:51:33.2252256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2252373Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2252672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2252782Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2253069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2253191Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2253196Z 2025-08-14T21:51:33.2253303Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2253515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2253585Z return mod(**inputs) 2025-08-14T21:51:33.2253874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2253973Z outputs = self.mobilebert( 2025-08-14T21:51:33.2254264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2254346Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2254631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2254705Z layer_outputs = layer_module( 2025-08-14T21:51:33.2254999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2255096Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2255402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2255542Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2255846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2255937Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2255940Z 2025-08-14T21:51:33.2256041Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2256235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2256311Z return mod(**inputs) 2025-08-14T21:51:33.2256593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2256672Z outputs = self.mobilebert( 2025-08-14T21:51:33.2256950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2257026Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2257315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2257386Z layer_outputs = layer_module( 2025-08-14T21:51:33.2257666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2257767Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2258049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2258207Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2258499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2258656Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2258956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2259052Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2259056Z 2025-08-14T21:51:33.2259169Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2259376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2259445Z return mod(**inputs) 2025-08-14T21:51:33.2259742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2259818Z outputs = self.mobilebert( 2025-08-14T21:51:33.2260110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2260194Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2260504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2260588Z layer_outputs = layer_module( 2025-08-14T21:51:33.2260883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2261006Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2261308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2261396Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2261399Z 2025-08-14T21:51:33.2261512Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2261740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2261812Z return mod(**inputs) 2025-08-14T21:51:33.2262113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2262188Z outputs = self.mobilebert( 2025-08-14T21:51:33.2262479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2262561Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2262853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2262937Z layer_outputs = layer_module( 2025-08-14T21:51:33.2263231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2263357Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2263656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2263774Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2263778Z 2025-08-14T21:51:33.2263891Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2264096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2264165Z return mod(**inputs) 2025-08-14T21:51:33.2264461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2264557Z outputs = self.mobilebert( 2025-08-14T21:51:33.2264853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2264937Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2265231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2265334Z layer_outputs = layer_module( 2025-08-14T21:51:33.2265628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2265795Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2266097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.2266197Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.2266202Z 2025-08-14T21:51:33.2266315Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2266519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2266588Z return mod(**inputs) 2025-08-14T21:51:33.2266885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2267001Z outputs = self.mobilebert( 2025-08-14T21:51:33.2267303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2267379Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2267670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2267752Z layer_outputs = layer_module( 2025-08-14T21:51:33.2268044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2268207Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2268525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.2268656Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.2268955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2269051Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2269055Z 2025-08-14T21:51:33.2269162Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2269376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2269446Z return mod(**inputs) 2025-08-14T21:51:33.2269747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2269821Z outputs = self.mobilebert( 2025-08-14T21:51:33.2270115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2270200Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2270497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2270573Z layer_outputs = layer_module( 2025-08-14T21:51:33.2270873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2271036Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2271359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2271485Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2271776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.2271888Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2271892Z 2025-08-14T21:51:33.2272000Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2272214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2272282Z return mod(**inputs) 2025-08-14T21:51:33.2272572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2272655Z outputs = self.mobilebert( 2025-08-14T21:51:33.2272946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2273023Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2273322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2273398Z layer_outputs = layer_module( 2025-08-14T21:51:33.2273716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2273881Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2274175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2274309Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2274600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.2274735Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2275046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2275144Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2275148Z 2025-08-14T21:51:33.2275266Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2275476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2275552Z return mod(**inputs) 2025-08-14T21:51:33.2275861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2275934Z outputs = self.mobilebert( 2025-08-14T21:51:33.2276252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2276329Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2276675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2276756Z layer_outputs = layer_module( 2025-08-14T21:51:33.2277055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2277228Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2277519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2277633Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2277929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2278040Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2278044Z 2025-08-14T21:51:33.2278163Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2278360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2278440Z return mod(**inputs) 2025-08-14T21:51:33.2278727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2278798Z outputs = self.mobilebert( 2025-08-14T21:51:33.2279074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2279152Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2279429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2279507Z layer_outputs = layer_module( 2025-08-14T21:51:33.2279783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2279870Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2280168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2280278Z self_outputs = self.self( 2025-08-14T21:51:33.2280580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.2280654Z self.value(value_tensor) 2025-08-14T21:51:33.2280658Z 2025-08-14T21:51:33.2280763Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2280980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2281051Z return mod(**inputs) 2025-08-14T21:51:33.2281335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2281416Z outputs = self.mobilebert( 2025-08-14T21:51:33.2281718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2281806Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2282099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2282173Z layer_outputs = layer_module( 2025-08-14T21:51:33.2282490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2282651Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2282949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.2283063Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.2283358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2283454Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2283458Z 2025-08-14T21:51:33.2283566Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2283779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2283848Z return mod(**inputs) 2025-08-14T21:51:33.2284138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2284217Z outputs = self.mobilebert( 2025-08-14T21:51:33.2284541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2284615Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2284914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2285006Z layer_outputs = layer_module( 2025-08-14T21:51:33.2285308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2285543Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2285846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2285969Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2286264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.2286364Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.2286660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2286757Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2286762Z 2025-08-14T21:51:33.2286894Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2287105Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2287173Z return mod(**inputs) 2025-08-14T21:51:33.2287472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2287547Z outputs = self.mobilebert( 2025-08-14T21:51:33.2287844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2287922Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2288230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2288316Z layer_outputs = layer_module( 2025-08-14T21:51:33.2288614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2288711Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2289003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2289076Z self_outputs = self.self( 2025-08-14T21:51:33.2289380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.2289457Z self.query(query_tensor) 2025-08-14T21:51:33.2289461Z 2025-08-14T21:51:33.2289572Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2289802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2289872Z return mod(**inputs) 2025-08-14T21:51:33.2290173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2290249Z outputs = self.mobilebert( 2025-08-14T21:51:33.2290541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2290627Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2290922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2290995Z layer_outputs = layer_module( 2025-08-14T21:51:33.2291318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2291408Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2291714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2291806Z self_outputs = self.self( 2025-08-14T21:51:33.2292099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.2292177Z self.key(key_tensor) 2025-08-14T21:51:33.2292181Z 2025-08-14T21:51:33.2292267Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2292359Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2292466Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2292672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2292754Z return mod(**inputs) 2025-08-14T21:51:33.2293052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2293130Z outputs = self.mobilebert( 2025-08-14T21:51:33.2293438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2293535Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2293843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2293919Z layer_outputs = layer_module( 2025-08-14T21:51:33.2294214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2294311Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2294612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2294742Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2295073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.2295168Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2295171Z 2025-08-14T21:51:33.2295290Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2295504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2295574Z return mod(**inputs) 2025-08-14T21:51:33.2295881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2295955Z outputs = self.mobilebert( 2025-08-14T21:51:33.2296263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2296340Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2296642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2296726Z layer_outputs = layer_module( 2025-08-14T21:51:33.2297025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2297114Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2297420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2297551Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2297858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.2298014Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2298319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2298423Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2298447Z 2025-08-14T21:51:33.2298559Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2298779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2298849Z return mod(**inputs) 2025-08-14T21:51:33.2299150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2299236Z outputs = self.mobilebert( 2025-08-14T21:51:33.2299538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2299624Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2299922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2300001Z layer_outputs = layer_module( 2025-08-14T21:51:33.2300326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2300431Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2300732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2300857Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2301160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2301258Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2301261Z 2025-08-14T21:51:33.2301372Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2301602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2301681Z return mod(**inputs) 2025-08-14T21:51:33.2301985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2302068Z outputs = self.mobilebert( 2025-08-14T21:51:33.2302367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2302444Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2302752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2302829Z layer_outputs = layer_module( 2025-08-14T21:51:33.2303131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2303241Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2303542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2303670Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2303971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2304090Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2304094Z 2025-08-14T21:51:33.2304210Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2304421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2304525Z return mod(**inputs) 2025-08-14T21:51:33.2304827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2304903Z outputs = self.mobilebert( 2025-08-14T21:51:33.2305209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2305307Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2305615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2305698Z layer_outputs = layer_module( 2025-08-14T21:51:33.2306001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2306108Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2306413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2306546Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2306857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2306950Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2306954Z 2025-08-14T21:51:33.2307088Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2307303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2307372Z return mod(**inputs) 2025-08-14T21:51:33.2307677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2307751Z outputs = self.mobilebert( 2025-08-14T21:51:33.2308047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2308132Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2308458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2308546Z layer_outputs = layer_module( 2025-08-14T21:51:33.2308850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2308950Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2309260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2309393Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2309706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2309837Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2310140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2310245Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2310250Z 2025-08-14T21:51:33.2310360Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2310583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2310654Z return mod(**inputs) 2025-08-14T21:51:33.2310962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2311044Z outputs = self.mobilebert( 2025-08-14T21:51:33.2311335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2311432Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2311731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2311806Z layer_outputs = layer_module( 2025-08-14T21:51:33.2312123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2312222Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2312513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2312633Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2312925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2313020Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2313023Z 2025-08-14T21:51:33.2313129Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2313338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2313414Z return mod(**inputs) 2025-08-14T21:51:33.2313719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2313794Z outputs = self.mobilebert( 2025-08-14T21:51:33.2314097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2314173Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2314478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2314555Z layer_outputs = layer_module( 2025-08-14T21:51:33.2314853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2314957Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2315270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2315395Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2315694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2315809Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2315813Z 2025-08-14T21:51:33.2315928Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2316135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2316205Z return mod(**inputs) 2025-08-14T21:51:33.2316509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2316583Z outputs = self.mobilebert( 2025-08-14T21:51:33.2316889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2316967Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2317263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2317345Z layer_outputs = layer_module( 2025-08-14T21:51:33.2317642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2317749Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2318045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2318192Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2318490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2318597Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2318601Z 2025-08-14T21:51:33.2318711Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2318927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2318998Z return mod(**inputs) 2025-08-14T21:51:33.2319295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2319371Z outputs = self.mobilebert( 2025-08-14T21:51:33.2319667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2319756Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2320060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2320146Z layer_outputs = layer_module( 2025-08-14T21:51:33.2320463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2320565Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2320870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2321003Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2321301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2321438Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2321755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2321862Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2321868Z 2025-08-14T21:51:33.2321979Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2322191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2322270Z return mod(**inputs) 2025-08-14T21:51:33.2322571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2322653Z outputs = self.mobilebert( 2025-08-14T21:51:33.2322954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2323033Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2323338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2323415Z layer_outputs = layer_module( 2025-08-14T21:51:33.2323715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2323825Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2324125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2324248Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2324546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2324660Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2324664Z 2025-08-14T21:51:33.2324783Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2324995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2325075Z return mod(**inputs) 2025-08-14T21:51:33.2325394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2325552Z outputs = self.mobilebert( 2025-08-14T21:51:33.2325873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2325953Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2326255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2326343Z layer_outputs = layer_module( 2025-08-14T21:51:33.2326648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2326757Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2327058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2327189Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2327517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2327635Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2327639Z 2025-08-14T21:51:33.2327756Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2327964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2328035Z return mod(**inputs) 2025-08-14T21:51:33.2328336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2328412Z outputs = self.mobilebert( 2025-08-14T21:51:33.2328743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2328826Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2329133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2329216Z layer_outputs = layer_module( 2025-08-14T21:51:33.2329516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2329616Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2329923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2330057Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2330365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2330457Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2330461Z 2025-08-14T21:51:33.2330571Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2330792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2330862Z return mod(**inputs) 2025-08-14T21:51:33.2331168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2331242Z outputs = self.mobilebert( 2025-08-14T21:51:33.2331542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2331648Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2331950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2332027Z layer_outputs = layer_module( 2025-08-14T21:51:33.2332358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2332457Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2332765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2332895Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2333191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2333329Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2333630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2333734Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2333739Z 2025-08-14T21:51:33.2333847Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2334087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2334171Z return mod(**inputs) 2025-08-14T21:51:33.2334471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2334547Z outputs = self.mobilebert( 2025-08-14T21:51:33.2334852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2334932Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2335239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2335331Z layer_outputs = layer_module( 2025-08-14T21:51:33.2335635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2335778Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2336083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2336179Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2336183Z 2025-08-14T21:51:33.2336290Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2336500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2336579Z return mod(**inputs) 2025-08-14T21:51:33.2336880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2336956Z outputs = self.mobilebert( 2025-08-14T21:51:33.2337266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2337346Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2337904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2337990Z layer_outputs = layer_module( 2025-08-14T21:51:33.2338290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2338427Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2338783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2338908Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2338913Z 2025-08-14T21:51:33.2339025Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2339300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2339381Z return mod(**inputs) 2025-08-14T21:51:33.2339681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2339757Z outputs = self.mobilebert( 2025-08-14T21:51:33.2340120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2340198Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2340506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2340583Z layer_outputs = layer_module( 2025-08-14T21:51:33.2340896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2341076Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2341417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.2341529Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.2341534Z 2025-08-14T21:51:33.2341645Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2341879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2341958Z return mod(**inputs) 2025-08-14T21:51:33.2342265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2342339Z outputs = self.mobilebert( 2025-08-14T21:51:33.2342685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2342768Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2343128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2343205Z layer_outputs = layer_module( 2025-08-14T21:51:33.2343510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2343681Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2343971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.2344109Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.2344401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2344498Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2344501Z 2025-08-14T21:51:33.2344618Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2344824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2344899Z return mod(**inputs) 2025-08-14T21:51:33.2345185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2345259Z outputs = self.mobilebert( 2025-08-14T21:51:33.2345558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2345656Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2345950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2346048Z layer_outputs = layer_module( 2025-08-14T21:51:33.2346341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2346512Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2346802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2346930Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2347226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.2347315Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2347319Z 2025-08-14T21:51:33.2347435Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2347656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2347725Z return mod(**inputs) 2025-08-14T21:51:33.2348042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2348118Z outputs = self.mobilebert( 2025-08-14T21:51:33.2348407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2348489Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2348778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2348859Z layer_outputs = layer_module( 2025-08-14T21:51:33.2349145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2349322Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2349628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2349753Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2350052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.2350176Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2350468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2350572Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2350576Z 2025-08-14T21:51:33.2350681Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2350893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2350962Z return mod(**inputs) 2025-08-14T21:51:33.2351256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2351336Z outputs = self.mobilebert( 2025-08-14T21:51:33.2351626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2351703Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2352003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2352099Z layer_outputs = layer_module( 2025-08-14T21:51:33.2352398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2352565Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2352880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2353004Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2353293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2353385Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2353389Z 2025-08-14T21:51:33.2353493Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2353701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2353775Z return mod(**inputs) 2025-08-14T21:51:33.2354059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2354129Z outputs = self.mobilebert( 2025-08-14T21:51:33.2354431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2354504Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2354789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2354861Z layer_outputs = layer_module( 2025-08-14T21:51:33.2355147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2355243Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2355531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2355613Z self_outputs = self.self( 2025-08-14T21:51:33.2355920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.2355996Z self.value(value_tensor) 2025-08-14T21:51:33.2355999Z 2025-08-14T21:51:33.2356112Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2356319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2356387Z return mod(**inputs) 2025-08-14T21:51:33.2356682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2356753Z outputs = self.mobilebert( 2025-08-14T21:51:33.2357052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2357129Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2357418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2357497Z layer_outputs = layer_module( 2025-08-14T21:51:33.2357781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2357951Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2358245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.2358359Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.2358659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2358764Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2358768Z 2025-08-14T21:51:33.2358875Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2359090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2359186Z return mod(**inputs) 2025-08-14T21:51:33.2359490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2359562Z outputs = self.mobilebert( 2025-08-14T21:51:33.2359858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2359941Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2360235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2360319Z layer_outputs = layer_module( 2025-08-14T21:51:33.2360615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2360781Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2361106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2361223Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2361511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.2361610Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.2361901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2362006Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2362010Z 2025-08-14T21:51:33.2362115Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2362338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2362421Z return mod(**inputs) 2025-08-14T21:51:33.2362718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2362800Z outputs = self.mobilebert( 2025-08-14T21:51:33.2363095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2363173Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2363476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2363552Z layer_outputs = layer_module( 2025-08-14T21:51:33.2363847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2363945Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2364238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2364324Z self_outputs = self.self( 2025-08-14T21:51:33.2364617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.2364691Z self.query(query_tensor) 2025-08-14T21:51:33.2364695Z 2025-08-14T21:51:33.2364816Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2365023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2365123Z return mod(**inputs) 2025-08-14T21:51:33.2365497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2365583Z outputs = self.mobilebert( 2025-08-14T21:51:33.2365900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2366005Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2366309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2366395Z layer_outputs = layer_module( 2025-08-14T21:51:33.2366696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2366795Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2367095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2367174Z self_outputs = self.self( 2025-08-14T21:51:33.2367482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.2367554Z self.key(key_tensor) 2025-08-14T21:51:33.2367560Z 2025-08-14T21:51:33.2367656Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2367761Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2367885Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2368104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2368174Z return mod(**inputs) 2025-08-14T21:51:33.2368469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2368551Z outputs = self.mobilebert( 2025-08-14T21:51:33.2368846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2368929Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2369237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2369314Z layer_outputs = layer_module( 2025-08-14T21:51:33.2369618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2369707Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2369997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2370134Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2370423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.2370521Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2370524Z 2025-08-14T21:51:33.2370630Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2370835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2370914Z return mod(**inputs) 2025-08-14T21:51:33.2371206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2371286Z outputs = self.mobilebert( 2025-08-14T21:51:33.2371581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2371656Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2371956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2372052Z layer_outputs = layer_module( 2025-08-14T21:51:33.2372350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2372445Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2372787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2372919Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2373208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.2373339Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2373634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2373731Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2373735Z 2025-08-14T21:51:33.2373848Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2374054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2374125Z return mod(**inputs) 2025-08-14T21:51:33.2374442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2374516Z outputs = self.mobilebert( 2025-08-14T21:51:33.2374816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2374891Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2375181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2375263Z layer_outputs = layer_module( 2025-08-14T21:51:33.2375558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2375674Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2375974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2376092Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2376394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2376477Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2376480Z 2025-08-14T21:51:33.2376582Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2376787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2376854Z return mod(**inputs) 2025-08-14T21:51:33.2377134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2377203Z outputs = self.mobilebert( 2025-08-14T21:51:33.2377478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2377560Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2377833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2377904Z layer_outputs = layer_module( 2025-08-14T21:51:33.2378189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2378283Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2378582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2378689Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2378979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2379124Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2379128Z 2025-08-14T21:51:33.2379236Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2379450Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2379521Z return mod(**inputs) 2025-08-14T21:51:33.2379807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2379887Z outputs = self.mobilebert( 2025-08-14T21:51:33.2380175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2380251Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2380548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2380624Z layer_outputs = layer_module( 2025-08-14T21:51:33.2380938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2381037Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2381330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2381468Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2381759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2381856Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2381859Z 2025-08-14T21:51:33.2381965Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2382187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2382268Z return mod(**inputs) 2025-08-14T21:51:33.2382559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2382636Z outputs = self.mobilebert( 2025-08-14T21:51:33.2382937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2383014Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2383309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2383387Z layer_outputs = layer_module( 2025-08-14T21:51:33.2383675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2383782Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2384072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2384208Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2384498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2384623Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2384919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2385048Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2385052Z 2025-08-14T21:51:33.2385169Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2385380Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2385467Z return mod(**inputs) 2025-08-14T21:51:33.2385774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2385849Z outputs = self.mobilebert( 2025-08-14T21:51:33.2386147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2386231Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2386529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2386615Z layer_outputs = layer_module( 2025-08-14T21:51:33.2386913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2387008Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2387312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2387445Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2387751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2387837Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2387840Z 2025-08-14T21:51:33.2387945Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2388154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2388224Z return mod(**inputs) 2025-08-14T21:51:33.2388516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2388598Z outputs = self.mobilebert( 2025-08-14T21:51:33.2388907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2388996Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2389289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2389365Z layer_outputs = layer_module( 2025-08-14T21:51:33.2389661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2389758Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2390054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2390175Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2390467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2390589Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2390593Z 2025-08-14T21:51:33.2390701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2390906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2390981Z return mod(**inputs) 2025-08-14T21:51:33.2391271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2391351Z outputs = self.mobilebert( 2025-08-14T21:51:33.2391663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2391737Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2392040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2392134Z layer_outputs = layer_module( 2025-08-14T21:51:33.2392437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2392532Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2392824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2392961Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2393254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2393343Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2393354Z 2025-08-14T21:51:33.2393461Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2393670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2393748Z return mod(**inputs) 2025-08-14T21:51:33.2394059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2394134Z outputs = self.mobilebert( 2025-08-14T21:51:33.2394440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2394516Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2394815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2394891Z layer_outputs = layer_module( 2025-08-14T21:51:33.2395187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2395308Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2395603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2395732Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2396036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2396161Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2396466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2396560Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2396564Z 2025-08-14T21:51:33.2396670Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2396883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2396954Z return mod(**inputs) 2025-08-14T21:51:33.2397257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2397332Z outputs = self.mobilebert( 2025-08-14T21:51:33.2397625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2397709Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2398000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2398098Z layer_outputs = layer_module( 2025-08-14T21:51:33.2398396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2398493Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2398793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2398927Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2399214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2399309Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2399312Z 2025-08-14T21:51:33.2399417Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2399631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2399701Z return mod(**inputs) 2025-08-14T21:51:33.2399990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2400072Z outputs = self.mobilebert( 2025-08-14T21:51:33.2400364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2400442Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2400756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2400832Z layer_outputs = layer_module( 2025-08-14T21:51:33.2401143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2401240Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2401531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2401653Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2401974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2402098Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2402102Z 2025-08-14T21:51:33.2402212Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2402422Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2402499Z return mod(**inputs) 2025-08-14T21:51:33.2402803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2402878Z outputs = self.mobilebert( 2025-08-14T21:51:33.2403177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2403253Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2403559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2403639Z layer_outputs = layer_module( 2025-08-14T21:51:33.2403941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2404059Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2404364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2404500Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2404801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2404910Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2404914Z 2025-08-14T21:51:33.2405027Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2405254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2405354Z return mod(**inputs) 2025-08-14T21:51:33.2405830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2405913Z outputs = self.mobilebert( 2025-08-14T21:51:33.2406222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2406300Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2406599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2406686Z layer_outputs = layer_module( 2025-08-14T21:51:33.2406983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2407103Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2407395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2407544Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2407842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2407968Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2408279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2408377Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2408381Z 2025-08-14T21:51:33.2408488Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2408743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2408813Z return mod(**inputs) 2025-08-14T21:51:33.2409116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2409196Z outputs = self.mobilebert( 2025-08-14T21:51:33.2409502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2409584Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2409879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2409953Z layer_outputs = layer_module( 2025-08-14T21:51:33.2410294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2410420Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2410723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2410813Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2410816Z 2025-08-14T21:51:33.2410923Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2411136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2411205Z return mod(**inputs) 2025-08-14T21:51:33.2411495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2411595Z outputs = self.mobilebert( 2025-08-14T21:51:33.2411887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2411969Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2412264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2412359Z layer_outputs = layer_module( 2025-08-14T21:51:33.2412665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2412789Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2413092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2413207Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2413213Z 2025-08-14T21:51:33.2413320Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2413536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2413605Z return mod(**inputs) 2025-08-14T21:51:33.2413897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2413981Z outputs = self.mobilebert( 2025-08-14T21:51:33.2414288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2414374Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2414674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2414750Z layer_outputs = layer_module( 2025-08-14T21:51:33.2415049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2415217Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2415538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.2415640Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.2415644Z 2025-08-14T21:51:33.2415751Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2415963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2416031Z return mod(**inputs) 2025-08-14T21:51:33.2416321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2416402Z outputs = self.mobilebert( 2025-08-14T21:51:33.2416692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2416776Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2417066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2417140Z layer_outputs = layer_module( 2025-08-14T21:51:33.2417441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2417604Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2417897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.2418024Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.2418313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2418441Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2418444Z 2025-08-14T21:51:33.2418551Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2418768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2418857Z return mod(**inputs) 2025-08-14T21:51:33.2419150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2419231Z outputs = self.mobilebert( 2025-08-14T21:51:33.2419521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2419597Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2419895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2419971Z layer_outputs = layer_module( 2025-08-14T21:51:33.2420268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2420434Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2420740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2420877Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2421168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.2421262Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2421265Z 2025-08-14T21:51:33.2421371Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2421580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2421656Z return mod(**inputs) 2025-08-14T21:51:33.2421967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2422043Z outputs = self.mobilebert( 2025-08-14T21:51:33.2422345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2422420Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2422719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2422793Z layer_outputs = layer_module( 2025-08-14T21:51:33.2423083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2423253Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2423544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2423678Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2423970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.2424098Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2424392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2424488Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2424491Z 2025-08-14T21:51:33.2424603Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2424830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2424899Z return mod(**inputs) 2025-08-14T21:51:33.2425193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2425267Z outputs = self.mobilebert( 2025-08-14T21:51:33.2425578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2425661Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2425952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2426035Z layer_outputs = layer_module( 2025-08-14T21:51:33.2426327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2426498Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2426799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2426917Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2427217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2427323Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2427328Z 2025-08-14T21:51:33.2427436Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2427652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2427720Z return mod(**inputs) 2025-08-14T21:51:33.2428010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2428092Z outputs = self.mobilebert( 2025-08-14T21:51:33.2428382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2428465Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2428772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2428852Z layer_outputs = layer_module( 2025-08-14T21:51:33.2429154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2429241Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2429545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2429621Z self_outputs = self.self( 2025-08-14T21:51:33.2429919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.2430003Z self.value(value_tensor) 2025-08-14T21:51:33.2430007Z 2025-08-14T21:51:33.2430124Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2430330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2430411Z return mod(**inputs) 2025-08-14T21:51:33.2430708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2430792Z outputs = self.mobilebert( 2025-08-14T21:51:33.2431090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2431168Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2431476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2431576Z layer_outputs = layer_module( 2025-08-14T21:51:33.2431890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2432062Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2432384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.2432508Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.2432805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2432892Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2432903Z 2025-08-14T21:51:33.2433013Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2433223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2433308Z return mod(**inputs) 2025-08-14T21:51:33.2433606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2433682Z outputs = self.mobilebert( 2025-08-14T21:51:33.2434012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2434092Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2434402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2434478Z layer_outputs = layer_module( 2025-08-14T21:51:33.2434778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2434953Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2435256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2435388Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2435699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.2435790Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.2436093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2436190Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2436193Z 2025-08-14T21:51:33.2436302Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2436519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2436590Z return mod(**inputs) 2025-08-14T21:51:33.2436894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2436971Z outputs = self.mobilebert( 2025-08-14T21:51:33.2437271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2437358Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2437904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2437989Z layer_outputs = layer_module( 2025-08-14T21:51:33.2438300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2438393Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2438763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2438840Z self_outputs = self.self( 2025-08-14T21:51:33.2439138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.2439249Z self.query(query_tensor) 2025-08-14T21:51:33.2439254Z 2025-08-14T21:51:33.2439367Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2439588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2439658Z return mod(**inputs) 2025-08-14T21:51:33.2439958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2440043Z outputs = self.mobilebert( 2025-08-14T21:51:33.2440345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2440424Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2440735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2440814Z layer_outputs = layer_module( 2025-08-14T21:51:33.2442233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2442363Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2442665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2442749Z self_outputs = self.self( 2025-08-14T21:51:33.2443049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.2443134Z self.key(key_tensor) 2025-08-14T21:51:33.2443137Z 2025-08-14T21:51:33.2443227Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2443315Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2443466Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2443681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2443753Z return mod(**inputs) 2025-08-14T21:51:33.2444061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2444136Z outputs = self.mobilebert( 2025-08-14T21:51:33.2444441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2444518Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2444820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2444905Z layer_outputs = layer_module( 2025-08-14T21:51:33.2445206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2445296Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2445696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2445837Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2446151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.2446243Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2446247Z 2025-08-14T21:51:33.2446358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2446608Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2446679Z return mod(**inputs) 2025-08-14T21:51:33.2446999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2447095Z outputs = self.mobilebert( 2025-08-14T21:51:33.2447408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2447495Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2447808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2447885Z layer_outputs = layer_module( 2025-08-14T21:51:33.2448205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2448294Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2448613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2448746Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2449058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.2449222Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2449579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2449684Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2449687Z 2025-08-14T21:51:33.2449794Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2449996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2450075Z return mod(**inputs) 2025-08-14T21:51:33.2450368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2450467Z outputs = self.mobilebert( 2025-08-14T21:51:33.2450760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2450839Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2451135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2451209Z layer_outputs = layer_module( 2025-08-14T21:51:33.2451497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2451606Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2451895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2452016Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2452310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2452398Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2452406Z 2025-08-14T21:51:33.2452520Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2452727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2452802Z return mod(**inputs) 2025-08-14T21:51:33.2453090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2453162Z outputs = self.mobilebert( 2025-08-14T21:51:33.2453478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2453553Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2453843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2453943Z layer_outputs = layer_module( 2025-08-14T21:51:33.2454236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2454340Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2454632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2454746Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2455041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2455156Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2455160Z 2025-08-14T21:51:33.2455273Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2455478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2455549Z return mod(**inputs) 2025-08-14T21:51:33.2455860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2455935Z outputs = self.mobilebert( 2025-08-14T21:51:33.2456228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2456313Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2456602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2456684Z layer_outputs = layer_module( 2025-08-14T21:51:33.2456990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2457089Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2457391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2457523Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2457816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2457904Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2457907Z 2025-08-14T21:51:33.2458013Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2458229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2458296Z return mod(**inputs) 2025-08-14T21:51:33.2458584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2458666Z outputs = self.mobilebert( 2025-08-14T21:51:33.2458956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2459036Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2459321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2459393Z layer_outputs = layer_module( 2025-08-14T21:51:33.2459686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2459803Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2460103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2460233Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2460523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2460691Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2460980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2461073Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2461084Z 2025-08-14T21:51:33.2461190Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2461395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2461472Z return mod(**inputs) 2025-08-14T21:51:33.2461759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2461832Z outputs = self.mobilebert( 2025-08-14T21:51:33.2462128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2462226Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2462523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2462598Z layer_outputs = layer_module( 2025-08-14T21:51:33.2462886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2462987Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2463276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2463391Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2463709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2463800Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2463805Z 2025-08-14T21:51:33.2463920Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2464127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2464198Z return mod(**inputs) 2025-08-14T21:51:33.2464498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2464572Z outputs = self.mobilebert( 2025-08-14T21:51:33.2464871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2464947Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2465243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2465329Z layer_outputs = layer_module( 2025-08-14T21:51:33.2465622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2465718Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2466021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2466135Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2466438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2466574Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2466577Z 2025-08-14T21:51:33.2466684Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2466900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2466989Z return mod(**inputs) 2025-08-14T21:51:33.2467289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2467364Z outputs = self.mobilebert( 2025-08-14T21:51:33.2467656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2467739Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2468029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2468105Z layer_outputs = layer_module( 2025-08-14T21:51:33.2468411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2468511Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2468840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2468975Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2469276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2469373Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2469377Z 2025-08-14T21:51:33.2469485Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2469703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2469774Z return mod(**inputs) 2025-08-14T21:51:33.2470101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2470184Z outputs = self.mobilebert( 2025-08-14T21:51:33.2470486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2470570Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2470863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2470938Z layer_outputs = layer_module( 2025-08-14T21:51:33.2471239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2471336Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2471629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2471770Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2472066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2472205Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2472501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2472595Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2472599Z 2025-08-14T21:51:33.2472714Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2472921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2473023Z return mod(**inputs) 2025-08-14T21:51:33.2473312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2473387Z outputs = self.mobilebert( 2025-08-14T21:51:33.2473703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2473805Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2474095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2474176Z layer_outputs = layer_module( 2025-08-14T21:51:33.2474465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2474567Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2474859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2474973Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2475274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2475363Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2475366Z 2025-08-14T21:51:33.2475496Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2475705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2475773Z return mod(**inputs) 2025-08-14T21:51:33.2476069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2476144Z outputs = self.mobilebert( 2025-08-14T21:51:33.2476440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2476523Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2476830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2476915Z layer_outputs = layer_module( 2025-08-14T21:51:33.2477211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2477307Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2477605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2477718Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2478014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2478129Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2478133Z 2025-08-14T21:51:33.2478239Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2478454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2478524Z return mod(**inputs) 2025-08-14T21:51:33.2478814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2478893Z outputs = self.mobilebert( 2025-08-14T21:51:33.2479186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2479272Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2479575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2479711Z layer_outputs = layer_module( 2025-08-14T21:51:33.2480009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2480107Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2480420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2480547Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2480834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2480930Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2480933Z 2025-08-14T21:51:33.2481039Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2481255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2481326Z return mod(**inputs) 2025-08-14T21:51:33.2481616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2481698Z outputs = self.mobilebert( 2025-08-14T21:51:33.2482005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2482083Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2482387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2482462Z layer_outputs = layer_module( 2025-08-14T21:51:33.2482761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2482860Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2483159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2483316Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2483616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2483755Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2484057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2484154Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2484157Z 2025-08-14T21:51:33.2484274Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2484487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2484563Z return mod(**inputs) 2025-08-14T21:51:33.2484874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2484955Z outputs = self.mobilebert( 2025-08-14T21:51:33.2485267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2485352Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2485717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2485810Z layer_outputs = layer_module( 2025-08-14T21:51:33.2486108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2486247Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2486588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2486678Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2486683Z 2025-08-14T21:51:33.2486805Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2487036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2487111Z return mod(**inputs) 2025-08-14T21:51:33.2487422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2487497Z outputs = self.mobilebert( 2025-08-14T21:51:33.2487804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2487883Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2488184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2488270Z layer_outputs = layer_module( 2025-08-14T21:51:33.2488575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2488710Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2489032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2489152Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2489156Z 2025-08-14T21:51:33.2489274Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2489486Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2489557Z return mod(**inputs) 2025-08-14T21:51:33.2489867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2489943Z outputs = self.mobilebert( 2025-08-14T21:51:33.2490266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2490348Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2490649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2490735Z layer_outputs = layer_module( 2025-08-14T21:51:33.2491032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2491207Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2491503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.2491604Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.2491608Z 2025-08-14T21:51:33.2491725Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2491936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2492009Z return mod(**inputs) 2025-08-14T21:51:33.2492314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2492390Z outputs = self.mobilebert( 2025-08-14T21:51:33.2492694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2492771Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2493066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2493170Z layer_outputs = layer_module( 2025-08-14T21:51:33.2493475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2493654Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2493981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.2494109Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.2494416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2494513Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2494518Z 2025-08-14T21:51:33.2494633Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2494845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2494914Z return mod(**inputs) 2025-08-14T21:51:33.2495225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2495299Z outputs = self.mobilebert( 2025-08-14T21:51:33.2495616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2495701Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2496000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2496084Z layer_outputs = layer_module( 2025-08-14T21:51:33.2496379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2496545Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2496862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2497007Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2497310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.2497400Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2497404Z 2025-08-14T21:51:33.2497512Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2497724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2497792Z return mod(**inputs) 2025-08-14T21:51:33.2498089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2498173Z outputs = self.mobilebert( 2025-08-14T21:51:33.2498474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2498562Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2498860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2498940Z layer_outputs = layer_module( 2025-08-14T21:51:33.2499247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2499414Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2499719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2499876Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2500164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.2500298Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2500591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2500704Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2500716Z 2025-08-14T21:51:33.2500823Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2501029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2501108Z return mod(**inputs) 2025-08-14T21:51:33.2501399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2501475Z outputs = self.mobilebert( 2025-08-14T21:51:33.2501774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2501852Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2502155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2502232Z layer_outputs = layer_module( 2025-08-14T21:51:33.2502539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2502719Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2503011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2503134Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2503425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2503511Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2503515Z 2025-08-14T21:51:33.2503645Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2503855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2503932Z return mod(**inputs) 2025-08-14T21:51:33.2504239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2504312Z outputs = self.mobilebert( 2025-08-14T21:51:33.2504607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2504684Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2504973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2505056Z layer_outputs = layer_module( 2025-08-14T21:51:33.2505346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2505443Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2505734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2505810Z self_outputs = self.self( 2025-08-14T21:51:33.2506104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.2506178Z self.value(value_tensor) 2025-08-14T21:51:33.2506182Z 2025-08-14T21:51:33.2506287Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2506525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2506593Z return mod(**inputs) 2025-08-14T21:51:33.2506893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2506984Z outputs = self.mobilebert( 2025-08-14T21:51:33.2507278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2507363Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2507656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2507729Z layer_outputs = layer_module( 2025-08-14T21:51:33.2508033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2508200Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2508503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.2508619Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.2508930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2509026Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2509030Z 2025-08-14T21:51:33.2509137Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2509349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2509417Z return mod(**inputs) 2025-08-14T21:51:33.2509705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2509789Z outputs = self.mobilebert( 2025-08-14T21:51:33.2510082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2510192Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2510481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2510560Z layer_outputs = layer_module( 2025-08-14T21:51:33.2510853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2511016Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2511306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2511428Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2511717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.2511815Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.2512105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2512203Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2512207Z 2025-08-14T21:51:33.2512322Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2512527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2512604Z return mod(**inputs) 2025-08-14T21:51:33.2512894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2513004Z outputs = self.mobilebert( 2025-08-14T21:51:33.2513306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2513383Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2513670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2513785Z layer_outputs = layer_module( 2025-08-14T21:51:33.2514077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2514172Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2514465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2514540Z self_outputs = self.self( 2025-08-14T21:51:33.2514841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.2514917Z self.query(query_tensor) 2025-08-14T21:51:33.2514921Z 2025-08-14T21:51:33.2515035Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2515243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2515315Z return mod(**inputs) 2025-08-14T21:51:33.2515633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2515709Z outputs = self.mobilebert( 2025-08-14T21:51:33.2516003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2516088Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2516382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2516464Z layer_outputs = layer_module( 2025-08-14T21:51:33.2516767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2516873Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2517177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2517251Z self_outputs = self.self( 2025-08-14T21:51:33.2517549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.2517621Z self.key(key_tensor) 2025-08-14T21:51:33.2517624Z 2025-08-14T21:51:33.2517714Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2517807Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2517916Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2518121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2518198Z return mod(**inputs) 2025-08-14T21:51:33.2518525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2518607Z outputs = self.mobilebert( 2025-08-14T21:51:33.2518906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2518985Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2519289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2519364Z layer_outputs = layer_module( 2025-08-14T21:51:33.2519662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2519782Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2520083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2520224Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2520542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.2520633Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2520637Z 2025-08-14T21:51:33.2520755Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2520968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2521045Z return mod(**inputs) 2025-08-14T21:51:33.2521342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2521418Z outputs = self.mobilebert( 2025-08-14T21:51:33.2521722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2521801Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2522100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2522200Z layer_outputs = layer_module( 2025-08-14T21:51:33.2522500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2522595Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2522895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2523026Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2523331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.2523462Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2523788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2523892Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2523896Z 2025-08-14T21:51:33.2524006Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2524224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2524294Z return mod(**inputs) 2025-08-14T21:51:33.2524592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2524677Z outputs = self.mobilebert( 2025-08-14T21:51:33.2524977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2525062Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2525364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2525512Z layer_outputs = layer_module( 2025-08-14T21:51:33.2525839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2525941Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2526250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2526370Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2526701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2526800Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2526804Z 2025-08-14T21:51:33.2526915Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2527138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2527234Z return mod(**inputs) 2025-08-14T21:51:33.2527549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2527632Z outputs = self.mobilebert( 2025-08-14T21:51:33.2527943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2528021Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2528341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2528420Z layer_outputs = layer_module( 2025-08-14T21:51:33.2528779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2528880Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2529201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2529329Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2529638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2529765Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2529769Z 2025-08-14T21:51:33.2529877Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2530089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2530165Z return mod(**inputs) 2025-08-14T21:51:33.2530492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2530569Z outputs = self.mobilebert( 2025-08-14T21:51:33.2530889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2530967Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2531288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2531365Z layer_outputs = layer_module( 2025-08-14T21:51:33.2531677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2531787Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2532087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2532231Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2532540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2532630Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2532635Z 2025-08-14T21:51:33.2532749Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2532972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2533041Z return mod(**inputs) 2025-08-14T21:51:33.2533341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2533435Z outputs = self.mobilebert( 2025-08-14T21:51:33.2533746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2533824Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2534113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2534217Z layer_outputs = layer_module( 2025-08-14T21:51:33.2534516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2534619Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2534914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2535043Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2535346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2535470Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2535766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2535902Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2535907Z 2025-08-14T21:51:33.2536015Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2536230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2536299Z return mod(**inputs) 2025-08-14T21:51:33.2536590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2536675Z outputs = self.mobilebert( 2025-08-14T21:51:33.2536966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2537049Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2537360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2537439Z layer_outputs = layer_module( 2025-08-14T21:51:33.2537970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2538077Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2538370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2538500Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2538805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2538907Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2538911Z 2025-08-14T21:51:33.2539023Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2539235Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2539319Z return mod(**inputs) 2025-08-14T21:51:33.2539620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2539707Z outputs = self.mobilebert( 2025-08-14T21:51:33.2540015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2540091Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2540390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2540540Z layer_outputs = layer_module( 2025-08-14T21:51:33.2540835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2540937Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2541262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2541385Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2541676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2541790Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2541794Z 2025-08-14T21:51:33.2541910Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2542121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2542197Z return mod(**inputs) 2025-08-14T21:51:33.2542488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2542563Z outputs = self.mobilebert( 2025-08-14T21:51:33.2542891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2542970Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2543263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2543346Z layer_outputs = layer_module( 2025-08-14T21:51:33.2543639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2543748Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2544052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2544213Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2544526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2544618Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2544622Z 2025-08-14T21:51:33.2544738Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2544957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2545027Z return mod(**inputs) 2025-08-14T21:51:33.2545324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2545400Z outputs = self.mobilebert( 2025-08-14T21:51:33.2545695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2545770Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2546061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2546144Z layer_outputs = layer_module( 2025-08-14T21:51:33.2546432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2546526Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2546822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2546947Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2547262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2547385Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2547673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2547794Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2547799Z 2025-08-14T21:51:33.2547910Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2548127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2548197Z return mod(**inputs) 2025-08-14T21:51:33.2548492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2548577Z outputs = self.mobilebert( 2025-08-14T21:51:33.2548874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2548953Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2549260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2549339Z layer_outputs = layer_module( 2025-08-14T21:51:33.2549659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2549758Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2550047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2550169Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2550456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2550550Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2550554Z 2025-08-14T21:51:33.2550661Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2550884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2550966Z return mod(**inputs) 2025-08-14T21:51:33.2551259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2551332Z outputs = self.mobilebert( 2025-08-14T21:51:33.2551635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2551713Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2552011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2552088Z layer_outputs = layer_module( 2025-08-14T21:51:33.2552378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2552482Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2552774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2552894Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2553182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2553298Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2553302Z 2025-08-14T21:51:33.2553415Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2553649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2553716Z return mod(**inputs) 2025-08-14T21:51:33.2554017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2554092Z outputs = self.mobilebert( 2025-08-14T21:51:33.2554410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2554485Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2554773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2554855Z layer_outputs = layer_module( 2025-08-14T21:51:33.2555144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2555249Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2555535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2555662Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2555953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2556060Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2556064Z 2025-08-14T21:51:33.2556173Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2556366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2556431Z return mod(**inputs) 2025-08-14T21:51:33.2556711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2556781Z outputs = self.mobilebert( 2025-08-14T21:51:33.2557054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2557135Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2557425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2557505Z layer_outputs = layer_module( 2025-08-14T21:51:33.2557784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2557875Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2558164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2558291Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2558594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2558720Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2559016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2559120Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2559125Z 2025-08-14T21:51:33.2559232Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2559435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2559513Z return mod(**inputs) 2025-08-14T21:51:33.2559807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2559888Z outputs = self.mobilebert( 2025-08-14T21:51:33.2560207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2560286Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2560603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2560705Z layer_outputs = layer_module( 2025-08-14T21:51:33.2561006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2561131Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2561427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2561523Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2561527Z 2025-08-14T21:51:33.2561637Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2561848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2561928Z return mod(**inputs) 2025-08-14T21:51:33.2562228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2562314Z outputs = self.mobilebert( 2025-08-14T21:51:33.2562639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2562717Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2563014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2563089Z layer_outputs = layer_module( 2025-08-14T21:51:33.2563384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2563509Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2563816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2563939Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2563944Z 2025-08-14T21:51:33.2564052Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2564254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2564330Z return mod(**inputs) 2025-08-14T21:51:33.2564618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2564696Z outputs = self.mobilebert( 2025-08-14T21:51:33.2564987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2565064Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2565365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2565506Z layer_outputs = layer_module( 2025-08-14T21:51:33.2565827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2565999Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2566297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.2566405Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.2566410Z 2025-08-14T21:51:33.2566519Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2566760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2566838Z return mod(**inputs) 2025-08-14T21:51:33.2567129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2567217Z outputs = self.mobilebert( 2025-08-14T21:51:33.2567528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2567604Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2567902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2567976Z layer_outputs = layer_module( 2025-08-14T21:51:33.2568273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2568440Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2568729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.2568864Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.2569151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2569270Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2569282Z 2025-08-14T21:51:33.2569393Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2569618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2569695Z return mod(**inputs) 2025-08-14T21:51:33.2569996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2570082Z outputs = self.mobilebert( 2025-08-14T21:51:33.2570387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2570482Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2570787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2570865Z layer_outputs = layer_module( 2025-08-14T21:51:33.2571155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2571324Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2571615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2571749Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2572038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.2572126Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2572131Z 2025-08-14T21:51:33.2572244Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2572454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2572521Z return mod(**inputs) 2025-08-14T21:51:33.2572819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2572899Z outputs = self.mobilebert( 2025-08-14T21:51:33.2573198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2573295Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2573592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2573677Z layer_outputs = layer_module( 2025-08-14T21:51:33.2573981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2574176Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2574467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2574592Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2574888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.2575011Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2575301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2575402Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2575406Z 2025-08-14T21:51:33.2575512Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2575724Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2575812Z return mod(**inputs) 2025-08-14T21:51:33.2576105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2576188Z outputs = self.mobilebert( 2025-08-14T21:51:33.2576477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2576560Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2576848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2576922Z layer_outputs = layer_module( 2025-08-14T21:51:33.2577235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2577403Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2577703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2577817Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2578110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2578202Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2578207Z 2025-08-14T21:51:33.2578314Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2578522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2578598Z return mod(**inputs) 2025-08-14T21:51:33.2578894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2578976Z outputs = self.mobilebert( 2025-08-14T21:51:33.2579268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2579344Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2579644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2579718Z layer_outputs = layer_module( 2025-08-14T21:51:33.2580015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2580271Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2580567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2580652Z self_outputs = self.self( 2025-08-14T21:51:33.2580972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.2581047Z self.value(value_tensor) 2025-08-14T21:51:33.2581051Z 2025-08-14T21:51:33.2581167Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2581373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2581451Z return mod(**inputs) 2025-08-14T21:51:33.2581742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2581818Z outputs = self.mobilebert( 2025-08-14T21:51:33.2582117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2582195Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2582488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2582594Z layer_outputs = layer_module( 2025-08-14T21:51:33.2582887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2583063Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2583355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.2583470Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.2583769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2583900Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2583904Z 2025-08-14T21:51:33.2584021Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2584228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2584298Z return mod(**inputs) 2025-08-14T21:51:33.2584596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2584668Z outputs = self.mobilebert( 2025-08-14T21:51:33.2584965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2585043Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2585334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2585415Z layer_outputs = layer_module( 2025-08-14T21:51:33.2585706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2585875Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2586177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2586291Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2586587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.2586679Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.2587002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2587106Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2587110Z 2025-08-14T21:51:33.2587219Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2587453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2587524Z return mod(**inputs) 2025-08-14T21:51:33.2587814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2587895Z outputs = self.mobilebert( 2025-08-14T21:51:33.2588185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2588262Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2588562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2588637Z layer_outputs = layer_module( 2025-08-14T21:51:33.2588937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2589028Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2589336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2589422Z self_outputs = self.self( 2025-08-14T21:51:33.2589714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.2589797Z self.query(query_tensor) 2025-08-14T21:51:33.2589800Z 2025-08-14T21:51:33.2589908Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2590117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2590192Z return mod(**inputs) 2025-08-14T21:51:33.2590503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2590577Z outputs = self.mobilebert( 2025-08-14T21:51:33.2590880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2590956Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2591260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2591335Z layer_outputs = layer_module( 2025-08-14T21:51:33.2591625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2591723Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2592020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2592103Z self_outputs = self.self( 2025-08-14T21:51:33.2592397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.2592469Z self.key(key_tensor) 2025-08-14T21:51:33.2592474Z 2025-08-14T21:51:33.2592569Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2592656Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2592765Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2592976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2593044Z return mod(**inputs) 2025-08-14T21:51:33.2593342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2593437Z outputs = self.mobilebert( 2025-08-14T21:51:33.2593734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2593820Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2594133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2594208Z layer_outputs = layer_module( 2025-08-14T21:51:33.2594510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2594598Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2594897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2595026Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2595343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.2595436Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2595439Z 2025-08-14T21:51:33.2595541Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2595763Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2595831Z return mod(**inputs) 2025-08-14T21:51:33.2596124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2596205Z outputs = self.mobilebert( 2025-08-14T21:51:33.2596496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2596585Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2596867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2596937Z layer_outputs = layer_module( 2025-08-14T21:51:33.2597232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2597316Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2597590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2597723Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2598012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.2598147Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2598446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2598543Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2598548Z 2025-08-14T21:51:33.2598667Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2598881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2598954Z return mod(**inputs) 2025-08-14T21:51:33.2599261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2599333Z outputs = self.mobilebert( 2025-08-14T21:51:33.2599640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2599718Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2600034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2600118Z layer_outputs = layer_module( 2025-08-14T21:51:33.2600426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2600550Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2600842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2600956Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2601255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2601341Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2601345Z 2025-08-14T21:51:33.2601460Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2601671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2601740Z return mod(**inputs) 2025-08-14T21:51:33.2602046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2602123Z outputs = self.mobilebert( 2025-08-14T21:51:33.2602442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2602530Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2602839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2602921Z layer_outputs = layer_module( 2025-08-14T21:51:33.2603210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2603309Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2603611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2603744Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2604048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2604164Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2604169Z 2025-08-14T21:51:33.2604275Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2604487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2604555Z return mod(**inputs) 2025-08-14T21:51:33.2604845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2604927Z outputs = self.mobilebert( 2025-08-14T21:51:33.2605219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2605306Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2605701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2605788Z layer_outputs = layer_module( 2025-08-14T21:51:33.2606103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2606203Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2606513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2606649Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2606975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2607077Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2607083Z 2025-08-14T21:51:33.2607192Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2607426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2607508Z return mod(**inputs) 2025-08-14T21:51:33.2607811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2607896Z outputs = self.mobilebert( 2025-08-14T21:51:33.2608198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2608274Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2608573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2608649Z layer_outputs = layer_module( 2025-08-14T21:51:33.2608948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2609048Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2609360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2609500Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2609795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2609923Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2610229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2610328Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2610332Z 2025-08-14T21:51:33.2610522Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2610738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2610810Z return mod(**inputs) 2025-08-14T21:51:33.2611122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2611198Z outputs = self.mobilebert( 2025-08-14T21:51:33.2611509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2611588Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2611903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2611985Z layer_outputs = layer_module( 2025-08-14T21:51:33.2612280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2612378Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2612681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2612795Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2613099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2613187Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2613190Z 2025-08-14T21:51:33.2613297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2613532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2613603Z return mod(**inputs) 2025-08-14T21:51:33.2613905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2613998Z outputs = self.mobilebert( 2025-08-14T21:51:33.2614297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2614380Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2614674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2614748Z layer_outputs = layer_module( 2025-08-14T21:51:33.2615050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2615148Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2615450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2615568Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2615862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2616002Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2616006Z 2025-08-14T21:51:33.2616113Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2616325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2616391Z return mod(**inputs) 2025-08-14T21:51:33.2616680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2616763Z outputs = self.mobilebert( 2025-08-14T21:51:33.2617053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2617148Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2617448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2617525Z layer_outputs = layer_module( 2025-08-14T21:51:33.2617821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2617918Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2618211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2618352Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2618662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2618759Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2618764Z 2025-08-14T21:51:33.2618873Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2619093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2619174Z return mod(**inputs) 2025-08-14T21:51:33.2619482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2619565Z outputs = self.mobilebert( 2025-08-14T21:51:33.2619871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2619949Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2620279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2620354Z layer_outputs = layer_module( 2025-08-14T21:51:33.2620652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2620782Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2621085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2621224Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2621524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2621654Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2621965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2622066Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2622071Z 2025-08-14T21:51:33.2622191Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2622403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2622475Z return mod(**inputs) 2025-08-14T21:51:33.2622807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2622885Z outputs = self.mobilebert( 2025-08-14T21:51:33.2623185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2623271Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2623571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2623657Z layer_outputs = layer_module( 2025-08-14T21:51:33.2623974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2624076Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2624390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2624508Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2624815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2624906Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2624910Z 2025-08-14T21:51:33.2625018Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2625241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2625311Z return mod(**inputs) 2025-08-14T21:51:33.2625616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2625700Z outputs = self.mobilebert( 2025-08-14T21:51:33.2626005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2626090Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2626394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2626470Z layer_outputs = layer_module( 2025-08-14T21:51:33.2626777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2626898Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2627210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2627329Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2627648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2627774Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2627778Z 2025-08-14T21:51:33.2627888Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2628100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2628179Z return mod(**inputs) 2025-08-14T21:51:33.2628476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2628561Z outputs = self.mobilebert( 2025-08-14T21:51:33.2628858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2628938Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2629246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2629341Z layer_outputs = layer_module( 2025-08-14T21:51:33.2629651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2629750Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2630050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2630189Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2630494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2630583Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2630594Z 2025-08-14T21:51:33.2630723Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2630936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2631018Z return mod(**inputs) 2025-08-14T21:51:33.2631319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2631398Z outputs = self.mobilebert( 2025-08-14T21:51:33.2631706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2631788Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2632099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2632178Z layer_outputs = layer_module( 2025-08-14T21:51:33.2632478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2632587Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2632896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2633030Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2633343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2633473Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2633796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2633893Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2633897Z 2025-08-14T21:51:33.2634007Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2634252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2634340Z return mod(**inputs) 2025-08-14T21:51:33.2634646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2634721Z outputs = self.mobilebert( 2025-08-14T21:51:33.2635029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2635112Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2635423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2635506Z layer_outputs = layer_module( 2025-08-14T21:51:33.2635819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2635947Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2636283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2636376Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2636380Z 2025-08-14T21:51:33.2636489Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2636732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2636803Z return mod(**inputs) 2025-08-14T21:51:33.2637110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2637187Z outputs = self.mobilebert( 2025-08-14T21:51:33.2637515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2637767Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2638109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2638196Z layer_outputs = layer_module( 2025-08-14T21:51:33.2638550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2638676Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2638998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2639118Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2639122Z 2025-08-14T21:51:33.2639233Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2639474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2639548Z return mod(**inputs) 2025-08-14T21:51:33.2639869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2639946Z outputs = self.mobilebert( 2025-08-14T21:51:33.2640253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2640343Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2640697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2640841Z layer_outputs = layer_module( 2025-08-14T21:51:33.2641202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2641376Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2641701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.2641834Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.2641839Z 2025-08-14T21:51:33.2641951Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2642173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2642243Z return mod(**inputs) 2025-08-14T21:51:33.2642550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2642635Z outputs = self.mobilebert( 2025-08-14T21:51:33.2642935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2643025Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2643328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2643443Z layer_outputs = layer_module( 2025-08-14T21:51:33.2643746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2643915Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2644219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.2644347Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.2644648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2644759Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2644790Z 2025-08-14T21:51:33.2644903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2645125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2645196Z return mod(**inputs) 2025-08-14T21:51:33.2645567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2645658Z outputs = self.mobilebert( 2025-08-14T21:51:33.2645960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2646046Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2646348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2646424Z layer_outputs = layer_module( 2025-08-14T21:51:33.2646735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2646903Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2647216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2647349Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2647650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.2647749Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2647777Z 2025-08-14T21:51:33.2647890Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2648101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2648180Z return mod(**inputs) 2025-08-14T21:51:33.2648478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2648578Z outputs = self.mobilebert( 2025-08-14T21:51:33.2648886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2648964Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2649276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2649354Z layer_outputs = layer_module( 2025-08-14T21:51:33.2649668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2649838Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2650148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2650291Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2650625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.2650757Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2651062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2651160Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2651166Z 2025-08-14T21:51:33.2651283Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2651492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2651563Z return mod(**inputs) 2025-08-14T21:51:33.2651885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2651963Z outputs = self.mobilebert( 2025-08-14T21:51:33.2652271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2652348Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2652645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2652729Z layer_outputs = layer_module( 2025-08-14T21:51:33.2653036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2653202Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2653501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2653617Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2653918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2654005Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2654009Z 2025-08-14T21:51:33.2654115Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2654329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2654398Z return mod(**inputs) 2025-08-14T21:51:33.2654711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2654784Z outputs = self.mobilebert( 2025-08-14T21:51:33.2655075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2655179Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2655476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2655560Z layer_outputs = layer_module( 2025-08-14T21:51:33.2655860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2655950Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2656264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2656341Z self_outputs = self.self( 2025-08-14T21:51:33.2656634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.2656719Z self.value(value_tensor) 2025-08-14T21:51:33.2656722Z 2025-08-14T21:51:33.2656828Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2657059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2657132Z return mod(**inputs) 2025-08-14T21:51:33.2657422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2657506Z outputs = self.mobilebert( 2025-08-14T21:51:33.2657799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2657879Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2658176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2658250Z layer_outputs = layer_module( 2025-08-14T21:51:33.2658566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2658735Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2659029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.2659153Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.2659455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2659550Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2659556Z 2025-08-14T21:51:33.2659663Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2659877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2659959Z return mod(**inputs) 2025-08-14T21:51:33.2660258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2660344Z outputs = self.mobilebert( 2025-08-14T21:51:33.2660645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2660722Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2661029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2661104Z layer_outputs = layer_module( 2025-08-14T21:51:33.2661423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2661600Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2661909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2662057Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2662346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.2662437Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.2662732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2662827Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2662832Z 2025-08-14T21:51:33.2662946Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2663157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2663228Z return mod(**inputs) 2025-08-14T21:51:33.2663533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2663611Z outputs = self.mobilebert( 2025-08-14T21:51:33.2663928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2664015Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2664313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2664397Z layer_outputs = layer_module( 2025-08-14T21:51:33.2664693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2664786Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2665111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2665187Z self_outputs = self.self( 2025-08-14T21:51:33.2665492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.2665567Z self.query(query_tensor) 2025-08-14T21:51:33.2665570Z 2025-08-14T21:51:33.2665676Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2665889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2665958Z return mod(**inputs) 2025-08-14T21:51:33.2666245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2666327Z outputs = self.mobilebert( 2025-08-14T21:51:33.2666616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2666701Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2666992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2667067Z layer_outputs = layer_module( 2025-08-14T21:51:33.2667364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2667450Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2667751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2667823Z self_outputs = self.self( 2025-08-14T21:51:33.2668131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.2668208Z self.key(key_tensor) 2025-08-14T21:51:33.2668212Z 2025-08-14T21:51:33.2668299Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2668381Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2668515Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2668730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2668807Z return mod(**inputs) 2025-08-14T21:51:33.2669104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2669179Z outputs = self.mobilebert( 2025-08-14T21:51:33.2669485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2669565Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2669864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2669948Z layer_outputs = layer_module( 2025-08-14T21:51:33.2670253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2670369Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2670674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2670805Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2671113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.2671203Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2671208Z 2025-08-14T21:51:33.2671322Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2671534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2671622Z return mod(**inputs) 2025-08-14T21:51:33.2671933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2672011Z outputs = self.mobilebert( 2025-08-14T21:51:33.2672317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2672403Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2672702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2672785Z layer_outputs = layer_module( 2025-08-14T21:51:33.2673089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2673179Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2673489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2673621Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2673932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.2674067Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2674370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2674475Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2674479Z 2025-08-14T21:51:33.2674610Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2674841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2674918Z return mod(**inputs) 2025-08-14T21:51:33.2675222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2675328Z outputs = self.mobilebert( 2025-08-14T21:51:33.2675631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2675709Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2676014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2676090Z layer_outputs = layer_module( 2025-08-14T21:51:33.2676406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2676510Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2676819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2676945Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2677276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2677369Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2677380Z 2025-08-14T21:51:33.2677491Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2677729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2677811Z return mod(**inputs) 2025-08-14T21:51:33.2678127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2678206Z outputs = self.mobilebert( 2025-08-14T21:51:33.2678515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2678614Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2678921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2678999Z layer_outputs = layer_module( 2025-08-14T21:51:33.2679296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2679405Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2679715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2679837Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2680154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2680274Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2680278Z 2025-08-14T21:51:33.2680397Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2680631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2680703Z return mod(**inputs) 2025-08-14T21:51:33.2681021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2681096Z outputs = self.mobilebert( 2025-08-14T21:51:33.2681400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2681499Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2681847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2681931Z layer_outputs = layer_module( 2025-08-14T21:51:33.2682241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2682361Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2682681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2682813Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2683133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2683222Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2683228Z 2025-08-14T21:51:33.2683337Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2683553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2683623Z return mod(**inputs) 2025-08-14T21:51:33.2683929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2684005Z outputs = self.mobilebert( 2025-08-14T21:51:33.2684323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2684409Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2684710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2684793Z layer_outputs = layer_module( 2025-08-14T21:51:33.2685093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2685193Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2685612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2685755Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2686058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2686194Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2686494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2686599Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2686603Z 2025-08-14T21:51:33.2686715Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2686926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2687007Z return mod(**inputs) 2025-08-14T21:51:33.2687310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2687394Z outputs = self.mobilebert( 2025-08-14T21:51:33.2687693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2687772Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2688078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2688157Z layer_outputs = layer_module( 2025-08-14T21:51:33.2688453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2688588Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2688890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2689017Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2689339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2689429Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2689433Z 2025-08-14T21:51:33.2689552Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2689762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2689841Z return mod(**inputs) 2025-08-14T21:51:33.2690138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2690217Z outputs = self.mobilebert( 2025-08-14T21:51:33.2690521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2690599Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2690899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2691035Z layer_outputs = layer_module( 2025-08-14T21:51:33.2691336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2691442Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2691743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2691863Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2692171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2692309Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2692313Z 2025-08-14T21:51:33.2692433Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2692650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2692721Z return mod(**inputs) 2025-08-14T21:51:33.2693029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2693104Z outputs = self.mobilebert( 2025-08-14T21:51:33.2693404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2693489Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2693788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2693870Z layer_outputs = layer_module( 2025-08-14T21:51:33.2694173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2694274Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2694580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2694712Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2695020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2695112Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2695138Z 2025-08-14T21:51:33.2695248Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2695470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2695542Z return mod(**inputs) 2025-08-14T21:51:33.2695842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2695946Z outputs = self.mobilebert( 2025-08-14T21:51:33.2696255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2696340Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2696652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2696728Z layer_outputs = layer_module( 2025-08-14T21:51:33.2697043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2697142Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2697457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2697589Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2697914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2698052Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2698352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2698457Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2698461Z 2025-08-14T21:51:33.2698569Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2698785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2698863Z return mod(**inputs) 2025-08-14T21:51:33.2699185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2699267Z outputs = self.mobilebert( 2025-08-14T21:51:33.2699581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2699660Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2699972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2700049Z layer_outputs = layer_module( 2025-08-14T21:51:33.2700355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2700464Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2700766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2700903Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2701200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2701287Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2701291Z 2025-08-14T21:51:33.2701407Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2701612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2701679Z return mod(**inputs) 2025-08-14T21:51:33.2701977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2702069Z outputs = self.mobilebert( 2025-08-14T21:51:33.2702368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2702445Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2703403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2703490Z layer_outputs = layer_module( 2025-08-14T21:51:33.2703777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2703880Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2704170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2704284Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2704580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2704692Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2704697Z 2025-08-14T21:51:33.2704804Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2705035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2705108Z return mod(**inputs) 2025-08-14T21:51:33.2705408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2705481Z outputs = self.mobilebert( 2025-08-14T21:51:33.2705770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2705855Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2706145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2706228Z layer_outputs = layer_module( 2025-08-14T21:51:33.2706533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2706633Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2706929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2707054Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2707345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2707441Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2707447Z 2025-08-14T21:51:33.2707554Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2707769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2707838Z return mod(**inputs) 2025-08-14T21:51:33.2708132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2708215Z outputs = self.mobilebert( 2025-08-14T21:51:33.2708509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2708594Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2708893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2708969Z layer_outputs = layer_module( 2025-08-14T21:51:33.2709277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2709395Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2709697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2709834Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2710166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2710297Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2710592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2710688Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2710692Z 2025-08-14T21:51:33.2710808Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2711013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2711088Z return mod(**inputs) 2025-08-14T21:51:33.2711382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2711457Z outputs = self.mobilebert( 2025-08-14T21:51:33.2711804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2711883Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2712176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2712258Z layer_outputs = layer_module( 2025-08-14T21:51:33.2712547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2712682Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2712973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2713089Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2713095Z 2025-08-14T21:51:33.2713220Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2713425Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2713503Z return mod(**inputs) 2025-08-14T21:51:33.2713796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2713868Z outputs = self.mobilebert( 2025-08-14T21:51:33.2714168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2714248Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2714538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2714623Z layer_outputs = layer_module( 2025-08-14T21:51:33.2714916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2715050Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2715354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2715470Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2715474Z 2025-08-14T21:51:33.2715590Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2715814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2715912Z return mod(**inputs) 2025-08-14T21:51:33.2716211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2716288Z outputs = self.mobilebert( 2025-08-14T21:51:33.2716594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2716691Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2717002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2717078Z layer_outputs = layer_module( 2025-08-14T21:51:33.2717374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2717549Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2717853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.2717954Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.2717967Z 2025-08-14T21:51:33.2718075Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2718289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2718382Z return mod(**inputs) 2025-08-14T21:51:33.2718687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2718761Z outputs = self.mobilebert( 2025-08-14T21:51:33.2719065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2719142Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2719452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2719526Z layer_outputs = layer_module( 2025-08-14T21:51:33.2719844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2720031Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2720331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.2720461Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.2720767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2720865Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2720870Z 2025-08-14T21:51:33.2720985Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2721208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2721277Z return mod(**inputs) 2025-08-14T21:51:33.2721586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2721664Z outputs = self.mobilebert( 2025-08-14T21:51:33.2721968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2722045Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2722344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2722425Z layer_outputs = layer_module( 2025-08-14T21:51:33.2722725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2722911Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2723220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2723369Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2723674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.2723764Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2723769Z 2025-08-14T21:51:33.2723877Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2724108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2724178Z return mod(**inputs) 2025-08-14T21:51:33.2724485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2724559Z outputs = self.mobilebert( 2025-08-14T21:51:33.2724858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2724946Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2725266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2725352Z layer_outputs = layer_module( 2025-08-14T21:51:33.2725736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2725908Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2726213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2726346Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2726668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.2726809Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2727107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2727213Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2727217Z 2025-08-14T21:51:33.2727328Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2727541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2727622Z return mod(**inputs) 2025-08-14T21:51:33.2727923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2728009Z outputs = self.mobilebert( 2025-08-14T21:51:33.2728306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2728386Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2728692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2728770Z layer_outputs = layer_module( 2025-08-14T21:51:33.2729067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2729247Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2729543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2729692Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2729989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2730077Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2730100Z 2025-08-14T21:51:33.2730220Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2730436Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2730512Z return mod(**inputs) 2025-08-14T21:51:33.2730815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2730891Z outputs = self.mobilebert( 2025-08-14T21:51:33.2731198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2731277Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2731577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2731661Z layer_outputs = layer_module( 2025-08-14T21:51:33.2731964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2732079Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2732380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2732457Z self_outputs = self.self( 2025-08-14T21:51:33.2732774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.2732851Z self.value(value_tensor) 2025-08-14T21:51:33.2732855Z 2025-08-14T21:51:33.2732969Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2733179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2733269Z return mod(**inputs) 2025-08-14T21:51:33.2733578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2733655Z outputs = self.mobilebert( 2025-08-14T21:51:33.2733954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2734041Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2734354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2734436Z layer_outputs = layer_module( 2025-08-14T21:51:33.2734739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2734910Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2735223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.2735342Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.2735653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2735741Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2735745Z 2025-08-14T21:51:33.2735853Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2736076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2736146Z return mod(**inputs) 2025-08-14T21:51:33.2736478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2736553Z outputs = self.mobilebert( 2025-08-14T21:51:33.2736853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2736958Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2737263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2737339Z layer_outputs = layer_module( 2025-08-14T21:51:33.2737886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2741418Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2743014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2743146Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2743468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.2743567Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.2743881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2743980Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2743985Z 2025-08-14T21:51:33.2744096Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2744321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2744430Z return mod(**inputs) 2025-08-14T21:51:33.2744752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2744831Z outputs = self.mobilebert( 2025-08-14T21:51:33.2745136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2745217Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2745516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2745598Z layer_outputs = layer_module( 2025-08-14T21:51:33.2745897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2745997Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2746300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2746377Z self_outputs = self.self( 2025-08-14T21:51:33.2746683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.2746759Z self.query(query_tensor) 2025-08-14T21:51:33.2746763Z 2025-08-14T21:51:33.2746872Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2747097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2747167Z return mod(**inputs) 2025-08-14T21:51:33.2747472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2747548Z outputs = self.mobilebert( 2025-08-14T21:51:33.2747849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2747978Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2748283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2748369Z layer_outputs = layer_module( 2025-08-14T21:51:33.2748676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2748800Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2749111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2749186Z self_outputs = self.self( 2025-08-14T21:51:33.2749494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.2749575Z self.key(key_tensor) 2025-08-14T21:51:33.2749579Z 2025-08-14T21:51:33.2749725Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2749823Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2749961Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2750176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2750255Z return mod(**inputs) 2025-08-14T21:51:33.2750557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2750631Z outputs = self.mobilebert( 2025-08-14T21:51:33.2750937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2751014Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2751322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2751399Z layer_outputs = layer_module( 2025-08-14T21:51:33.2751699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2751797Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2752106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2752248Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2752548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.2752640Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2752644Z 2025-08-14T21:51:33.2752761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2752977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2753047Z return mod(**inputs) 2025-08-14T21:51:33.2753357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2753431Z outputs = self.mobilebert( 2025-08-14T21:51:33.2753735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2753813Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2754113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2754197Z layer_outputs = layer_module( 2025-08-14T21:51:33.2754493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2754590Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2754901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2755052Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2755361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.2755492Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2755813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2755911Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2755914Z 2025-08-14T21:51:33.2756019Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2756239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2756307Z return mod(**inputs) 2025-08-14T21:51:33.2756630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2756738Z outputs = self.mobilebert( 2025-08-14T21:51:33.2757033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2757117Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2757406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2757479Z layer_outputs = layer_module( 2025-08-14T21:51:33.2757775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2757873Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2758170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2758288Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2758577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2758672Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2758676Z 2025-08-14T21:51:33.2758784Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2758993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2759070Z return mod(**inputs) 2025-08-14T21:51:33.2759358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2759437Z outputs = self.mobilebert( 2025-08-14T21:51:33.2759729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2759806Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2760106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2760182Z layer_outputs = layer_module( 2025-08-14T21:51:33.2760479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2760580Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2760871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2760991Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2761279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2761394Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2761429Z 2025-08-14T21:51:33.2761539Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2761741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2761814Z return mod(**inputs) 2025-08-14T21:51:33.2762125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2762218Z outputs = self.mobilebert( 2025-08-14T21:51:33.2762526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2762603Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2762912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2763008Z layer_outputs = layer_module( 2025-08-14T21:51:33.2763329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2763440Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2763742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2763877Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2764183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2764273Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2764277Z 2025-08-14T21:51:33.2764392Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2764614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2764685Z return mod(**inputs) 2025-08-14T21:51:33.2764995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2765070Z outputs = self.mobilebert( 2025-08-14T21:51:33.2765380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2765538Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2765850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2765934Z layer_outputs = layer_module( 2025-08-14T21:51:33.2766238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2766340Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2766654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2766790Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2767101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2767231Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2767534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2767638Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2767642Z 2025-08-14T21:51:33.2767750Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2767966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2768039Z return mod(**inputs) 2025-08-14T21:51:33.2768340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2768456Z outputs = self.mobilebert( 2025-08-14T21:51:33.2768757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2768836Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2769173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2769250Z layer_outputs = layer_module( 2025-08-14T21:51:33.2769553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2769653Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2769975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2770103Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2770423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2770535Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2770539Z 2025-08-14T21:51:33.2770648Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2770858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2770935Z return mod(**inputs) 2025-08-14T21:51:33.2771223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2771295Z outputs = self.mobilebert( 2025-08-14T21:51:33.2771589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2771666Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2771974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2772051Z layer_outputs = layer_module( 2025-08-14T21:51:33.2772348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2772459Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2772757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2772879Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2773176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2773295Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2773300Z 2025-08-14T21:51:33.2773419Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2773629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2773709Z return mod(**inputs) 2025-08-14T21:51:33.2774010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2774086Z outputs = self.mobilebert( 2025-08-14T21:51:33.2774393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2774470Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2774768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2774853Z layer_outputs = layer_module( 2025-08-14T21:51:33.2775175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2775283Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2775586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2775740Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2776048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2776140Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2776143Z 2025-08-14T21:51:33.2776261Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2776474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2776564Z return mod(**inputs) 2025-08-14T21:51:33.2776892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2776969Z outputs = self.mobilebert( 2025-08-14T21:51:33.2777268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2777357Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2777658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2777742Z layer_outputs = layer_module( 2025-08-14T21:51:33.2778043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2778141Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2778449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2778582Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2778888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2779018Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2779319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2779422Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2779426Z 2025-08-14T21:51:33.2779534Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2779744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2779820Z return mod(**inputs) 2025-08-14T21:51:33.2780119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2780203Z outputs = self.mobilebert( 2025-08-14T21:51:33.2780507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2780586Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2780898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2780973Z layer_outputs = layer_module( 2025-08-14T21:51:33.2781280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2781379Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2781679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2781828Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2782129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2782220Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2782230Z 2025-08-14T21:51:33.2782359Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2782575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2782653Z return mod(**inputs) 2025-08-14T21:51:33.2782953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2783028Z outputs = self.mobilebert( 2025-08-14T21:51:33.2783355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2783437Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2783760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2783840Z layer_outputs = layer_module( 2025-08-14T21:51:33.2784143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2784254Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2784554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2784672Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2784984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2785104Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2785110Z 2025-08-14T21:51:33.2785228Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2785443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2785514Z return mod(**inputs) 2025-08-14T21:51:33.2785823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2785901Z outputs = self.mobilebert( 2025-08-14T21:51:33.2786205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2786282Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2786585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2786669Z layer_outputs = layer_module( 2025-08-14T21:51:33.2786968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2787072Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2787377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2787536Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2787837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2787936Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2787940Z 2025-08-14T21:51:33.2788048Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2788262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2788342Z return mod(**inputs) 2025-08-14T21:51:33.2788665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2788739Z outputs = self.mobilebert( 2025-08-14T21:51:33.2789033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2789128Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2789431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2789507Z layer_outputs = layer_module( 2025-08-14T21:51:33.2789804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2789909Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2790228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2790389Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2790691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2790820Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2791131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2791227Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2791231Z 2025-08-14T21:51:33.2791338Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2791552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2791622Z return mod(**inputs) 2025-08-14T21:51:33.2791920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2791995Z outputs = self.mobilebert( 2025-08-14T21:51:33.2792288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2792372Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2792663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2792745Z layer_outputs = layer_module( 2025-08-14T21:51:33.2793034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2793159Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2793458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2793548Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2793551Z 2025-08-14T21:51:33.2793667Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2793873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2793941Z return mod(**inputs) 2025-08-14T21:51:33.2794241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2794312Z outputs = self.mobilebert( 2025-08-14T21:51:33.2794602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2794683Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2794973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2795053Z layer_outputs = layer_module( 2025-08-14T21:51:33.2795377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2795499Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2795791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2795926Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2795930Z 2025-08-14T21:51:33.2796046Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2796250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2796317Z return mod(**inputs) 2025-08-14T21:51:33.2796636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2796712Z outputs = self.mobilebert( 2025-08-14T21:51:33.2797041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2797127Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2797432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2797515Z layer_outputs = layer_module( 2025-08-14T21:51:33.2797818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2797983Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2798288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.2798392Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.2798398Z 2025-08-14T21:51:33.2798515Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2798740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2798810Z return mod(**inputs) 2025-08-14T21:51:33.2799120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2799197Z outputs = self.mobilebert( 2025-08-14T21:51:33.2799499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2799587Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2799889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2799971Z layer_outputs = layer_module( 2025-08-14T21:51:33.2800275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2800448Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2800759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.2800892Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.2801203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2801302Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2801306Z 2025-08-14T21:51:33.2801417Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2801644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2801717Z return mod(**inputs) 2025-08-14T21:51:33.2802038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2802121Z outputs = self.mobilebert( 2025-08-14T21:51:33.2802415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2802553Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2802853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2802931Z layer_outputs = layer_module( 2025-08-14T21:51:33.2803238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2803406Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2803730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2803883Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2804183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.2804284Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2804288Z 2025-08-14T21:51:33.2804396Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2804609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2804687Z return mod(**inputs) 2025-08-14T21:51:33.2804988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2805073Z outputs = self.mobilebert( 2025-08-14T21:51:33.2805373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2805532Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2805858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2805936Z layer_outputs = layer_module( 2025-08-14T21:51:33.2806245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2806415Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2806711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2806849Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2807148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.2807289Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2807590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2807688Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2807694Z 2025-08-14T21:51:33.2807812Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2808025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2808096Z return mod(**inputs) 2025-08-14T21:51:33.2808403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2808477Z outputs = self.mobilebert( 2025-08-14T21:51:33.2808784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2808895Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2809200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2809286Z layer_outputs = layer_module( 2025-08-14T21:51:33.2809607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2809788Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2810098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2810216Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2810549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2810642Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2810663Z 2025-08-14T21:51:33.2810775Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2810996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2811068Z return mod(**inputs) 2025-08-14T21:51:33.2811376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2811454Z outputs = self.mobilebert( 2025-08-14T21:51:33.2811755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2811839Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2812141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2812227Z layer_outputs = layer_module( 2025-08-14T21:51:33.2812528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2812621Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2812930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2813010Z self_outputs = self.self( 2025-08-14T21:51:33.2813312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.2813395Z self.value(value_tensor) 2025-08-14T21:51:33.2813399Z 2025-08-14T21:51:33.2813508Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2813728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2813799Z return mod(**inputs) 2025-08-14T21:51:33.2814102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2814184Z outputs = self.mobilebert( 2025-08-14T21:51:33.2814484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2814570Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2814870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2814945Z layer_outputs = layer_module( 2025-08-14T21:51:33.2815254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2815425Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2815732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.2815882Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.2816184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2816300Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2816304Z 2025-08-14T21:51:33.2816413Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2816623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2816702Z return mod(**inputs) 2025-08-14T21:51:33.2816997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2817078Z outputs = self.mobilebert( 2025-08-14T21:51:33.2817394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2817501Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2817810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2817887Z layer_outputs = layer_module( 2025-08-14T21:51:33.2818186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2818364Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2818665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2818792Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2819091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.2819188Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.2819497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2819594Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2819600Z 2025-08-14T21:51:33.2819719Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2819933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2820003Z return mod(**inputs) 2025-08-14T21:51:33.2820309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2820386Z outputs = self.mobilebert( 2025-08-14T21:51:33.2820687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2820774Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2821073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2821154Z layer_outputs = layer_module( 2025-08-14T21:51:33.2821456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2821548Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2821860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2821936Z self_outputs = self.self( 2025-08-14T21:51:33.2822241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.2822317Z self.query(query_tensor) 2025-08-14T21:51:33.2822347Z 2025-08-14T21:51:33.2822459Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2822680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2822750Z return mod(**inputs) 2025-08-14T21:51:33.2823049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2823150Z outputs = self.mobilebert( 2025-08-14T21:51:33.2823454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2823539Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2823839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2823934Z layer_outputs = layer_module( 2025-08-14T21:51:33.2824262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2824354Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2824658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2824734Z self_outputs = self.self( 2025-08-14T21:51:33.2825034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.2825111Z self.key(key_tensor) 2025-08-14T21:51:33.2825115Z 2025-08-14T21:51:33.2825206Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2825292Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2825408Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2825622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2825703Z return mod(**inputs) 2025-08-14T21:51:33.2826006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2826082Z outputs = self.mobilebert( 2025-08-14T21:51:33.2826389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2826469Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2826770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2826853Z layer_outputs = layer_module( 2025-08-14T21:51:33.2827152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2827248Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2827547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2827683Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2827992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.2828084Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2828088Z 2025-08-14T21:51:33.2828206Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2828418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2828487Z return mod(**inputs) 2025-08-14T21:51:33.2828793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2828867Z outputs = self.mobilebert( 2025-08-14T21:51:33.2829167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2829279Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2829579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2829661Z layer_outputs = layer_module( 2025-08-14T21:51:33.2829984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2830070Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2830379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2830509Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2830834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.2830989Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2831289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2831392Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2831398Z 2025-08-14T21:51:33.2831504Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2831720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2831790Z return mod(**inputs) 2025-08-14T21:51:33.2832101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2832187Z outputs = self.mobilebert( 2025-08-14T21:51:33.2832487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2832564Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2832869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2832945Z layer_outputs = layer_module( 2025-08-14T21:51:33.2833251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2833352Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2833648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2833775Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2834072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2834170Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2834173Z 2025-08-14T21:51:33.2834284Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2834493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2834572Z return mod(**inputs) 2025-08-14T21:51:33.2834871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2834947Z outputs = self.mobilebert( 2025-08-14T21:51:33.2835249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2835326Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2835630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2835707Z layer_outputs = layer_module( 2025-08-14T21:51:33.2836034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2836144Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2836448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2836593Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2836889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2837007Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2837011Z 2025-08-14T21:51:33.2837128Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2837356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2837431Z return mod(**inputs) 2025-08-14T21:51:33.2837961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2838045Z outputs = self.mobilebert( 2025-08-14T21:51:33.2838355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2838437Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2838741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2838828Z layer_outputs = layer_module( 2025-08-14T21:51:33.2839125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2839235Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2839535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2839672Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2839977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2840067Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2840072Z 2025-08-14T21:51:33.2840180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2840401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2840470Z return mod(**inputs) 2025-08-14T21:51:33.2840770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2840845Z outputs = self.mobilebert( 2025-08-14T21:51:33.2841143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2841234Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2841530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2841615Z layer_outputs = layer_module( 2025-08-14T21:51:33.2841914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2842015Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2842321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2842451Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2842750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2842926Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2843227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2843332Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2843364Z 2025-08-14T21:51:33.2843475Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2843700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2843781Z return mod(**inputs) 2025-08-14T21:51:33.2844080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2844161Z outputs = self.mobilebert( 2025-08-14T21:51:33.2844499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2844581Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2844915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2844993Z layer_outputs = layer_module( 2025-08-14T21:51:33.2845297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2845405Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2845759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2845892Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2846191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2846279Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2846285Z 2025-08-14T21:51:33.2846405Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2846618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2846695Z return mod(**inputs) 2025-08-14T21:51:33.2846993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2847069Z outputs = self.mobilebert( 2025-08-14T21:51:33.2847371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2847448Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2847744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2847831Z layer_outputs = layer_module( 2025-08-14T21:51:33.2848132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2848238Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2848534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2848655Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2848963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2849080Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2849085Z 2025-08-14T21:51:33.2849202Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2849412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2849485Z return mod(**inputs) 2025-08-14T21:51:33.2849826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2849901Z outputs = self.mobilebert( 2025-08-14T21:51:33.2850204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2850305Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2850605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2850688Z layer_outputs = layer_module( 2025-08-14T21:51:33.2850989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2851088Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2851413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2851564Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2851878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2851965Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2851971Z 2025-08-14T21:51:33.2852076Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2852300Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2852371Z return mod(**inputs) 2025-08-14T21:51:33.2852673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2852749Z outputs = self.mobilebert( 2025-08-14T21:51:33.2853050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2853138Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2853436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2853510Z layer_outputs = layer_module( 2025-08-14T21:51:33.2853820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2853919Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2854226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2854357Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2854659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2854802Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2855104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2855209Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2855215Z 2025-08-14T21:51:33.2855325Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2855535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2855614Z return mod(**inputs) 2025-08-14T21:51:33.2855913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2855988Z outputs = self.mobilebert( 2025-08-14T21:51:33.2856294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2856399Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2856711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2856788Z layer_outputs = layer_module( 2025-08-14T21:51:33.2857088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2857214Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2857515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2857652Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2857950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2858081Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2858087Z 2025-08-14T21:51:33.2858236Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2858451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2858523Z return mod(**inputs) 2025-08-14T21:51:33.2858830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2858905Z outputs = self.mobilebert( 2025-08-14T21:51:33.2859213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2859292Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2859594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2859679Z layer_outputs = layer_module( 2025-08-14T21:51:33.2859979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2860086Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2860384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2860502Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2860811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2860926Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2860930Z 2025-08-14T21:51:33.2861045Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2861257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2861328Z return mod(**inputs) 2025-08-14T21:51:33.2861638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2861714Z outputs = self.mobilebert( 2025-08-14T21:51:33.2862012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2862100Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2862399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2862482Z layer_outputs = layer_module( 2025-08-14T21:51:33.2862778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2862879Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2863189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2863350Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2863654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2863742Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2863765Z 2025-08-14T21:51:33.2863877Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2864095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2864165Z return mod(**inputs) 2025-08-14T21:51:33.2864464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2864547Z outputs = self.mobilebert( 2025-08-14T21:51:33.2864863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2864968Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2865273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2865349Z layer_outputs = layer_module( 2025-08-14T21:51:33.2865656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2865754Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2866059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2866187Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2866488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2866625Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2866924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2867020Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2867033Z 2025-08-14T21:51:33.2867142Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2867356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2867435Z return mod(**inputs) 2025-08-14T21:51:33.2867734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2867810Z outputs = self.mobilebert( 2025-08-14T21:51:33.2868118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2868197Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2868502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2868577Z layer_outputs = layer_module( 2025-08-14T21:51:33.2868877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2869015Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2869317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2869405Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2869417Z 2025-08-14T21:51:33.2869527Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2869741Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2869852Z return mod(**inputs) 2025-08-14T21:51:33.2870157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2870233Z outputs = self.mobilebert( 2025-08-14T21:51:33.2870549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2870657Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2870962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2871038Z layer_outputs = layer_module( 2025-08-14T21:51:33.2871334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2871489Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2871805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2871928Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2871940Z 2025-08-14T21:51:33.2872051Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2872267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2872348Z return mod(**inputs) 2025-08-14T21:51:33.2872647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2872723Z outputs = self.mobilebert( 2025-08-14T21:51:33.2873031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2873109Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2873417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2873496Z layer_outputs = layer_module( 2025-08-14T21:51:33.2873796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2873974Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2874274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.2874377Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.2874391Z 2025-08-14T21:51:33.2874500Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2874713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2874793Z return mod(**inputs) 2025-08-14T21:51:33.2875098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2875174Z outputs = self.mobilebert( 2025-08-14T21:51:33.2875481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2875561Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2875869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2875947Z layer_outputs = layer_module( 2025-08-14T21:51:33.2876247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2876424Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2876727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.2876899Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.2877200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2877297Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2877323Z 2025-08-14T21:51:33.2877441Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2877652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2877723Z return mod(**inputs) 2025-08-14T21:51:33.2878026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2878100Z outputs = self.mobilebert( 2025-08-14T21:51:33.2878421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2878517Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2878816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2878901Z layer_outputs = layer_module( 2025-08-14T21:51:33.2879199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2879372Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2879670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2879812Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2880162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.2880251Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2880258Z 2025-08-14T21:51:33.2880364Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2880583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2880654Z return mod(**inputs) 2025-08-14T21:51:33.2880967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2881041Z outputs = self.mobilebert( 2025-08-14T21:51:33.2881343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2881427Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2881733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2881815Z layer_outputs = layer_module( 2025-08-14T21:51:33.2882123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2882289Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2882602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2882730Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2883036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.2883170Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2883475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2883605Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2883610Z 2025-08-14T21:51:33.2883721Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2883933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2884038Z return mod(**inputs) 2025-08-14T21:51:33.2884341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2884423Z outputs = self.mobilebert( 2025-08-14T21:51:33.2884723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2884801Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2885125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2885207Z layer_outputs = layer_module( 2025-08-14T21:51:33.2885629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2885810Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2886115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2886244Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2886549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2886640Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2886653Z 2025-08-14T21:51:33.2886762Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2886988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2887070Z return mod(**inputs) 2025-08-14T21:51:33.2887380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2887456Z outputs = self.mobilebert( 2025-08-14T21:51:33.2887761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2887841Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2888154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2888232Z layer_outputs = layer_module( 2025-08-14T21:51:33.2888535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2888635Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2888941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2889017Z self_outputs = self.self( 2025-08-14T21:51:33.2889333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.2889409Z self.value(value_tensor) 2025-08-14T21:51:33.2889413Z 2025-08-14T21:51:33.2889526Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2889731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2889799Z return mod(**inputs) 2025-08-14T21:51:33.2890102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2890175Z outputs = self.mobilebert( 2025-08-14T21:51:33.2890478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2890584Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2890872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2890954Z layer_outputs = layer_module( 2025-08-14T21:51:33.2891262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2891425Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2891724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.2891838Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.2892153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2892258Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2892263Z 2025-08-14T21:51:33.2892371Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2892587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2892656Z return mod(**inputs) 2025-08-14T21:51:33.2892952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2893026Z outputs = self.mobilebert( 2025-08-14T21:51:33.2893314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2893397Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2893692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2893768Z layer_outputs = layer_module( 2025-08-14T21:51:33.2894071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2894236Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2894539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2894655Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2894958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.2895059Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.2895360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2895466Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2895471Z 2025-08-14T21:51:33.2895582Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2895794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2895873Z return mod(**inputs) 2025-08-14T21:51:33.2896180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2896253Z outputs = self.mobilebert( 2025-08-14T21:51:33.2896553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2896628Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2896927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2897043Z layer_outputs = layer_module( 2025-08-14T21:51:33.2897339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2897435Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2897725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2897828Z self_outputs = self.self( 2025-08-14T21:51:33.2898116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.2898190Z self.query(query_tensor) 2025-08-14T21:51:33.2898194Z 2025-08-14T21:51:33.2898308Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2898512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2898603Z return mod(**inputs) 2025-08-14T21:51:33.2898919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2898993Z outputs = self.mobilebert( 2025-08-14T21:51:33.2899289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2899365Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2899652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2899733Z layer_outputs = layer_module( 2025-08-14T21:51:33.2900024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2900119Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2900409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2900483Z self_outputs = self.self( 2025-08-14T21:51:33.2900778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.2900846Z self.key(key_tensor) 2025-08-14T21:51:33.2900850Z 2025-08-14T21:51:33.2900937Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2901030Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.2901137Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2901348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2901416Z return mod(**inputs) 2025-08-14T21:51:33.2901703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2901785Z outputs = self.mobilebert( 2025-08-14T21:51:33.2902075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2902151Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2902447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2902521Z layer_outputs = layer_module( 2025-08-14T21:51:33.2902820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2902907Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2903197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2903331Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2903622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.2903747Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2903751Z 2025-08-14T21:51:33.2903855Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2904059Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2904155Z return mod(**inputs) 2025-08-14T21:51:33.2904447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2904519Z outputs = self.mobilebert( 2025-08-14T21:51:33.2904822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2904897Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2905222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2905300Z layer_outputs = layer_module( 2025-08-14T21:51:33.2905617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2905717Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2906021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.2906160Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.2906456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.2906591Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2906907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2907004Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2907008Z 2025-08-14T21:51:33.2907121Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2907325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2907395Z return mod(**inputs) 2025-08-14T21:51:33.2907694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2907767Z outputs = self.mobilebert( 2025-08-14T21:51:33.2908060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2908147Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2908446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2908531Z layer_outputs = layer_module( 2025-08-14T21:51:33.2908832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2908935Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2909240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2909360Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2909668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2909758Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2909762Z 2025-08-14T21:51:33.2909868Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2910089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2910217Z return mod(**inputs) 2025-08-14T21:51:33.2910518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2910602Z outputs = self.mobilebert( 2025-08-14T21:51:33.2910901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2911006Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2911306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2911381Z layer_outputs = layer_module( 2025-08-14T21:51:33.2911686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2911787Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2912114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2912254Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2912552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2912680Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2912688Z 2025-08-14T21:51:33.2912796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2913006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2913085Z return mod(**inputs) 2025-08-14T21:51:33.2913381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2913461Z outputs = self.mobilebert( 2025-08-14T21:51:33.2913758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2913838Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2914141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2914217Z layer_outputs = layer_module( 2025-08-14T21:51:33.2914522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2914622Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2914919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2915060Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2915359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2915451Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2915463Z 2025-08-14T21:51:33.2915572Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2915784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2915863Z return mod(**inputs) 2025-08-14T21:51:33.2916159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2916232Z outputs = self.mobilebert( 2025-08-14T21:51:33.2916533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2916611Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2916913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2917010Z layer_outputs = layer_module( 2025-08-14T21:51:33.2917310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2917416Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2917714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2917864Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2918166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2918293Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2918611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2918712Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2918716Z 2025-08-14T21:51:33.2918851Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2919071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2919140Z return mod(**inputs) 2025-08-14T21:51:33.2919450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2919524Z outputs = self.mobilebert( 2025-08-14T21:51:33.2919818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2919905Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2920200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2920277Z layer_outputs = layer_module( 2025-08-14T21:51:33.2920583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2920683Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2920998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2921117Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2921425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2921522Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2921526Z 2025-08-14T21:51:33.2921636Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2921876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2921947Z return mod(**inputs) 2025-08-14T21:51:33.2922298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2922380Z outputs = self.mobilebert( 2025-08-14T21:51:33.2922736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2922817Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2923123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2923199Z layer_outputs = layer_module( 2025-08-14T21:51:33.2923504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2923604Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2923905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2924053Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2924356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2924499Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2924503Z 2025-08-14T21:51:33.2924612Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2924821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2924901Z return mod(**inputs) 2025-08-14T21:51:33.2925198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2925280Z outputs = self.mobilebert( 2025-08-14T21:51:33.2925686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2925794Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2926107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2926184Z layer_outputs = layer_module( 2025-08-14T21:51:33.2926482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2926594Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2926890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2927032Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2927332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2927441Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2927446Z 2025-08-14T21:51:33.2927567Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2927792Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2927870Z return mod(**inputs) 2025-08-14T21:51:33.2928165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2928239Z outputs = self.mobilebert( 2025-08-14T21:51:33.2928542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2928618Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2928917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2929004Z layer_outputs = layer_module( 2025-08-14T21:51:33.2929301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2929408Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2929704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2929834Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2930138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2930266Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2930571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2930692Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2930696Z 2025-08-14T21:51:33.2930805Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2931024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2931095Z return mod(**inputs) 2025-08-14T21:51:33.2931415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2931496Z outputs = self.mobilebert( 2025-08-14T21:51:33.2931795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2931880Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2932178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2932273Z layer_outputs = layer_module( 2025-08-14T21:51:33.2932604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2932705Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2933012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2933131Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2933429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2933525Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2933529Z 2025-08-14T21:51:33.2933637Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2933850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2933928Z return mod(**inputs) 2025-08-14T21:51:33.2934230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2934313Z outputs = self.mobilebert( 2025-08-14T21:51:33.2934613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2934692Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2934995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2935069Z layer_outputs = layer_module( 2025-08-14T21:51:33.2935373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2935472Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2935767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.2935890Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.2936190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2936309Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2936322Z 2025-08-14T21:51:33.2936431Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2936640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2936716Z return mod(**inputs) 2025-08-14T21:51:33.2937008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2937082Z outputs = self.mobilebert( 2025-08-14T21:51:33.2937388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2937488Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2937995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2938080Z layer_outputs = layer_module( 2025-08-14T21:51:33.2938454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2938563Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2938860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2938992Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2939330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.2939423Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2939455Z 2025-08-14T21:51:33.2939576Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2939789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2939864Z return mod(**inputs) 2025-08-14T21:51:33.2940178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2940254Z outputs = self.mobilebert( 2025-08-14T21:51:33.2940560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2940639Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2940942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2941028Z layer_outputs = layer_module( 2025-08-14T21:51:33.2941332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.2941431Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.2941740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.2941874Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.2942182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.2942313Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2942616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2942723Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2942727Z 2025-08-14T21:51:33.2942839Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2943058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2943129Z return mod(**inputs) 2025-08-14T21:51:33.2943430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2943515Z outputs = self.mobilebert( 2025-08-14T21:51:33.2943813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2943907Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2944203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2944278Z layer_outputs = layer_module( 2025-08-14T21:51:33.2944606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2944730Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2945017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.2945129Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.2945133Z 2025-08-14T21:51:33.2945239Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2945451Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2945519Z return mod(**inputs) 2025-08-14T21:51:33.2945807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2945914Z outputs = self.mobilebert( 2025-08-14T21:51:33.2946222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2946306Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2946596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2946671Z layer_outputs = layer_module( 2025-08-14T21:51:33.2946966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.2947089Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.2947378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.2947501Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.2947506Z 2025-08-14T21:51:33.2947611Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2947824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2947893Z return mod(**inputs) 2025-08-14T21:51:33.2948185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2948269Z outputs = self.mobilebert( 2025-08-14T21:51:33.2948562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2948644Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2948933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2949007Z layer_outputs = layer_module( 2025-08-14T21:51:33.2949306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2949473Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2949761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.2949866Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.2949871Z 2025-08-14T21:51:33.2949977Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2950187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2950256Z return mod(**inputs) 2025-08-14T21:51:33.2950547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2950627Z outputs = self.mobilebert( 2025-08-14T21:51:33.2950914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2951053Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2951344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2951418Z layer_outputs = layer_module( 2025-08-14T21:51:33.2951738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2951900Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2952186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.2952318Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.2952623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2952750Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2952755Z 2025-08-14T21:51:33.2952866Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2953077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2953160Z return mod(**inputs) 2025-08-14T21:51:33.2953459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2953540Z outputs = self.mobilebert( 2025-08-14T21:51:33.2953842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2953930Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2954228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2954304Z layer_outputs = layer_module( 2025-08-14T21:51:33.2954599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2954771Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2955063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2955197Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2955489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.2955577Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.2955592Z 2025-08-14T21:51:33.2955701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2955911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2955989Z return mod(**inputs) 2025-08-14T21:51:33.2956278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2956351Z outputs = self.mobilebert( 2025-08-14T21:51:33.2956659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2956737Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2957045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2957121Z layer_outputs = layer_module( 2025-08-14T21:51:33.2957424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.2957599Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.2957923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.2958050Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.2958352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.2958501Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.2958812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2958910Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2958914Z 2025-08-14T21:51:33.2959025Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2959265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2959340Z return mod(**inputs) 2025-08-14T21:51:33.2959668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2959746Z outputs = self.mobilebert( 2025-08-14T21:51:33.2960049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2960136Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2960437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2960514Z layer_outputs = layer_module( 2025-08-14T21:51:33.2960822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2960993Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2961303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2961422Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2961721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2961818Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2961822Z 2025-08-14T21:51:33.2961931Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2962152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2962223Z return mod(**inputs) 2025-08-14T21:51:33.2962522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2962606Z outputs = self.mobilebert( 2025-08-14T21:51:33.2962903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2962989Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2963288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2963367Z layer_outputs = layer_module( 2025-08-14T21:51:33.2963671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2963763Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2964062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2964148Z self_outputs = self.self( 2025-08-14T21:51:33.2964446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.2964551Z self.value(value_tensor) 2025-08-14T21:51:33.2964555Z 2025-08-14T21:51:33.2964723Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2964971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2965134Z return mod(**inputs) 2025-08-14T21:51:33.2965521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2965672Z outputs = self.mobilebert( 2025-08-14T21:51:33.2965974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2966310Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2966676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2966813Z layer_outputs = layer_module( 2025-08-14T21:51:33.2967166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2967409Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2967782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.2967945Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.2968296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.2968414Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.2968421Z 2025-08-14T21:51:33.2968557Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2968820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2968944Z return mod(**inputs) 2025-08-14T21:51:33.2969317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2969419Z outputs = self.mobilebert( 2025-08-14T21:51:33.2969746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2969885Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2970196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2970365Z layer_outputs = layer_module( 2025-08-14T21:51:33.2970693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.2970888Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.2971249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.2971391Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.2971758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.2971889Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.2972213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.2972364Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.2972369Z 2025-08-14T21:51:33.2972506Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.2972781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.2973907Z return mod(**inputs) 2025-08-14T21:51:33.2974249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.2974376Z outputs = self.mobilebert( 2025-08-14T21:51:33.2974703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.2974835Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.2975176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.2975312Z layer_outputs = layer_module( 2025-08-14T21:51:33.2975678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.2975801Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.2998978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.2999191Z self_outputs = self.self( 2025-08-14T21:51:33.2999574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.2999676Z self.query(query_tensor) 2025-08-14T21:51:33.2999684Z 2025-08-14T21:51:33.2999818Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3000057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3000137Z return mod(**inputs) 2025-08-14T21:51:33.3000453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3000548Z outputs = self.mobilebert( 2025-08-14T21:51:33.3000853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3000945Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3001258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3001340Z layer_outputs = layer_module( 2025-08-14T21:51:33.3001652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.3001751Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.3002049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.3002135Z self_outputs = self.self( 2025-08-14T21:51:33.3002436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.3002512Z self.key(key_tensor) 2025-08-14T21:51:33.3002526Z 2025-08-14T21:51:33.3002623Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.3002708Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.3002837Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3003057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3003132Z return mod(**inputs) 2025-08-14T21:51:33.3003441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3003524Z outputs = self.mobilebert( 2025-08-14T21:51:33.3003824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3003917Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3004218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3004368Z layer_outputs = layer_module( 2025-08-14T21:51:33.3004674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.3004767Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.3005102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.3005240Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.3005721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.3005823Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3005829Z 2025-08-14T21:51:33.3005973Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3006206Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3006300Z return mod(**inputs) 2025-08-14T21:51:33.3006605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3006693Z outputs = self.mobilebert( 2025-08-14T21:51:33.3006998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3007086Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3007386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3007464Z layer_outputs = layer_module( 2025-08-14T21:51:33.3007774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.3007866Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.3008174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.3008306Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.3008607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.3008755Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3009055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3009166Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3009170Z 2025-08-14T21:51:33.3009284Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3009500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3009578Z return mod(**inputs) 2025-08-14T21:51:33.3009880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3009959Z outputs = self.mobilebert( 2025-08-14T21:51:33.3010264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3010353Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3010655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3010741Z layer_outputs = layer_module( 2025-08-14T21:51:33.3011045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3011146Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3011472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3011594Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3011893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.3012003Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.3012007Z 2025-08-14T21:51:33.3012114Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3012328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3012398Z return mod(**inputs) 2025-08-14T21:51:33.3012687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3012788Z outputs = self.mobilebert( 2025-08-14T21:51:33.3013098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3013185Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3013473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3013549Z layer_outputs = layer_module( 2025-08-14T21:51:33.3013847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3013944Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3014241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3014358Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3014648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.3014776Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.3014780Z 2025-08-14T21:51:33.3014887Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3015099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3015169Z return mod(**inputs) 2025-08-14T21:51:33.3015460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3015542Z outputs = self.mobilebert( 2025-08-14T21:51:33.3015843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3015918Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3016220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3016294Z layer_outputs = layer_module( 2025-08-14T21:51:33.3016594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3016691Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3016995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3017137Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3017439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.3017526Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3017537Z 2025-08-14T21:51:33.3017642Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3017852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3017945Z return mod(**inputs) 2025-08-14T21:51:33.3018238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3018315Z outputs = self.mobilebert( 2025-08-14T21:51:33.3018645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3018725Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3019034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3019110Z layer_outputs = layer_module( 2025-08-14T21:51:33.3019423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3019548Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3019870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3020002Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3020314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.3020441Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3020745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3020842Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3020846Z 2025-08-14T21:51:33.3020952Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3021169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3021238Z return mod(**inputs) 2025-08-14T21:51:33.3021536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3021608Z outputs = self.mobilebert( 2025-08-14T21:51:33.3021897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3021982Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3022284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3022367Z layer_outputs = layer_module( 2025-08-14T21:51:33.3022666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3022761Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3023058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3023174Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3023473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.3023568Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.3023572Z 2025-08-14T21:51:33.3023677Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3023888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3023957Z return mod(**inputs) 2025-08-14T21:51:33.3024256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3024338Z outputs = self.mobilebert( 2025-08-14T21:51:33.3024629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3024735Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3025030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3025125Z layer_outputs = layer_module( 2025-08-14T21:51:33.3025431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3025528Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3025828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3025952Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3026269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.3026432Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.3026436Z 2025-08-14T21:51:33.3026545Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3026754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3026834Z return mod(**inputs) 2025-08-14T21:51:33.3027126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3027208Z outputs = self.mobilebert( 2025-08-14T21:51:33.3027499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3027576Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3027879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3027958Z layer_outputs = layer_module( 2025-08-14T21:51:33.3028262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3028371Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3028670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3028813Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3029112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.3029214Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3029218Z 2025-08-14T21:51:33.3029333Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3029544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3029622Z return mod(**inputs) 2025-08-14T21:51:33.3029913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3029988Z outputs = self.mobilebert( 2025-08-14T21:51:33.3030292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3030371Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3030674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3030758Z layer_outputs = layer_module( 2025-08-14T21:51:33.3031057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3031164Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3031485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3031617Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3031927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.3032078Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3032387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3032486Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3032490Z 2025-08-14T21:51:33.3032599Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3032840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3032915Z return mod(**inputs) 2025-08-14T21:51:33.3033241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3033319Z outputs = self.mobilebert( 2025-08-14T21:51:33.3033621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3033710Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3034010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3034086Z layer_outputs = layer_module( 2025-08-14T21:51:33.3034393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3034492Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3034800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3034922Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3035224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.3035323Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.3035327Z 2025-08-14T21:51:33.3035435Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3035655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3035727Z return mod(**inputs) 2025-08-14T21:51:33.3036028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3036111Z outputs = self.mobilebert( 2025-08-14T21:51:33.3036410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3036491Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3036798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3036875Z layer_outputs = layer_module( 2025-08-14T21:51:33.3037182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3037280Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3037577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3037993Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3038303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.3038520Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.3038524Z 2025-08-14T21:51:33.3038633Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3038846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3038952Z return mod(**inputs) 2025-08-14T21:51:33.3039257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3039332Z outputs = self.mobilebert( 2025-08-14T21:51:33.3039640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3039719Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3040063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3040144Z layer_outputs = layer_module( 2025-08-14T21:51:33.3040479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3040588Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3040887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3041028Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3041328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.3041418Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3041422Z 2025-08-14T21:51:33.3041537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3041751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3041824Z return mod(**inputs) 2025-08-14T21:51:33.3042134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3042209Z outputs = self.mobilebert( 2025-08-14T21:51:33.3042514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3042594Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3042892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3042976Z layer_outputs = layer_module( 2025-08-14T21:51:33.3043279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3043389Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3043703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3043836Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3044143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.3044276Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3044578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3044683Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3044687Z 2025-08-14T21:51:33.3044795Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3045009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3045106Z return mod(**inputs) 2025-08-14T21:51:33.3045409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3045552Z outputs = self.mobilebert( 2025-08-14T21:51:33.3045862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3045964Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3046275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3046352Z layer_outputs = layer_module( 2025-08-14T21:51:33.3046658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.3046789Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.3047104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.3047222Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.3047227Z 2025-08-14T21:51:33.3047338Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3047553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3047634Z return mod(**inputs) 2025-08-14T21:51:33.3047935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3048017Z outputs = self.mobilebert( 2025-08-14T21:51:33.3048322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3048400Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3048710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3048790Z layer_outputs = layer_module( 2025-08-14T21:51:33.3049104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.3049228Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.3049521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.3049646Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.3049649Z 2025-08-14T21:51:33.3049756Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3049972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3050041Z return mod(**inputs) 2025-08-14T21:51:33.3050334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3050419Z outputs = self.mobilebert( 2025-08-14T21:51:33.3050714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3050789Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3051093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3051168Z layer_outputs = layer_module( 2025-08-14T21:51:33.3051468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.3051637Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.3051927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.3052051Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.3052057Z 2025-08-14T21:51:33.3052162Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3052379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3052466Z return mod(**inputs) 2025-08-14T21:51:33.3052765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3052846Z outputs = self.mobilebert( 2025-08-14T21:51:33.3053145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3053221Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3053555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3053631Z layer_outputs = layer_module( 2025-08-14T21:51:33.3053946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.3054114Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.3054411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.3054550Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.3054847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3054952Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3054956Z 2025-08-14T21:51:33.3055065Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3055280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3055363Z return mod(**inputs) 2025-08-14T21:51:33.3055661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3055736Z outputs = self.mobilebert( 2025-08-14T21:51:33.3056049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3056127Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3056422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3056495Z layer_outputs = layer_module( 2025-08-14T21:51:33.3056799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.3056977Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.3057280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.3057417Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.3057714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.3057806Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3057811Z 2025-08-14T21:51:33.3057926Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3058138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3058209Z return mod(**inputs) 2025-08-14T21:51:33.3058515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3058649Z outputs = self.mobilebert( 2025-08-14T21:51:33.3058955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3059034Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3059343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3059454Z layer_outputs = layer_module( 2025-08-14T21:51:33.3059759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.3059935Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.3060237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.3060384Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.3060712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.3060846Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3061161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3061266Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3061270Z 2025-08-14T21:51:33.3061381Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3061607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3061680Z return mod(**inputs) 2025-08-14T21:51:33.3061987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3062075Z outputs = self.mobilebert( 2025-08-14T21:51:33.3062386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3062470Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3062791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3062871Z layer_outputs = layer_module( 2025-08-14T21:51:33.3063180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.3063360Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.3063684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.3063806Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.3064104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.3064200Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.3064205Z 2025-08-14T21:51:33.3064314Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3064528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3064609Z return mod(**inputs) 2025-08-14T21:51:33.3064903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3064987Z outputs = self.mobilebert( 2025-08-14T21:51:33.3065282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3065362Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3065669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3065765Z layer_outputs = layer_module( 2025-08-14T21:51:33.3066062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.3066170Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.3066460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.3066543Z self_outputs = self.self( 2025-08-14T21:51:33.3066856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.3066932Z self.value(value_tensor) 2025-08-14T21:51:33.3066943Z 2025-08-14T21:51:33.3067067Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3067295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3067391Z return mod(**inputs) 2025-08-14T21:51:33.3067699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3067772Z outputs = self.mobilebert( 2025-08-14T21:51:33.3068071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3068147Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3068446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3068522Z layer_outputs = layer_module( 2025-08-14T21:51:33.3068816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.3068997Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.3069298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.3069418Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.3069728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.3069818Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.3069822Z 2025-08-14T21:51:33.3069937Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3070151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3070221Z return mod(**inputs) 2025-08-14T21:51:33.3070531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3070608Z outputs = self.mobilebert( 2025-08-14T21:51:33.3070918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3070998Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3071298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3071384Z layer_outputs = layer_module( 2025-08-14T21:51:33.3071684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.3071853Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.3072159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.3072277Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.3072608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.3072699Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.3072997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3073122Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3073126Z 2025-08-14T21:51:33.3073233Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3073452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3073523Z return mod(**inputs) 2025-08-14T21:51:33.3073835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3073936Z outputs = self.mobilebert( 2025-08-14T21:51:33.3074255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3074344Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3074659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3074738Z layer_outputs = layer_module( 2025-08-14T21:51:33.3075060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.3075152Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.3075458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.3075545Z self_outputs = self.self( 2025-08-14T21:51:33.3075907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.3075998Z self.query(query_tensor) 2025-08-14T21:51:33.3076001Z 2025-08-14T21:51:33.3076115Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3076347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3076430Z return mod(**inputs) 2025-08-14T21:51:33.3076744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3076820Z outputs = self.mobilebert( 2025-08-14T21:51:33.3077144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3077222Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3077533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3077612Z layer_outputs = layer_module( 2025-08-14T21:51:33.3077916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.3078015Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.3078316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.3078429Z self_outputs = self.self( 2025-08-14T21:51:33.3078731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.3078807Z self.key(key_tensor) 2025-08-14T21:51:33.3078811Z 2025-08-14T21:51:33.3078909Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.3078994Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.3079107Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3079357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3079430Z return mod(**inputs) 2025-08-14T21:51:33.3079741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3079815Z outputs = self.mobilebert( 2025-08-14T21:51:33.3080149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3080240Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3080539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3080617Z layer_outputs = layer_module( 2025-08-14T21:51:33.3080947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.3081046Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.3081375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.3081510Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.3081810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.3081914Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3081918Z 2025-08-14T21:51:33.3082027Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3082249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3082321Z return mod(**inputs) 2025-08-14T21:51:33.3082631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3082718Z outputs = self.mobilebert( 2025-08-14T21:51:33.3083030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3083110Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3083427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3083506Z layer_outputs = layer_module( 2025-08-14T21:51:33.3083824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.3083914Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.3084223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.3084367Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.3084676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.3084823Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3085133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3085235Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3085239Z 2025-08-14T21:51:33.3085360Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3085673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3085767Z return mod(**inputs) 2025-08-14T21:51:33.3086068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3086151Z outputs = self.mobilebert( 2025-08-14T21:51:33.3086490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3086574Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3086878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3086988Z layer_outputs = layer_module( 2025-08-14T21:51:33.3087290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3087407Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3087708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3087833Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3088169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.3088289Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.3088294Z 2025-08-14T21:51:33.3088421Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3088640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3088716Z return mod(**inputs) 2025-08-14T21:51:33.3089029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3089107Z outputs = self.mobilebert( 2025-08-14T21:51:33.3089409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3089499Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3089804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3089895Z layer_outputs = layer_module( 2025-08-14T21:51:33.3090196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3090299Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3090616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3090737Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3091050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.3091173Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.3091177Z 2025-08-14T21:51:33.3091292Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3091519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3091593Z return mod(**inputs) 2025-08-14T21:51:33.3091901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3092011Z outputs = self.mobilebert( 2025-08-14T21:51:33.3092318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3092408Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3092707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3092789Z layer_outputs = layer_module( 2025-08-14T21:51:33.3093105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3093232Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3093546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3093681Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3093981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.3094106Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3094110Z 2025-08-14T21:51:33.3094222Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3094441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3094525Z return mod(**inputs) 2025-08-14T21:51:33.3094855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3094945Z outputs = self.mobilebert( 2025-08-14T21:51:33.3095267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3095348Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3095667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3095747Z layer_outputs = layer_module( 2025-08-14T21:51:33.3096056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3096160Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3096466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3096613Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3096917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.3097048Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3097368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3097470Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3097474Z 2025-08-14T21:51:33.3097598Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3097811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3097883Z return mod(**inputs) 2025-08-14T21:51:33.3098195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3098274Z outputs = self.mobilebert( 2025-08-14T21:51:33.3098593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3098674Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3098979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3099069Z layer_outputs = layer_module( 2025-08-14T21:51:33.3099376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3099476Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3099788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3099909Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3100225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.3100345Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.3100349Z 2025-08-14T21:51:33.3100460Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3100683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3100775Z return mod(**inputs) 2025-08-14T21:51:33.3101086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3101160Z outputs = self.mobilebert( 2025-08-14T21:51:33.3101457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3101545Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3101883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3101975Z layer_outputs = layer_module( 2025-08-14T21:51:33.3102301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3102403Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3102719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3102837Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3103134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.3103267Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.3103270Z 2025-08-14T21:51:33.3103390Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3103606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3103681Z return mod(**inputs) 2025-08-14T21:51:33.3103992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3104067Z outputs = self.mobilebert( 2025-08-14T21:51:33.3104377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3104457Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3104759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3104848Z layer_outputs = layer_module( 2025-08-14T21:51:33.3105147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3105249Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3105563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3105697Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3106001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.3106093Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3106097Z 2025-08-14T21:51:33.3106211Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3106434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3106507Z return mod(**inputs) 2025-08-14T21:51:33.3106817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3106935Z outputs = self.mobilebert( 2025-08-14T21:51:33.3107241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3107332Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3107632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3107733Z layer_outputs = layer_module( 2025-08-14T21:51:33.3108051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3108150Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3108465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3108620Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3108942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.3109086Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3109390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3109498Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3109502Z 2025-08-14T21:51:33.3109613Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3109825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3109907Z return mod(**inputs) 2025-08-14T21:51:33.3110207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3110285Z outputs = self.mobilebert( 2025-08-14T21:51:33.3110597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3110677Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3110987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3111065Z layer_outputs = layer_module( 2025-08-14T21:51:33.3111361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3111469Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3111771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3111898Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3112196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.3112289Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.3112293Z 2025-08-14T21:51:33.3112410Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3112623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3112698Z return mod(**inputs) 2025-08-14T21:51:33.3113012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3113085Z outputs = self.mobilebert( 2025-08-14T21:51:33.3113383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3113459Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3113751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3113861Z layer_outputs = layer_module( 2025-08-14T21:51:33.3114158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3114264Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3114578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3114692Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3114995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.3115108Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.3115112Z 2025-08-14T21:51:33.3115229Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3115456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3115546Z return mod(**inputs) 2025-08-14T21:51:33.3115848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3115921Z outputs = self.mobilebert( 2025-08-14T21:51:33.3116214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3116299Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3116587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3116672Z layer_outputs = layer_module( 2025-08-14T21:51:33.3116961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3117061Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3117364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3117494Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3117794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.3117888Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3117892Z 2025-08-14T21:51:33.3118001Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3118223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3118295Z return mod(**inputs) 2025-08-14T21:51:33.3118593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3118685Z outputs = self.mobilebert( 2025-08-14T21:51:33.3118987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3119075Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3119374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3119454Z layer_outputs = layer_module( 2025-08-14T21:51:33.3119760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3119863Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3120173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3120309Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3120638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.3120773Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3121076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3121193Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3121208Z 2025-08-14T21:51:33.3121320Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3121532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3121616Z return mod(**inputs) 2025-08-14T21:51:33.3121912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3122014Z outputs = self.mobilebert( 2025-08-14T21:51:33.3122345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3122426Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3122737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3122817Z layer_outputs = layer_module( 2025-08-14T21:51:33.3123120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.3123309Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.3123611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.3123703Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.3123715Z 2025-08-14T21:51:33.3123829Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3124046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3124129Z return mod(**inputs) 2025-08-14T21:51:33.3124428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3124505Z outputs = self.mobilebert( 2025-08-14T21:51:33.3124814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3124893Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3125199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3125278Z layer_outputs = layer_module( 2025-08-14T21:51:33.3125685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.3125834Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.3126138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.3126259Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.3126275Z 2025-08-14T21:51:33.3126387Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3126610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3126693Z return mod(**inputs) 2025-08-14T21:51:33.3126997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3127075Z outputs = self.mobilebert( 2025-08-14T21:51:33.3127391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3127498Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3127816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3127897Z layer_outputs = layer_module( 2025-08-14T21:51:33.3128202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.3128408Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.3128712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.3128819Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.3128833Z 2025-08-14T21:51:33.3128945Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3129181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3129266Z return mod(**inputs) 2025-08-14T21:51:33.3129600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3129678Z outputs = self.mobilebert( 2025-08-14T21:51:33.3129990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3130070Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3130379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3130457Z layer_outputs = layer_module( 2025-08-14T21:51:33.3130759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.3130941Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.3131247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.3131386Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.3131693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3131793Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3131797Z 2025-08-14T21:51:33.3131917Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3132136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3132211Z return mod(**inputs) 2025-08-14T21:51:33.3132523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3132600Z outputs = self.mobilebert( 2025-08-14T21:51:33.3132909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3132991Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3133289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3133376Z layer_outputs = layer_module( 2025-08-14T21:51:33.3133678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.3133855Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.3134156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.3134290Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.3134632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.3134724Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3134728Z 2025-08-14T21:51:33.3134838Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3135087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3135157Z return mod(**inputs) 2025-08-14T21:51:33.3135468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3135542Z outputs = self.mobilebert( 2025-08-14T21:51:33.3135843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3135955Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3136285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3136375Z layer_outputs = layer_module( 2025-08-14T21:51:33.3136677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.3136849Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.3137162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.3137294Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.3137839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.3138000Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3138310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3138424Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3138429Z 2025-08-14T21:51:33.3138543Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3138766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3138852Z return mod(**inputs) 2025-08-14T21:51:33.3139153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3139244Z outputs = self.mobilebert( 2025-08-14T21:51:33.3139544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3139624Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3139939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3140020Z layer_outputs = layer_module( 2025-08-14T21:51:33.3140335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.3140511Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.3140815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.3140947Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.3141245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.3141334Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.3141349Z 2025-08-14T21:51:33.3141527Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3141742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3141823Z return mod(**inputs) 2025-08-14T21:51:33.3142121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3142226Z outputs = self.mobilebert( 2025-08-14T21:51:33.3142537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3142615Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3142928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3143004Z layer_outputs = layer_module( 2025-08-14T21:51:33.3143329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.3143464Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.3143769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.3143847Z self_outputs = self.self( 2025-08-14T21:51:33.3144160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.3144237Z self.value(value_tensor) 2025-08-14T21:51:33.3144240Z 2025-08-14T21:51:33.3144358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3144570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3144639Z return mod(**inputs) 2025-08-14T21:51:33.3144946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3145022Z outputs = self.mobilebert( 2025-08-14T21:51:33.3145330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3145407Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3145732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3145818Z layer_outputs = layer_module( 2025-08-14T21:51:33.3146113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.3146282Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.3146591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.3146709Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.3147017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.3147104Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.3147108Z 2025-08-14T21:51:33.3147216Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3147435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3147504Z return mod(**inputs) 2025-08-14T21:51:33.3147807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3147881Z outputs = self.mobilebert( 2025-08-14T21:51:33.3148176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3148261Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3148586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3148661Z layer_outputs = layer_module( 2025-08-14T21:51:33.3148967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.3149160Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.3149464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.3149580Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.3149878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.3149995Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.3150314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3150421Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3150425Z 2025-08-14T21:51:33.3150533Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3150746Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3150825Z return mod(**inputs) 2025-08-14T21:51:33.3151125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3151199Z outputs = self.mobilebert( 2025-08-14T21:51:33.3151507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3151584Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3151943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3152023Z layer_outputs = layer_module( 2025-08-14T21:51:33.3152328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.3152428Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.3152745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.3152827Z self_outputs = self.self( 2025-08-14T21:51:33.3153138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.3153214Z self.query(query_tensor) 2025-08-14T21:51:33.3153218Z 2025-08-14T21:51:33.3153336Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3153567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3153640Z return mod(**inputs) 2025-08-14T21:51:33.3154007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3154082Z outputs = self.mobilebert( 2025-08-14T21:51:33.3154393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3154472Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3154790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3154874Z layer_outputs = layer_module( 2025-08-14T21:51:33.3155190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.3155289Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.3155609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.3155684Z self_outputs = self.self( 2025-08-14T21:51:33.3156003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.3156094Z self.key(key_tensor) 2025-08-14T21:51:33.3156098Z 2025-08-14T21:51:33.3156186Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.3156279Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.3156388Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3156610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3156680Z return mod(**inputs) 2025-08-14T21:51:33.3157022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3157107Z outputs = self.mobilebert( 2025-08-14T21:51:33.3157445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3157524Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3157825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3157900Z layer_outputs = layer_module( 2025-08-14T21:51:33.3158203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.3158292Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.3158589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.3158727Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.3159027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.3159124Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3159128Z 2025-08-14T21:51:33.3159236Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3159449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3159529Z return mod(**inputs) 2025-08-14T21:51:33.3159826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3159899Z outputs = self.mobilebert( 2025-08-14T21:51:33.3160205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3160286Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3160595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3160672Z layer_outputs = layer_module( 2025-08-14T21:51:33.3160970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.3161067Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.3161367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.3161504Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.3161803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.3161941Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3162250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3162370Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3162374Z 2025-08-14T21:51:33.3162490Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3162702Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3162796Z return mod(**inputs) 2025-08-14T21:51:33.3163101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3163177Z outputs = self.mobilebert( 2025-08-14T21:51:33.3163475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3163561Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3163877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3164012Z layer_outputs = layer_module( 2025-08-14T21:51:33.3164313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3164412Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3164719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3164837Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3165143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.3165233Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.3165237Z 2025-08-14T21:51:33.3165345Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3165636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3165715Z return mod(**inputs) 2025-08-14T21:51:33.3166017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3166104Z outputs = self.mobilebert( 2025-08-14T21:51:33.3166408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3166493Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3166792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3166869Z layer_outputs = layer_module( 2025-08-14T21:51:33.3167178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3167281Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3167592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3167713Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3168017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.3168146Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.3168151Z 2025-08-14T21:51:33.3168261Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3168475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3168554Z return mod(**inputs) 2025-08-14T21:51:33.3168858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3168967Z outputs = self.mobilebert( 2025-08-14T21:51:33.3169274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3169352Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3169660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3169763Z layer_outputs = layer_module( 2025-08-14T21:51:33.3170073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3170174Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3170480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3170643Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3170963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.3171057Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3171070Z 2025-08-14T21:51:33.3171180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3171393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3171473Z return mod(**inputs) 2025-08-14T21:51:33.3171796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3171871Z outputs = self.mobilebert( 2025-08-14T21:51:33.3172177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3172258Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3172570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3172646Z layer_outputs = layer_module( 2025-08-14T21:51:33.3172945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3173053Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3173353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3173486Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3173794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.3173924Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3174232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3174344Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3174348Z 2025-08-14T21:51:33.3174457Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3174675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3174748Z return mod(**inputs) 2025-08-14T21:51:33.3175052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3175127Z outputs = self.mobilebert( 2025-08-14T21:51:33.3175423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3175507Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3175806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3175905Z layer_outputs = layer_module( 2025-08-14T21:51:33.3176213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3176311Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3176642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3176760Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3177064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.3177160Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.3177164Z 2025-08-14T21:51:33.3177289Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3177512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3177600Z return mod(**inputs) 2025-08-14T21:51:33.3177901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3177985Z outputs = self.mobilebert( 2025-08-14T21:51:33.3178286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3178366Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3178671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3178747Z layer_outputs = layer_module( 2025-08-14T21:51:33.3179056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3179156Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3179455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3179581Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3179878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.3180004Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.3180007Z 2025-08-14T21:51:33.3180115Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3180327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3180406Z return mod(**inputs) 2025-08-14T21:51:33.3180704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3180786Z outputs = self.mobilebert( 2025-08-14T21:51:33.3181085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3181162Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3181466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3181544Z layer_outputs = layer_module( 2025-08-14T21:51:33.3181840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3181944Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3182241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3182378Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3182700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.3182789Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3182793Z 2025-08-14T21:51:33.3182909Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3183149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3183226Z return mod(**inputs) 2025-08-14T21:51:33.3183525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3183599Z outputs = self.mobilebert( 2025-08-14T21:51:33.3183904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3183999Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3184317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3184404Z layer_outputs = layer_module( 2025-08-14T21:51:33.3184703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3184809Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3185110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3185240Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3185547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.3185674Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3185981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3186083Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3186087Z 2025-08-14T21:51:33.3186194Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3186413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3186487Z return mod(**inputs) 2025-08-14T21:51:33.3186784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3186867Z outputs = self.mobilebert( 2025-08-14T21:51:33.3187163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3187246Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3187546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3187625Z layer_outputs = layer_module( 2025-08-14T21:51:33.3187933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3188031Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3188341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3188459Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3188755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.3188853Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.3188856Z 2025-08-14T21:51:33.3188965Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3189204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3189277Z return mod(**inputs) 2025-08-14T21:51:33.3189575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3189658Z outputs = self.mobilebert( 2025-08-14T21:51:33.3189978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3190056Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3190364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3190439Z layer_outputs = layer_module( 2025-08-14T21:51:33.3190764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3190867Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3191182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3191310Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3191608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.3191726Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.3191739Z 2025-08-14T21:51:33.3191848Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3192062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3192138Z return mod(**inputs) 2025-08-14T21:51:33.3192438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3192515Z outputs = self.mobilebert( 2025-08-14T21:51:33.3192821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3192899Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3193204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3193281Z layer_outputs = layer_module( 2025-08-14T21:51:33.3193579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3193684Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3193987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3194120Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3194429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.3194519Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3194523Z 2025-08-14T21:51:33.3194640Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3194854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3194924Z return mod(**inputs) 2025-08-14T21:51:33.3195232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3195308Z outputs = self.mobilebert( 2025-08-14T21:51:33.3195616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3195693Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3195990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3196095Z layer_outputs = layer_module( 2025-08-14T21:51:33.3196402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3196527Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3196828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3196958Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3197265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.3197395Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3197715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3197842Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3197846Z 2025-08-14T21:51:33.3197960Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3198177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3198248Z return mod(**inputs) 2025-08-14T21:51:33.3198547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3198631Z outputs = self.mobilebert( 2025-08-14T21:51:33.3198927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3199012Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3199310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3199389Z layer_outputs = layer_module( 2025-08-14T21:51:33.3199695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.3199823Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.3200126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.3200223Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.3200227Z 2025-08-14T21:51:33.3200334Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3200552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3200621Z return mod(**inputs) 2025-08-14T21:51:33.3200924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3201007Z outputs = self.mobilebert( 2025-08-14T21:51:33.3201306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3201394Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3201690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3201766Z layer_outputs = layer_module( 2025-08-14T21:51:33.3202069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.3202195Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.3202496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.3202641Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.3202645Z 2025-08-14T21:51:33.3202753Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3202972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3203042Z return mod(**inputs) 2025-08-14T21:51:33.3203363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3203448Z outputs = self.mobilebert( 2025-08-14T21:51:33.3203752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3203837Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3204138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3204232Z layer_outputs = layer_module( 2025-08-14T21:51:33.3204560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.3204732Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.3205033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.3205145Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.3205149Z 2025-08-14T21:51:33.3205255Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3205552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3205633Z return mod(**inputs) 2025-08-14T21:51:33.3205940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3206027Z outputs = self.mobilebert( 2025-08-14T21:51:33.3206334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3206422Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3206721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3206800Z layer_outputs = layer_module( 2025-08-14T21:51:33.3207109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.3207278Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.3207576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.3207718Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.3208022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3208132Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3208136Z 2025-08-14T21:51:33.3208245Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3208461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3208542Z return mod(**inputs) 2025-08-14T21:51:33.3208841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3208927Z outputs = self.mobilebert( 2025-08-14T21:51:33.3209228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3209308Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3209653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3209731Z layer_outputs = layer_module( 2025-08-14T21:51:33.3210038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.3210226Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.3210525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.3210660Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.3210960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.3211100Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3211113Z 2025-08-14T21:51:33.3211227Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3211469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3211550Z return mod(**inputs) 2025-08-14T21:51:33.3211855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3211932Z outputs = self.mobilebert( 2025-08-14T21:51:33.3212243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3212321Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3212631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3212708Z layer_outputs = layer_module( 2025-08-14T21:51:33.3213009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.3213188Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.3213489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.3213620Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.3213931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.3214061Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3214373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3214472Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3214477Z 2025-08-14T21:51:33.3214589Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3214817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3214888Z return mod(**inputs) 2025-08-14T21:51:33.3215194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3215274Z outputs = self.mobilebert( 2025-08-14T21:51:33.3215580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3215665Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3215964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3216040Z layer_outputs = layer_module( 2025-08-14T21:51:33.3216348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.3216544Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.3216849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.3216968Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.3217292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.3217388Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.3217392Z 2025-08-14T21:51:33.3217499Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3217722Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3217792Z return mod(**inputs) 2025-08-14T21:51:33.3218106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3218210Z outputs = self.mobilebert( 2025-08-14T21:51:33.3218512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3218598Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3218900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3218976Z layer_outputs = layer_module( 2025-08-14T21:51:33.3219280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.3219369Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.3219670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.3219754Z self_outputs = self.self( 2025-08-14T21:51:33.3220052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.3220135Z self.value(value_tensor) 2025-08-14T21:51:33.3220139Z 2025-08-14T21:51:33.3220248Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3220463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3220542Z return mod(**inputs) 2025-08-14T21:51:33.3220840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3220920Z outputs = self.mobilebert( 2025-08-14T21:51:33.3221220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3221300Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3221615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3221691Z layer_outputs = layer_module( 2025-08-14T21:51:33.3221989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.3222168Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.3222469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.3222595Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.3222898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.3222989Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.3223015Z 2025-08-14T21:51:33.3223133Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3223348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3223428Z return mod(**inputs) 2025-08-14T21:51:33.3223728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3223826Z outputs = self.mobilebert( 2025-08-14T21:51:33.3224133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3224210Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3224509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3224593Z layer_outputs = layer_module( 2025-08-14T21:51:33.3224910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.3225107Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.3225413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.3225532Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.3225842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.3225934Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.3226244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3226342Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3226346Z 2025-08-14T21:51:33.3226457Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3226682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3226751Z return mod(**inputs) 2025-08-14T21:51:33.3227054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3227140Z outputs = self.mobilebert( 2025-08-14T21:51:33.3227440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3227525Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3227823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3227898Z layer_outputs = layer_module( 2025-08-14T21:51:33.3228206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.3228297Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.3228608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.3228683Z self_outputs = self.self( 2025-08-14T21:51:33.3228982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.3229070Z self.query(query_tensor) 2025-08-14T21:51:33.3229073Z 2025-08-14T21:51:33.3229183Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3229396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3229474Z return mod(**inputs) 2025-08-14T21:51:33.3229777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3229884Z outputs = self.mobilebert( 2025-08-14T21:51:33.3230187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3230265Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3230572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3230671Z layer_outputs = layer_module( 2025-08-14T21:51:33.3230979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.3231069Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.3231369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.3231457Z self_outputs = self.self( 2025-08-14T21:51:33.3231775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.3231879Z self.key(key_tensor) 2025-08-14T21:51:33.3231883Z 2025-08-14T21:51:33.3231984Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.3232072Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.3232188Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3232403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3232476Z return mod(**inputs) 2025-08-14T21:51:33.3232785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3232861Z outputs = self.mobilebert( 2025-08-14T21:51:33.3233171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3233260Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3233573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3233659Z layer_outputs = layer_module( 2025-08-14T21:51:33.3233959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.3234052Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.3234375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.3234506Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.3234818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.3234911Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3234916Z 2025-08-14T21:51:33.3235026Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3235247Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3235324Z return mod(**inputs) 2025-08-14T21:51:33.3235624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3235709Z outputs = self.mobilebert( 2025-08-14T21:51:33.3236011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3236099Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3236398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3236476Z layer_outputs = layer_module( 2025-08-14T21:51:33.3236787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.3236901Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.3237211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.3237342Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.3237859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.3238013Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3238312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3238410Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3238425Z 2025-08-14T21:51:33.3238587Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3238847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3238930Z return mod(**inputs) 2025-08-14T21:51:33.3239237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3239314Z outputs = self.mobilebert( 2025-08-14T21:51:33.3239628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3239707Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3240016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3240096Z layer_outputs = layer_module( 2025-08-14T21:51:33.3240395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3240508Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3240811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3240943Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3241247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.3241339Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.3241342Z 2025-08-14T21:51:33.3241465Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3241680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3241754Z return mod(**inputs) 2025-08-14T21:51:33.3242073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3242152Z outputs = self.mobilebert( 2025-08-14T21:51:33.3242462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3242541Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3242846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3242939Z layer_outputs = layer_module( 2025-08-14T21:51:33.3243240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3243353Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3243651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3243776Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3244133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.3244254Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.3244258Z 2025-08-14T21:51:33.3244368Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3244630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3244702Z return mod(**inputs) 2025-08-14T21:51:33.3245009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3245089Z outputs = self.mobilebert( 2025-08-14T21:51:33.3245386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3245554Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3245890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3245982Z layer_outputs = layer_module( 2025-08-14T21:51:33.3246285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3246391Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3246699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3246835Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3247138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.3247238Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3247243Z 2025-08-14T21:51:33.3247355Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3247581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3247655Z return mod(**inputs) 2025-08-14T21:51:33.3247958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3248049Z outputs = self.mobilebert( 2025-08-14T21:51:33.3248353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3248441Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3248750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3248827Z layer_outputs = layer_module( 2025-08-14T21:51:33.3249140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3249245Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3249545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3249688Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3249991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.3250128Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3250429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3250528Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3250532Z 2025-08-14T21:51:33.3250654Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3250897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3250977Z return mod(**inputs) 2025-08-14T21:51:33.3251279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3251378Z outputs = self.mobilebert( 2025-08-14T21:51:33.3251689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3251767Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3252069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3252158Z layer_outputs = layer_module( 2025-08-14T21:51:33.3252475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3252586Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3252945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3253064Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3253367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.3253457Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.3253462Z 2025-08-14T21:51:33.3253578Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3253790Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3253861Z return mod(**inputs) 2025-08-14T21:51:33.3254167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3254248Z outputs = self.mobilebert( 2025-08-14T21:51:33.3254547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3254636Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3254935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3255023Z layer_outputs = layer_module( 2025-08-14T21:51:33.3255326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3255437Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3255737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3255853Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3256156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.3256275Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.3256279Z 2025-08-14T21:51:33.3256384Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3256604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3256677Z return mod(**inputs) 2025-08-14T21:51:33.3256988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3257067Z outputs = self.mobilebert( 2025-08-14T21:51:33.3257366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3257458Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3257791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3257870Z layer_outputs = layer_module( 2025-08-14T21:51:33.3258182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3258306Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3258636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3258766Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3259059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.3259160Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3259190Z 2025-08-14T21:51:33.3259300Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3260420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3260510Z return mod(**inputs) 2025-08-14T21:51:33.3260822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3260916Z outputs = self.mobilebert( 2025-08-14T21:51:33.3261212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3261291Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3261599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3261678Z layer_outputs = layer_module( 2025-08-14T21:51:33.3261982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3262084Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3262376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3262514Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3262810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.3262943Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3263237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3263333Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3263336Z 2025-08-14T21:51:33.3263451Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3263668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3263741Z return mod(**inputs) 2025-08-14T21:51:33.3264049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3264137Z outputs = self.mobilebert( 2025-08-14T21:51:33.3264441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3264519Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3264815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3264902Z layer_outputs = layer_module( 2025-08-14T21:51:33.3265204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3265340Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3265642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3265759Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3266070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.3266183Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.3266187Z 2025-08-14T21:51:33.3266298Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3266517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3266588Z return mod(**inputs) 2025-08-14T21:51:33.3266919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3266998Z outputs = self.mobilebert( 2025-08-14T21:51:33.3267312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3267401Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3267698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3267783Z layer_outputs = layer_module( 2025-08-14T21:51:33.3268082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3268181Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3268494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3268612Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3268919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.3269047Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.3269051Z 2025-08-14T21:51:33.3269159Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3269382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3269453Z return mod(**inputs) 2025-08-14T21:51:33.3269754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3269848Z outputs = self.mobilebert( 2025-08-14T21:51:33.3270138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3270224Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3270520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3270594Z layer_outputs = layer_module( 2025-08-14T21:51:33.3270894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3270992Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3271294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3271434Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3271736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.3271834Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3271839Z 2025-08-14T21:51:33.3271971Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3272187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3272265Z return mod(**inputs) 2025-08-14T21:51:33.3272566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3272672Z outputs = self.mobilebert( 2025-08-14T21:51:33.3272974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3273051Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3273360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3273437Z layer_outputs = layer_module( 2025-08-14T21:51:33.3273797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3273926Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3274227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3274367Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3274667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.3274794Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3275102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3275200Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3275203Z 2025-08-14T21:51:33.3275320Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3275534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3275609Z return mod(**inputs) 2025-08-14T21:51:33.3275917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3275993Z outputs = self.mobilebert( 2025-08-14T21:51:33.3276299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3276378Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3276675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3276760Z layer_outputs = layer_module( 2025-08-14T21:51:33.3277060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.3277192Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.3277502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.3277590Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.3277593Z 2025-08-14T21:51:33.3277711Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3277923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3277993Z return mod(**inputs) 2025-08-14T21:51:33.3278299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3278375Z outputs = self.mobilebert( 2025-08-14T21:51:33.3278680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3278784Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3279088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3279173Z layer_outputs = layer_module( 2025-08-14T21:51:33.3279471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.3279622Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.3279938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.3280056Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.3280059Z 2025-08-14T21:51:33.3280177Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3280411Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3280485Z return mod(**inputs) 2025-08-14T21:51:33.3280815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3280894Z outputs = self.mobilebert( 2025-08-14T21:51:33.3281202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3281281Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3281581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3281664Z layer_outputs = layer_module( 2025-08-14T21:51:33.3281965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.3282135Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.3282443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.3282544Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.3282548Z 2025-08-14T21:51:33.3282663Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3282879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3282949Z return mod(**inputs) 2025-08-14T21:51:33.3283255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3283330Z outputs = self.mobilebert( 2025-08-14T21:51:33.3283634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3283714Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3284016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3284101Z layer_outputs = layer_module( 2025-08-14T21:51:33.3284403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.3284574Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.3284880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.3285011Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.3285321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3285418Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3285495Z 2025-08-14T21:51:33.3285652Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3285872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3285940Z return mod(**inputs) 2025-08-14T21:51:33.3286249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3286358Z outputs = self.mobilebert( 2025-08-14T21:51:33.3286662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3286749Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3287052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3287128Z layer_outputs = layer_module( 2025-08-14T21:51:33.3287464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.3287651Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.3287961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.3288094Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.3288394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.3288490Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3288493Z 2025-08-14T21:51:33.3288601Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3288820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3288889Z return mod(**inputs) 2025-08-14T21:51:33.3289190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3289272Z outputs = self.mobilebert( 2025-08-14T21:51:33.3289560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3289642Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3289941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3290015Z layer_outputs = layer_module( 2025-08-14T21:51:33.3290329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.3290487Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.3290774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.3290908Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.3291196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.3291328Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3291620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3291713Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3291717Z 2025-08-14T21:51:33.3291833Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3292037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3292112Z return mod(**inputs) 2025-08-14T21:51:33.3292400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3292495Z outputs = self.mobilebert( 2025-08-14T21:51:33.3292797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3292874Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3293180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3293263Z layer_outputs = layer_module( 2025-08-14T21:51:33.3293553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.3293725Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.3294035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.3294170Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.3294470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.3294555Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.3294561Z 2025-08-14T21:51:33.3294675Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3294880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3294950Z return mod(**inputs) 2025-08-14T21:51:33.3295248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3295321Z outputs = self.mobilebert( 2025-08-14T21:51:33.3295618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3295696Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3295987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3296071Z layer_outputs = layer_module( 2025-08-14T21:51:33.3296358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.3296447Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.3296743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.3296819Z self_outputs = self.self( 2025-08-14T21:51:33.3297117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:51:33.3297191Z self.value(value_tensor) 2025-08-14T21:51:33.3297196Z 2025-08-14T21:51:33.3297303Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3297517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3297585Z return mod(**inputs) 2025-08-14T21:51:33.3297876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3297959Z outputs = self.mobilebert( 2025-08-14T21:51:33.3298247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3298329Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3298616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3298689Z layer_outputs = layer_module( 2025-08-14T21:51:33.3298985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.3299174Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.3299473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:51:33.3299605Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:51:33.3299895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:51:33.3299987Z layer_input = self.dense(hidden_states) 2025-08-14T21:51:33.3299991Z 2025-08-14T21:51:33.3300095Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3300309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3300394Z return mod(**inputs) 2025-08-14T21:51:33.3300706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3300791Z outputs = self.mobilebert( 2025-08-14T21:51:33.3301087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3301164Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3301465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3301540Z layer_outputs = layer_module( 2025-08-14T21:51:33.3301841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:51:33.3302006Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:51:33.3302303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:51:33.3302437Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:51:33.3302729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:51:33.3302825Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:51:33.3303119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3303214Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3303217Z 2025-08-14T21:51:33.3303332Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3303540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3303609Z return mod(**inputs) 2025-08-14T21:51:33.3303907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3303983Z outputs = self.mobilebert( 2025-08-14T21:51:33.3304279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3304354Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3304647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3304730Z layer_outputs = layer_module( 2025-08-14T21:51:33.3305019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.3305114Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.3305406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.3305504Z self_outputs = self.self( 2025-08-14T21:51:33.3305804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:51:33.3305877Z self.query(query_tensor) 2025-08-14T21:51:33.3305881Z 2025-08-14T21:51:33.3305987Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3306223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3306289Z return mod(**inputs) 2025-08-14T21:51:33.3306583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3306655Z outputs = self.mobilebert( 2025-08-14T21:51:33.3306944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3307044Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3307354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3307438Z layer_outputs = layer_module( 2025-08-14T21:51:33.3307731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.3307822Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.3308133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:51:33.3308208Z self_outputs = self.self( 2025-08-14T21:51:33.3308505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:51:33.3308586Z self.key(key_tensor) 2025-08-14T21:51:33.3308590Z 2025-08-14T21:51:33.3308678Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.3308770Z cudagraph partition due to non gpu ops 2025-08-14T21:51:33.3308878Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3309091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3309170Z return mod(**inputs) 2025-08-14T21:51:33.3309469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3309546Z outputs = self.mobilebert( 2025-08-14T21:51:33.3309855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3309933Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3310241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3310319Z layer_outputs = layer_module( 2025-08-14T21:51:33.3310622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.3310722Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.3311023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.3311165Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.3311464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:51:33.3311554Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3311558Z 2025-08-14T21:51:33.3311673Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3311887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3311959Z return mod(**inputs) 2025-08-14T21:51:33.3312297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3312372Z outputs = self.mobilebert( 2025-08-14T21:51:33.3312676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3312773Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3313073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3313156Z layer_outputs = layer_module( 2025-08-14T21:51:33.3313455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:51:33.3313560Z self_attention_outputs = self.attention( 2025-08-14T21:51:33.3313867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:51:33.3314013Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:51:33.3314314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:51:33.3314444Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3314739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3314844Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3314847Z 2025-08-14T21:51:33.3314953Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3315180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3315249Z return mod(**inputs) 2025-08-14T21:51:33.3315549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3315635Z outputs = self.mobilebert( 2025-08-14T21:51:33.3315933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3316026Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3316320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3316394Z layer_outputs = layer_module( 2025-08-14T21:51:33.3316693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3316792Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3317087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3317212Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3317504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.3317596Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.3317601Z 2025-08-14T21:51:33.3317705Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3317916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3317994Z return mod(**inputs) 2025-08-14T21:51:33.3318294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3318376Z outputs = self.mobilebert( 2025-08-14T21:51:33.3318674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3318754Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3319086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3319164Z layer_outputs = layer_module( 2025-08-14T21:51:33.3319463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3319595Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3319890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3320017Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3320314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.3320461Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.3320467Z 2025-08-14T21:51:33.3320585Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3320822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3320901Z return mod(**inputs) 2025-08-14T21:51:33.3321201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3321280Z outputs = self.mobilebert( 2025-08-14T21:51:33.3321585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3321662Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3321968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3322046Z layer_outputs = layer_module( 2025-08-14T21:51:33.3322344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3322453Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3322754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3322885Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3323195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.3323285Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3323288Z 2025-08-14T21:51:33.3323402Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3323616Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3323686Z return mod(**inputs) 2025-08-14T21:51:33.3323991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3324069Z outputs = self.mobilebert( 2025-08-14T21:51:33.3324372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3324449Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3324751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3324840Z layer_outputs = layer_module( 2025-08-14T21:51:33.3325139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3325239Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3325636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3325805Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3326119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.3326248Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3326571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3326680Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3326684Z 2025-08-14T21:51:33.3326792Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3327014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3327085Z return mod(**inputs) 2025-08-14T21:51:33.3327405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3327495Z outputs = self.mobilebert( 2025-08-14T21:51:33.3327820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3327903Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3328218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3328296Z layer_outputs = layer_module( 2025-08-14T21:51:33.3328605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3328705Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3329008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3329136Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3329442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.3329541Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.3329544Z 2025-08-14T21:51:33.3329654Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3329871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3329951Z return mod(**inputs) 2025-08-14T21:51:33.3330254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3330330Z outputs = self.mobilebert( 2025-08-14T21:51:33.3330636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3330715Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3331026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3331104Z layer_outputs = layer_module( 2025-08-14T21:51:33.3331403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3331512Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3331814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3331940Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3332244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.3332364Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.3332386Z 2025-08-14T21:51:33.3332505Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3332719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3332789Z return mod(**inputs) 2025-08-14T21:51:33.3333095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3333193Z outputs = self.mobilebert( 2025-08-14T21:51:33.3333506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3333583Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3333884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3333968Z layer_outputs = layer_module( 2025-08-14T21:51:33.3334296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3334431Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3334733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3334866Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3335177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.3335266Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3335270Z 2025-08-14T21:51:33.3335387Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3335601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3335673Z return mod(**inputs) 2025-08-14T21:51:33.3335983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3336061Z outputs = self.mobilebert( 2025-08-14T21:51:33.3336360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3336444Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3336744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3336827Z layer_outputs = layer_module( 2025-08-14T21:51:33.3337127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3337225Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3337538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3337866Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3338183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.3338314Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3338615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3338721Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3338725Z 2025-08-14T21:51:33.3338835Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3339045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3339127Z return mod(**inputs) 2025-08-14T21:51:33.3339425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3339583Z outputs = self.mobilebert( 2025-08-14T21:51:33.3339885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3339964Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3340268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3340373Z layer_outputs = layer_module( 2025-08-14T21:51:33.3340686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3340783Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3341086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3341242Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3341576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.3341670Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.3341682Z 2025-08-14T21:51:33.3341792Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3342004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3342085Z return mod(**inputs) 2025-08-14T21:51:33.3342386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3342461Z outputs = self.mobilebert( 2025-08-14T21:51:33.3342767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3342849Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3343157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3343233Z layer_outputs = layer_module( 2025-08-14T21:51:33.3343531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3343638Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3343938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:51:33.3344056Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:51:33.3344363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.3344481Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.3344485Z 2025-08-14T21:51:33.3344602Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3344816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3344886Z return mod(**inputs) 2025-08-14T21:51:33.3345192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3345269Z outputs = self.mobilebert( 2025-08-14T21:51:33.3345575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3345662Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3345953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3346035Z layer_outputs = layer_module( 2025-08-14T21:51:33.3346323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3346443Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3346746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3346873Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3347217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:51:33.3347305Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3347309Z 2025-08-14T21:51:33.3347414Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3347627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3347696Z return mod(**inputs) 2025-08-14T21:51:33.3348039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3348132Z outputs = self.mobilebert( 2025-08-14T21:51:33.3348427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3348510Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3348817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3348892Z layer_outputs = layer_module( 2025-08-14T21:51:33.3349214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:51:33.3349312Z attention_output = ffn_module(attention_output) 2025-08-14T21:51:33.3349631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:51:33.3349766Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:51:33.3350069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:51:33.3350207Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3350508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3350613Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3350617Z 2025-08-14T21:51:33.3350726Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3350937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3351014Z return mod(**inputs) 2025-08-14T21:51:33.3351316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3351400Z outputs = self.mobilebert( 2025-08-14T21:51:33.3351706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3351783Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3352096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3352173Z layer_outputs = layer_module( 2025-08-14T21:51:33.3352473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.3352608Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.3352912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:51:33.3353008Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.3353031Z 2025-08-14T21:51:33.3353142Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3353356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3353436Z return mod(**inputs) 2025-08-14T21:51:33.3353736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3353839Z outputs = self.mobilebert( 2025-08-14T21:51:33.3354142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3354220Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3354532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3354625Z layer_outputs = layer_module( 2025-08-14T21:51:33.3354934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:51:33.3355069Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:51:33.3355364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:51:33.3355488Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:51:33.3355492Z 2025-08-14T21:51:33.3355599Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3355805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3355883Z return mod(**inputs) 2025-08-14T21:51:33.3356174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3356258Z outputs = self.mobilebert( 2025-08-14T21:51:33.3356560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3356638Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3356951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3357026Z layer_outputs = layer_module( 2025-08-14T21:51:33.3357317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.3357492Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.3357781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:51:33.3357886Z layer_output = self.dense(intermediate_states) 2025-08-14T21:51:33.3357891Z 2025-08-14T21:51:33.3357997Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3358209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3358287Z return mod(**inputs) 2025-08-14T21:51:33.3358584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3358668Z outputs = self.mobilebert( 2025-08-14T21:51:33.3358964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3359040Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3359346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3359423Z layer_outputs = layer_module( 2025-08-14T21:51:33.3359722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.3359923Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.3360222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:51:33.3360360Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:51:33.3360680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3360779Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3360783Z 2025-08-14T21:51:33.3360899Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3361111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3361191Z return mod(**inputs) 2025-08-14T21:51:33.3361516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3361611Z outputs = self.mobilebert( 2025-08-14T21:51:33.3361922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3362000Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3362300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3362382Z layer_outputs = layer_module( 2025-08-14T21:51:33.3362678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.3362852Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.3363151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.3363287Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.3363594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:51:33.3363684Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:51:33.3363689Z 2025-08-14T21:51:33.3363808Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3364018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3364088Z return mod(**inputs) 2025-08-14T21:51:33.3364396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 976, in forward 2025-08-14T21:51:33.3364473Z outputs = self.mobilebert( 2025-08-14T21:51:33.3364778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:51:33.3364858Z encoder_outputs = self.encoder( 2025-08-14T21:51:33.3365157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:51:33.3365242Z layer_outputs = layer_module( 2025-08-14T21:51:33.3365618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:51:33.3365794Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:51:33.3366109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:51:33.3366240Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:51:33.3366554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:51:33.3366713Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:51:33.3367019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:51:33.3367132Z return input_tensor * self.weight + self.bias 2025-08-14T21:51:33.3367153Z 2025-08-14T21:51:33.3367265Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3367491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3367563Z return mod(**inputs) 2025-08-14T21:51:33.3367864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-08-14T21:51:33.3367974Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:51:33.3368296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-08-14T21:51:33.3368436Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:51:33.3368748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 631, in forward 2025-08-14T21:51:33.3368846Z hidden_states = self.transform(hidden_states) 2025-08-14T21:51:33.3369156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 609, in forward 2025-08-14T21:51:33.3369244Z hidden_states = self.dense(hidden_states) 2025-08-14T21:51:33.3369248Z 2025-08-14T21:51:33.3369357Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3369576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3369650Z return mod(**inputs) 2025-08-14T21:51:33.3369953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-08-14T21:51:33.3370050Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:51:33.3370350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-08-14T21:51:33.3370473Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:51:33.3370775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 632, in forward 2025-08-14T21:51:33.3371005Z hidden_states = hidden_states.matmul(torch.cat([self.decoder.weight.t(), self.dense.weight], dim=0)) 2025-08-14T21:51:33.3371009Z 2025-08-14T21:51:33.3371118Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3371331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3371406Z return mod(**inputs) 2025-08-14T21:51:33.3371709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 989, in forward 2025-08-14T21:51:33.3371807Z prediction_scores = self.cls(sequence_output) 2025-08-14T21:51:33.3372109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 643, in forward 2025-08-14T21:51:33.3372225Z prediction_scores = self.predictions(sequence_output) 2025-08-14T21:51:33.3372527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 633, in forward 2025-08-14T21:51:33.3372614Z hidden_states += self.decoder.bias 2025-08-14T21:51:33.3372618Z 2025-08-14T21:51:33.3372724Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:51:33.3372945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:51:33.3373015Z return mod(**inputs) 2025-08-14T21:51:33.3373318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 994, in forward 2025-08-14T21:51:33.3373548Z masked_lm_loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:51:33.3373552Z 2025-08-14T21:51:45.7653698Z Compilation time (from dynamo_timed): 37.058442889 2025-08-14T21:51:45.7676040Z pass 2025-08-14T21:51:45.7681799Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:51:45.7682781Z TIMING: _recursive_pre_grad_passes:0.02181 _recursive_joint_graph_passes:1.35 _recursive_post_grad_passes:0.22536 async_compile.wait:0.71694 code_gen:9.47435 inductor_compile:13.85666 backend_compile:25.79376 gc:0.00069 entire_frame_compile:37.05844 total_wall_time:37.05844 2025-08-14T21:51:45.7683877Z STATS: call_* op count: 1449 | FakeTensorMode.__torch_dispatch__:56776 | FakeTensor.__torch_dispatch__:16414 | ProxyTorchDispatchMode.__torch_dispatch__:21632 2025-08-14T21:51:45.7684394Z Dynamo produced 1 graphs covering 1449 ops with 0 graph breaks (0 unique) 2025-08-14T21:51:51.9797010Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:51:51.9798174Z from pkg_resources import resource_filename 2025-08-14T21:51:52.5444433Z 2025-08-14T21:51:53.1497421Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:51:53.1501651Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:51:53.1565267Z cpu eval MobileBertForQuestionAnswering 2025-08-14T21:51:53.3701666Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:51:53.5066523Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:51:53.6364520Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:52:19.5293074Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5293676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5294093Z return mod(**inputs) 2025-08-14T21:52:19.5294607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5295160Z outputs = self.mobilebert( 2025-08-14T21:52:19.5295646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-08-14T21:52:19.5296133Z embedding_output = self.embeddings( 2025-08-14T21:52:19.5296585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 199, in forward 2025-08-14T21:52:19.5297061Z inputs_embeds = torch.cat( 2025-08-14T21:52:19.5297181Z 2025-08-14T21:52:19.5297307Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5297730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5298095Z return mod(**inputs) 2025-08-14T21:52:19.5298532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5298970Z outputs = self.mobilebert( 2025-08-14T21:52:19.5299392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-08-14T21:52:19.5299857Z embedding_output = self.embeddings( 2025-08-14T21:52:19.5300335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 208, in forward 2025-08-14T21:52:19.5300843Z inputs_embeds = self.embedding_transformation(inputs_embeds) 2025-08-14T21:52:19.5301925Z 2025-08-14T21:52:19.5302039Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5302414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5302743Z return mod(**inputs) 2025-08-14T21:52:19.5303145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5303623Z outputs = self.mobilebert( 2025-08-14T21:52:19.5304023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 791, in forward 2025-08-14T21:52:19.5304438Z embedding_output = self.embeddings( 2025-08-14T21:52:19.5304881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 215, in forward 2025-08-14T21:52:19.5305400Z embeddings = self.LayerNorm(embeddings) 2025-08-14T21:52:19.5305869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5306311Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5306463Z 2025-08-14T21:52:19.5306568Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5306929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5307262Z return mod(**inputs) 2025-08-14T21:52:19.5307656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5308140Z outputs = self.mobilebert( 2025-08-14T21:52:19.5308538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5308955Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5309409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5309825Z layer_outputs = layer_module( 2025-08-14T21:52:19.5310242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.5310746Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.5311245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.5311699Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.5312144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.5312582Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.5312719Z 2025-08-14T21:52:19.5312824Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5313182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5313511Z return mod(**inputs) 2025-08-14T21:52:19.5313902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5314310Z outputs = self.mobilebert( 2025-08-14T21:52:19.5314711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5315127Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5315535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5315937Z layer_outputs = layer_module( 2025-08-14T21:52:19.5316330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.5316840Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.5317313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.5317748Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.5318214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.5318629Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.5319034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5319451Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5319592Z 2025-08-14T21:52:19.5319716Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5320075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5320416Z return mod(**inputs) 2025-08-14T21:52:19.5320808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5321221Z outputs = self.mobilebert( 2025-08-14T21:52:19.5321610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5322023Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5322426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5322841Z layer_outputs = layer_module( 2025-08-14T21:52:19.5323238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5323665Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5324111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.5324545Z self_outputs = self.self( 2025-08-14T21:52:19.5324964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.5325557Z self.query(query_tensor) 2025-08-14T21:52:19.5325692Z 2025-08-14T21:52:19.5325816Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5326208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5326588Z return mod(**inputs) 2025-08-14T21:52:19.5327015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5327489Z outputs = self.mobilebert( 2025-08-14T21:52:19.5327879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5328296Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5328704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5329107Z layer_outputs = layer_module( 2025-08-14T21:52:19.5329528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5329978Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5330395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.5330795Z self_outputs = self.self( 2025-08-14T21:52:19.5331188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.5331621Z self.key(key_tensor) 2025-08-14T21:52:19.5331727Z 2025-08-14T21:52:19.5331837Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5332189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5332531Z return mod(**inputs) 2025-08-14T21:52:19.5332917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5333322Z outputs = self.mobilebert( 2025-08-14T21:52:19.5333746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5334178Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5334621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5335058Z layer_outputs = layer_module( 2025-08-14T21:52:19.5335504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5335933Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5336349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.5336769Z self_outputs = self.self( 2025-08-14T21:52:19.5337185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.5337794Z self.value(value_tensor) 2025-08-14T21:52:19.5337929Z 2025-08-14T21:52:19.5338021Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.5338257Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.5338513Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5338901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5339238Z return mod(**inputs) 2025-08-14T21:52:19.5339631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5340084Z outputs = self.mobilebert( 2025-08-14T21:52:19.5340486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5340897Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5341298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5341716Z layer_outputs = layer_module( 2025-08-14T21:52:19.5342128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5342559Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5342992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.5343484Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.5343941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.5344365Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5344510Z 2025-08-14T21:52:19.5344617Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5344995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5345346Z return mod(**inputs) 2025-08-14T21:52:19.5345755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5346254Z outputs = self.mobilebert( 2025-08-14T21:52:19.5346670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5347100Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5347522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5347989Z layer_outputs = layer_module( 2025-08-14T21:52:19.5348438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.5348971Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.5349508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.5350044Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.5350550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.5351001Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.5351154Z 2025-08-14T21:52:19.5351261Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5351639Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5351990Z return mod(**inputs) 2025-08-14T21:52:19.5352391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5352829Z outputs = self.mobilebert( 2025-08-14T21:52:19.5353310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5353751Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5354181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5354621Z layer_outputs = layer_module( 2025-08-14T21:52:19.5355046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5355498Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5355944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.5356438Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.5356917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.5357411Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5357899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5358365Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5358517Z 2025-08-14T21:52:19.5358632Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5359013Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5359364Z return mod(**inputs) 2025-08-14T21:52:19.5359773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5360211Z outputs = self.mobilebert( 2025-08-14T21:52:19.5360755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5361208Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5361637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5362094Z layer_outputs = layer_module( 2025-08-14T21:52:19.5362520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5363013Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5363498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5363991Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5364489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5364938Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5365086Z 2025-08-14T21:52:19.5365247Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5365700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5366097Z return mod(**inputs) 2025-08-14T21:52:19.5366534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5366985Z outputs = self.mobilebert( 2025-08-14T21:52:19.5367411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5367859Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5368283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5368719Z layer_outputs = layer_module( 2025-08-14T21:52:19.5369151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5369618Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5370073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5370531Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5370979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5371433Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5371600Z 2025-08-14T21:52:19.5371702Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5372073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5372435Z return mod(**inputs) 2025-08-14T21:52:19.5372825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5373239Z outputs = self.mobilebert( 2025-08-14T21:52:19.5373637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5374045Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5374440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5374849Z layer_outputs = layer_module( 2025-08-14T21:52:19.5375247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5375677Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5376096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5376556Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5377062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.5377494Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5377639Z 2025-08-14T21:52:19.5377747Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5378152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5378494Z return mod(**inputs) 2025-08-14T21:52:19.5378901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5379343Z outputs = self.mobilebert( 2025-08-14T21:52:19.5379739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5380170Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5380583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5380994Z layer_outputs = layer_module( 2025-08-14T21:52:19.5381398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5381826Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5382242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5382722Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5383203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.5383682Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5384163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5384612Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5384766Z 2025-08-14T21:52:19.5384882Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5385250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5385594Z return mod(**inputs) 2025-08-14T21:52:19.5385988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5386397Z outputs = self.mobilebert( 2025-08-14T21:52:19.5386795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5387226Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5387652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5388082Z layer_outputs = layer_module( 2025-08-14T21:52:19.5388517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5388976Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5389433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5389893Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5390360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5390808Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5390954Z 2025-08-14T21:52:19.5391075Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5391481Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5391834Z return mod(**inputs) 2025-08-14T21:52:19.5392243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5392678Z outputs = self.mobilebert( 2025-08-14T21:52:19.5393107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5393547Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5393966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5394398Z layer_outputs = layer_module( 2025-08-14T21:52:19.5394843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5395303Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5395777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5396231Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5396674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5397129Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5397296Z 2025-08-14T21:52:19.5397407Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5397760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5398084Z return mod(**inputs) 2025-08-14T21:52:19.5398476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5398883Z outputs = self.mobilebert( 2025-08-14T21:52:19.5399279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5399697Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5400122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5400532Z layer_outputs = layer_module( 2025-08-14T21:52:19.5400938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5401372Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5401803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5402310Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5402802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.5403251Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5403400Z 2025-08-14T21:52:19.5403511Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5403899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5404246Z return mod(**inputs) 2025-08-14T21:52:19.5404659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5405090Z outputs = self.mobilebert( 2025-08-14T21:52:19.5405609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5406054Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5406506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5406949Z layer_outputs = layer_module( 2025-08-14T21:52:19.5407360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5407832Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5408256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5408717Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5409175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.5409629Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5410115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5410573Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5410721Z 2025-08-14T21:52:19.5410832Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5411183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5411498Z return mod(**inputs) 2025-08-14T21:52:19.5411884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5412289Z outputs = self.mobilebert( 2025-08-14T21:52:19.5412669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5413085Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5413500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5413901Z layer_outputs = layer_module( 2025-08-14T21:52:19.5414289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5414717Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5415146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5415594Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5416035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5416445Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5416579Z 2025-08-14T21:52:19.5416687Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5417024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5417347Z return mod(**inputs) 2025-08-14T21:52:19.5417721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5418119Z outputs = self.mobilebert( 2025-08-14T21:52:19.5418497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5418899Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5419291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5419689Z layer_outputs = layer_module( 2025-08-14T21:52:19.5420080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5420514Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5420923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5421352Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5421783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5422250Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5422423Z 2025-08-14T21:52:19.5422539Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5422903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5423238Z return mod(**inputs) 2025-08-14T21:52:19.5423666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5424078Z outputs = self.mobilebert( 2025-08-14T21:52:19.5424479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5424893Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5425314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5425733Z layer_outputs = layer_module( 2025-08-14T21:52:19.5426154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5426621Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5427057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5427510Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5427977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.5428381Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5428512Z 2025-08-14T21:52:19.5428619Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5428956Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5429274Z return mod(**inputs) 2025-08-14T21:52:19.5429660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5430058Z outputs = self.mobilebert( 2025-08-14T21:52:19.5430473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5430880Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5431288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5431686Z layer_outputs = layer_module( 2025-08-14T21:52:19.5432086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5432514Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5432956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5433439Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5433916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.5434395Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5434860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5435310Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5435464Z 2025-08-14T21:52:19.5435568Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5435934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5436282Z return mod(**inputs) 2025-08-14T21:52:19.5436688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5437122Z outputs = self.mobilebert( 2025-08-14T21:52:19.5437536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5438120Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5439276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5439769Z layer_outputs = layer_module( 2025-08-14T21:52:19.5440198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.5440684Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.5441173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5441621Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5441771Z 2025-08-14T21:52:19.5441878Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5442254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5442593Z return mod(**inputs) 2025-08-14T21:52:19.5443008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5443444Z outputs = self.mobilebert( 2025-08-14T21:52:19.5443874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5444335Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5444767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5445201Z layer_outputs = layer_module( 2025-08-14T21:52:19.5445688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.5446176Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.5446655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5447105Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5447276Z 2025-08-14T21:52:19.5447383Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5447742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5448060Z return mod(**inputs) 2025-08-14T21:52:19.5448452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5448863Z outputs = self.mobilebert( 2025-08-14T21:52:19.5449250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5449667Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5450071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5450478Z layer_outputs = layer_module( 2025-08-14T21:52:19.5450906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5451400Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5451892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.5452348Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.5452496Z 2025-08-14T21:52:19.5452598Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5452953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5453280Z return mod(**inputs) 2025-08-14T21:52:19.5453676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5454099Z outputs = self.mobilebert( 2025-08-14T21:52:19.5454516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5454930Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5455325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5455741Z layer_outputs = layer_module( 2025-08-14T21:52:19.5456141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5456630Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5457114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.5457573Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.5458037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5458456Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5458596Z 2025-08-14T21:52:19.5458695Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5459048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5459375Z return mod(**inputs) 2025-08-14T21:52:19.5459757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5460164Z outputs = self.mobilebert( 2025-08-14T21:52:19.5460555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5460972Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5461357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5461752Z layer_outputs = layer_module( 2025-08-14T21:52:19.5463067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5463566Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5464047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.5464500Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.5464957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.5465377Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5465552Z 2025-08-14T21:52:19.5465653Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5466008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5466344Z return mod(**inputs) 2025-08-14T21:52:19.5466744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5467196Z outputs = self.mobilebert( 2025-08-14T21:52:19.5467616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5468048Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5468460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5468888Z layer_outputs = layer_module( 2025-08-14T21:52:19.5469305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5469835Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5470323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.5470780Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.5471248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.5471721Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5472205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5472654Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5472807Z 2025-08-14T21:52:19.5472925Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5473297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5473638Z return mod(**inputs) 2025-08-14T21:52:19.5474046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5474484Z outputs = self.mobilebert( 2025-08-14T21:52:19.5474895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5475327Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5475750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5476174Z layer_outputs = layer_module( 2025-08-14T21:52:19.5476603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.5477142Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.5477662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.5478131Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.5478600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.5479041Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.5479182Z 2025-08-14T21:52:19.5479297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5479672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5480015Z return mod(**inputs) 2025-08-14T21:52:19.5480422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5480877Z outputs = self.mobilebert( 2025-08-14T21:52:19.5481293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5481727Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5482177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5482636Z layer_outputs = layer_module( 2025-08-14T21:52:19.5483080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.5483625Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.5484184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.5484683Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.5485166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.5485723Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.5486187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5486668Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5486840Z 2025-08-14T21:52:19.5486961Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5487343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5487683Z return mod(**inputs) 2025-08-14T21:52:19.5488103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5488544Z outputs = self.mobilebert( 2025-08-14T21:52:19.5488973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5489384Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5489820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5490260Z layer_outputs = layer_module( 2025-08-14T21:52:19.5490656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5491083Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5491504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.5491918Z self_outputs = self.self( 2025-08-14T21:52:19.5492317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.5492732Z self.query(query_tensor) 2025-08-14T21:52:19.5492847Z 2025-08-14T21:52:19.5492955Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5493310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5493632Z return mod(**inputs) 2025-08-14T21:52:19.5494019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5494433Z outputs = self.mobilebert( 2025-08-14T21:52:19.5494818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5495231Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5495704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5496114Z layer_outputs = layer_module( 2025-08-14T21:52:19.5496506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5496948Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5497361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.5497765Z self_outputs = self.self( 2025-08-14T21:52:19.5498150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.5498549Z self.key(key_tensor) 2025-08-14T21:52:19.5498653Z 2025-08-14T21:52:19.5498779Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5499134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5499476Z return mod(**inputs) 2025-08-14T21:52:19.5499868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5500277Z outputs = self.mobilebert( 2025-08-14T21:52:19.5500667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5501084Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5501490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5501897Z layer_outputs = layer_module( 2025-08-14T21:52:19.5502303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5502734Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5503164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.5503572Z self_outputs = self.self( 2025-08-14T21:52:19.5503969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.5504383Z self.value(value_tensor) 2025-08-14T21:52:19.5504496Z 2025-08-14T21:52:19.5504584Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.5504794Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.5505033Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5505392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5505709Z return mod(**inputs) 2025-08-14T21:52:19.5506107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5506536Z outputs = self.mobilebert( 2025-08-14T21:52:19.5506961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5507387Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5507821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5508269Z layer_outputs = layer_module( 2025-08-14T21:52:19.5508686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5509149Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5509610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.5510110Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.5510610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.5511055Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5511199Z 2025-08-14T21:52:19.5511313Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5511707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5512040Z return mod(**inputs) 2025-08-14T21:52:19.5512450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5512886Z outputs = self.mobilebert( 2025-08-14T21:52:19.5513363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5513819Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5514273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5514708Z layer_outputs = layer_module( 2025-08-14T21:52:19.5515124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.5515647Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.5516167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.5516628Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.5517051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.5517466Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.5517603Z 2025-08-14T21:52:19.5517711Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5518061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5518381Z return mod(**inputs) 2025-08-14T21:52:19.5518775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5519186Z outputs = self.mobilebert( 2025-08-14T21:52:19.5519573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5519983Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5520384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5520788Z layer_outputs = layer_module( 2025-08-14T21:52:19.5521179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5521602Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5522032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.5522517Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.5523004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.5523493Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5523978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5524421Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5524583Z 2025-08-14T21:52:19.5524690Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5525093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5525508Z return mod(**inputs) 2025-08-14T21:52:19.5525940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5526415Z outputs = self.mobilebert( 2025-08-14T21:52:19.5526837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5527265Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5527694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5528083Z layer_outputs = layer_module( 2025-08-14T21:52:19.5528478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5528903Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5529318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5529755Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5530189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5530590Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5530730Z 2025-08-14T21:52:19.5530830Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5531183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5531493Z return mod(**inputs) 2025-08-14T21:52:19.5531877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5532279Z outputs = self.mobilebert( 2025-08-14T21:52:19.5532666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5533064Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5533471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5533886Z layer_outputs = layer_module( 2025-08-14T21:52:19.5534275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5534690Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5535147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5535624Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5536065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5536518Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5536686Z 2025-08-14T21:52:19.5536801Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5537154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5537465Z return mod(**inputs) 2025-08-14T21:52:19.5537970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5538425Z outputs = self.mobilebert( 2025-08-14T21:52:19.5538845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5539323Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5539738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5540149Z layer_outputs = layer_module( 2025-08-14T21:52:19.5540538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5540999Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5541435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5541901Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5542354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.5542812Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5542964Z 2025-08-14T21:52:19.5543069Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5543453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5543770Z return mod(**inputs) 2025-08-14T21:52:19.5544160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5544578Z outputs = self.mobilebert( 2025-08-14T21:52:19.5544962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5545373Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5545771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5546179Z layer_outputs = layer_module( 2025-08-14T21:52:19.5546573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5547008Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5547435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5547890Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5548366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.5548853Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5549306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5549731Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5549884Z 2025-08-14T21:52:19.5549994Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5550384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5550705Z return mod(**inputs) 2025-08-14T21:52:19.5551089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5551504Z outputs = self.mobilebert( 2025-08-14T21:52:19.5551916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5552346Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5552761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5553197Z layer_outputs = layer_module( 2025-08-14T21:52:19.5553623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5554078Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5554505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5554959Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5555449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5555872Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5556019Z 2025-08-14T21:52:19.5556121Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5556475Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5556795Z return mod(**inputs) 2025-08-14T21:52:19.5557190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5557605Z outputs = self.mobilebert( 2025-08-14T21:52:19.5558019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5558430Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5558829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5559237Z layer_outputs = layer_module( 2025-08-14T21:52:19.5559635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5560058Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5560492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5560962Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5561430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5561893Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5562067Z 2025-08-14T21:52:19.5562173Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5562552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5562898Z return mod(**inputs) 2025-08-14T21:52:19.5563313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5563758Z outputs = self.mobilebert( 2025-08-14T21:52:19.5564187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5564627Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5565072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5565600Z layer_outputs = layer_module( 2025-08-14T21:52:19.5566043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5566509Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5566971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5567433Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5567892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.5568347Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5568533Z 2025-08-14T21:52:19.5568645Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5569033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5569371Z return mod(**inputs) 2025-08-14T21:52:19.5569791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5570261Z outputs = self.mobilebert( 2025-08-14T21:52:19.5570690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5571131Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5571570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5572014Z layer_outputs = layer_module( 2025-08-14T21:52:19.5572486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5572980Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5573446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5573944Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5574435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.5574934Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5575409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5575857Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5576009Z 2025-08-14T21:52:19.5576118Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5576493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5576828Z return mod(**inputs) 2025-08-14T21:52:19.5577235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5577670Z outputs = self.mobilebert( 2025-08-14T21:52:19.5578084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5578527Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5578942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5579381Z layer_outputs = layer_module( 2025-08-14T21:52:19.5579800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5580265Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5580710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5581195Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5581659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5582110Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5582254Z 2025-08-14T21:52:19.5582360Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5582733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5583074Z return mod(**inputs) 2025-08-14T21:52:19.5583473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5583931Z outputs = self.mobilebert( 2025-08-14T21:52:19.5584343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5584752Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5585147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5585572Z layer_outputs = layer_module( 2025-08-14T21:52:19.5585968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5586399Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5586816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5587282Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5587744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5588185Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5588357Z 2025-08-14T21:52:19.5588461Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5588816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5589134Z return mod(**inputs) 2025-08-14T21:52:19.5589512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5589920Z outputs = self.mobilebert( 2025-08-14T21:52:19.5590308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5590714Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5591106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5591514Z layer_outputs = layer_module( 2025-08-14T21:52:19.5591908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5592333Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5592763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5593253Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5593728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.5594142Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5594287Z 2025-08-14T21:52:19.5594387Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5594744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5595066Z return mod(**inputs) 2025-08-14T21:52:19.5595448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5595858Z outputs = self.mobilebert( 2025-08-14T21:52:19.5596247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5596645Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5597048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5597448Z layer_outputs = layer_module( 2025-08-14T21:52:19.5597849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5598300Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5598728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5599184Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5599694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.5600141Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5600592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5601019Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5601163Z 2025-08-14T21:52:19.5601282Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5601655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5601980Z return mod(**inputs) 2025-08-14T21:52:19.5602370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5602779Z outputs = self.mobilebert( 2025-08-14T21:52:19.5603175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5603589Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5603986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5604396Z layer_outputs = layer_module( 2025-08-14T21:52:19.5604798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.5605258Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.5605803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5606273Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5606436Z 2025-08-14T21:52:19.5606548Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5606921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5607234Z return mod(**inputs) 2025-08-14T21:52:19.5607628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5608048Z outputs = self.mobilebert( 2025-08-14T21:52:19.5608426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5608830Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5609230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5609627Z layer_outputs = layer_module( 2025-08-14T21:52:19.5610006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.5610446Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.5610882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5611316Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5611473Z 2025-08-14T21:52:19.5611571Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5611916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5612250Z return mod(**inputs) 2025-08-14T21:52:19.5612619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5613024Z outputs = self.mobilebert( 2025-08-14T21:52:19.5613408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5613830Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5614215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5614613Z layer_outputs = layer_module( 2025-08-14T21:52:19.5615006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5615500Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5615984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.5616393Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.5616533Z 2025-08-14T21:52:19.5616638Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5616975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5617271Z return mod(**inputs) 2025-08-14T21:52:19.5617641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5618029Z outputs = self.mobilebert( 2025-08-14T21:52:19.5618396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5618787Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5619171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5619563Z layer_outputs = layer_module( 2025-08-14T21:52:19.5619937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5620402Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5620872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.5621300Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.5621724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5622129Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5622268Z 2025-08-14T21:52:19.5622371Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5622700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5623006Z return mod(**inputs) 2025-08-14T21:52:19.5623378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5623773Z outputs = self.mobilebert( 2025-08-14T21:52:19.5624137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5624529Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5624912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5625288Z layer_outputs = layer_module( 2025-08-14T21:52:19.5625671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5626158Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5626622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.5627080Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.5627518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.5627920Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5628050Z 2025-08-14T21:52:19.5628153Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5628483Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5628804Z return mod(**inputs) 2025-08-14T21:52:19.5629200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5629588Z outputs = self.mobilebert( 2025-08-14T21:52:19.5629954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5630344Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5630721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5631098Z layer_outputs = layer_module( 2025-08-14T21:52:19.5631485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5631957Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5632436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.5632886Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.5633341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.5633799Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5634225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5634625Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5634768Z 2025-08-14T21:52:19.5634869Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5635222Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5635531Z return mod(**inputs) 2025-08-14T21:52:19.5635921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5636333Z outputs = self.mobilebert( 2025-08-14T21:52:19.5636721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5637124Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5637523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5638148Z layer_outputs = layer_module( 2025-08-14T21:52:19.5638555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.5639048Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.5639544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.5640050Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.5640486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.5640903Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.5641076Z 2025-08-14T21:52:19.5641180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5641537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5641858Z return mod(**inputs) 2025-08-14T21:52:19.5642254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5642670Z outputs = self.mobilebert( 2025-08-14T21:52:19.5643105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5643543Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5643950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5644363Z layer_outputs = layer_module( 2025-08-14T21:52:19.5644755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.5645247Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.5645807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.5646296Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.5646754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.5647174Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.5647585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5648001Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5648143Z 2025-08-14T21:52:19.5648245Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5648592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5648912Z return mod(**inputs) 2025-08-14T21:52:19.5649281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5649685Z outputs = self.mobilebert( 2025-08-14T21:52:19.5650071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5650470Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5650854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5651248Z layer_outputs = layer_module( 2025-08-14T21:52:19.5651634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5652048Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5652445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.5652840Z self_outputs = self.self( 2025-08-14T21:52:19.5653224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.5653617Z self.query(query_tensor) 2025-08-14T21:52:19.5653759Z 2025-08-14T21:52:19.5653857Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5654199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5654508Z return mod(**inputs) 2025-08-14T21:52:19.5654874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5655292Z outputs = self.mobilebert( 2025-08-14T21:52:19.5655674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5656075Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5656460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5656858Z layer_outputs = layer_module( 2025-08-14T21:52:19.5657262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5657693Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5658090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.5658477Z self_outputs = self.self( 2025-08-14T21:52:19.5658850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.5659223Z self.key(key_tensor) 2025-08-14T21:52:19.5659331Z 2025-08-14T21:52:19.5659427Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5659762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5660058Z return mod(**inputs) 2025-08-14T21:52:19.5660430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5660826Z outputs = self.mobilebert( 2025-08-14T21:52:19.5661202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5661581Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5661962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5662351Z layer_outputs = layer_module( 2025-08-14T21:52:19.5662728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5663130Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5663533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.5663934Z self_outputs = self.self( 2025-08-14T21:52:19.5664312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.5664707Z self.value(value_tensor) 2025-08-14T21:52:19.5664820Z 2025-08-14T21:52:19.5664899Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.5665104Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.5665319Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5665655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5665961Z return mod(**inputs) 2025-08-14T21:52:19.5666320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5666710Z outputs = self.mobilebert( 2025-08-14T21:52:19.5667084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5667501Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5667877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5668268Z layer_outputs = layer_module( 2025-08-14T21:52:19.5668655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5669073Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5669466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.5669912Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.5670351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.5670779Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5670920Z 2025-08-14T21:52:19.5671032Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5671368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5671671Z return mod(**inputs) 2025-08-14T21:52:19.5672030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5672429Z outputs = self.mobilebert( 2025-08-14T21:52:19.5672822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5673241Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5673624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5674026Z layer_outputs = layer_module( 2025-08-14T21:52:19.5674416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.5674896Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.5675379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.5675814Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.5676243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.5676643Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.5676784Z 2025-08-14T21:52:19.5676883Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5677228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5677539Z return mod(**inputs) 2025-08-14T21:52:19.5677910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5678311Z outputs = self.mobilebert( 2025-08-14T21:52:19.5678695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5679091Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5679483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5679878Z layer_outputs = layer_module( 2025-08-14T21:52:19.5680265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5680670Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5681074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.5681561Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.5682002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.5682444Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5682923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5683354Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5683496Z 2025-08-14T21:52:19.5683598Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5683951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5684255Z return mod(**inputs) 2025-08-14T21:52:19.5684637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5685045Z outputs = self.mobilebert( 2025-08-14T21:52:19.5685486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5685887Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5686276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5686658Z layer_outputs = layer_module( 2025-08-14T21:52:19.5687049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5687477Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5687892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5688334Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5688770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5689186Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5689332Z 2025-08-14T21:52:19.5689426Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5689753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5690052Z return mod(**inputs) 2025-08-14T21:52:19.5690408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5690780Z outputs = self.mobilebert( 2025-08-14T21:52:19.5691145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5691525Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5691892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5692272Z layer_outputs = layer_module( 2025-08-14T21:52:19.5692641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5693048Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5693444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5693865Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5694295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5694733Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5694913Z 2025-08-14T21:52:19.5695012Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5695350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5695654Z return mod(**inputs) 2025-08-14T21:52:19.5696018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5696414Z outputs = self.mobilebert( 2025-08-14T21:52:19.5696777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5697152Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5697514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5697903Z layer_outputs = layer_module( 2025-08-14T21:52:19.5698296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5698690Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5699086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5699513Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5699944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.5700337Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5700475Z 2025-08-14T21:52:19.5700571Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5700907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5701212Z return mod(**inputs) 2025-08-14T21:52:19.5701573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5701961Z outputs = self.mobilebert( 2025-08-14T21:52:19.5702332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5702714Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5703095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5703483Z layer_outputs = layer_module( 2025-08-14T21:52:19.5703859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5704261Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5704665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5705107Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5705529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.5705937Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5706356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5706750Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5706884Z 2025-08-14T21:52:19.5706984Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5707302Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5707597Z return mod(**inputs) 2025-08-14T21:52:19.5707960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5708359Z outputs = self.mobilebert( 2025-08-14T21:52:19.5708729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5709115Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5709513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5709896Z layer_outputs = layer_module( 2025-08-14T21:52:19.5710269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5710672Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5711076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5711191Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5711465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5711553Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5711557Z 2025-08-14T21:52:19.5711652Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5711833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5711902Z return mod(**inputs) 2025-08-14T21:52:19.5712156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5712230Z outputs = self.mobilebert( 2025-08-14T21:52:19.5712486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5712558Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5712834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5712907Z layer_outputs = layer_module( 2025-08-14T21:52:19.5713232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5713330Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5713590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5713701Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5713959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5714064Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5714069Z 2025-08-14T21:52:19.5714176Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5714369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5714437Z return mod(**inputs) 2025-08-14T21:52:19.5714693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5714758Z outputs = self.mobilebert( 2025-08-14T21:52:19.5715017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5715085Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5715347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5715415Z layer_outputs = layer_module( 2025-08-14T21:52:19.5715686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5715780Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5716033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5716165Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5716427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.5716506Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5716509Z 2025-08-14T21:52:19.5716610Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5716793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5716873Z return mod(**inputs) 2025-08-14T21:52:19.5717167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5717234Z outputs = self.mobilebert( 2025-08-14T21:52:19.5717496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5717564Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5717819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5717894Z layer_outputs = layer_module( 2025-08-14T21:52:19.5718149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5718235Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5718498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5718612Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5718872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.5718982Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5719238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5719329Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5719333Z 2025-08-14T21:52:19.5719425Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5719613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5719674Z return mod(**inputs) 2025-08-14T21:52:19.5719931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5720007Z outputs = self.mobilebert( 2025-08-14T21:52:19.5720262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5720330Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5720592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5720657Z layer_outputs = layer_module( 2025-08-14T21:52:19.5720915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5720998Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5721251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5721380Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5721640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5721725Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5721729Z 2025-08-14T21:52:19.5721840Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5722024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5722093Z return mod(**inputs) 2025-08-14T21:52:19.5722355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5722421Z outputs = self.mobilebert( 2025-08-14T21:52:19.5722703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5722780Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5723069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5723139Z layer_outputs = layer_module( 2025-08-14T21:52:19.5723412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5723509Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5723779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5723891Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5724160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5724267Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5724272Z 2025-08-14T21:52:19.5724378Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5724574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5724642Z return mod(**inputs) 2025-08-14T21:52:19.5724905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5724973Z outputs = self.mobilebert( 2025-08-14T21:52:19.5725242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5725311Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5725658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5725744Z layer_outputs = layer_module( 2025-08-14T21:52:19.5726011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5726109Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5726396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5726531Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5726843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.5726926Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5726930Z 2025-08-14T21:52:19.5727036Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5727223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5727289Z return mod(**inputs) 2025-08-14T21:52:19.5727587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5727655Z outputs = self.mobilebert( 2025-08-14T21:52:19.5727920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5728023Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5728349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5728425Z layer_outputs = layer_module( 2025-08-14T21:52:19.5728686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5728773Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5729058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5729191Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5729461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.5729577Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5729842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5729936Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5729939Z 2025-08-14T21:52:19.5730035Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5730223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5730292Z return mod(**inputs) 2025-08-14T21:52:19.5730554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5730629Z outputs = self.mobilebert( 2025-08-14T21:52:19.5730891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5730959Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5731231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5731297Z layer_outputs = layer_module( 2025-08-14T21:52:19.5731565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.5731679Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.5731939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5732026Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5732029Z 2025-08-14T21:52:19.5732125Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5732310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5732379Z return mod(**inputs) 2025-08-14T21:52:19.5732643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5732715Z outputs = self.mobilebert( 2025-08-14T21:52:19.5732976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5733044Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5733314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5733403Z layer_outputs = layer_module( 2025-08-14T21:52:19.5733673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.5733785Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.5734046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5734174Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5734177Z 2025-08-14T21:52:19.5734274Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5734458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5734530Z return mod(**inputs) 2025-08-14T21:52:19.5734809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5734888Z outputs = self.mobilebert( 2025-08-14T21:52:19.5735167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5735236Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5735503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5735571Z layer_outputs = layer_module( 2025-08-14T21:52:19.5735837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5735985Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5736242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.5736340Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.5736345Z 2025-08-14T21:52:19.5736440Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5736630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5736692Z return mod(**inputs) 2025-08-14T21:52:19.5736952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5737028Z outputs = self.mobilebert( 2025-08-14T21:52:19.5737287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5737355Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5737781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5737857Z layer_outputs = layer_module( 2025-08-14T21:52:19.5738138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5738287Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5738542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.5738664Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.5738921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5739014Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5739017Z 2025-08-14T21:52:19.5739113Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5739294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5739364Z return mod(**inputs) 2025-08-14T21:52:19.5739673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5739738Z outputs = self.mobilebert( 2025-08-14T21:52:19.5740002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5740097Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5740355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5740420Z layer_outputs = layer_module( 2025-08-14T21:52:19.5740676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5740826Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5741120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.5741268Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.5741525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.5741604Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5741607Z 2025-08-14T21:52:19.5741711Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5741890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5741951Z return mod(**inputs) 2025-08-14T21:52:19.5742214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5742277Z outputs = self.mobilebert( 2025-08-14T21:52:19.5742538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5742608Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5742862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5742936Z layer_outputs = layer_module( 2025-08-14T21:52:19.5743189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5743341Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5743602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.5743715Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.5743985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.5744101Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5744383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5744468Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5744472Z 2025-08-14T21:52:19.5744565Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5744753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5744816Z return mod(**inputs) 2025-08-14T21:52:19.5745077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5745150Z outputs = self.mobilebert( 2025-08-14T21:52:19.5745403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5745498Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5745761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5745827Z layer_outputs = layer_module( 2025-08-14T21:52:19.5746103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.5746251Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.5746513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.5746614Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.5746884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.5746994Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.5746998Z 2025-08-14T21:52:19.5747093Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5747272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5747341Z return mod(**inputs) 2025-08-14T21:52:19.5747599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5747671Z outputs = self.mobilebert( 2025-08-14T21:52:19.5747923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5747988Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5748248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5748317Z layer_outputs = layer_module( 2025-08-14T21:52:19.5748575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.5748720Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.5748977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.5749082Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.5749339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.5749419Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.5749685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5749772Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5749776Z 2025-08-14T21:52:19.5749884Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5750066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5750127Z return mod(**inputs) 2025-08-14T21:52:19.5750399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5750467Z outputs = self.mobilebert( 2025-08-14T21:52:19.5750729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5750798Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5751058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5751154Z layer_outputs = layer_module( 2025-08-14T21:52:19.5751424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5751534Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5751795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.5751878Z self_outputs = self.self( 2025-08-14T21:52:19.5752138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.5752205Z self.query(query_tensor) 2025-08-14T21:52:19.5752208Z 2025-08-14T21:52:19.5752303Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5752492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5752571Z return mod(**inputs) 2025-08-14T21:52:19.5752863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5752933Z outputs = self.mobilebert( 2025-08-14T21:52:19.5753198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5753278Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5753544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5753614Z layer_outputs = layer_module( 2025-08-14T21:52:19.5753890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5753968Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5754246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.5754311Z self_outputs = self.self( 2025-08-14T21:52:19.5754564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.5754634Z self.key(key_tensor) 2025-08-14T21:52:19.5754637Z 2025-08-14T21:52:19.5754729Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5754917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5754977Z return mod(**inputs) 2025-08-14T21:52:19.5755232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5755301Z outputs = self.mobilebert( 2025-08-14T21:52:19.5755553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5755624Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5755884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5755949Z layer_outputs = layer_module( 2025-08-14T21:52:19.5756206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5756283Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5756536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.5756605Z self_outputs = self.self( 2025-08-14T21:52:19.5756856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.5756928Z self.value(value_tensor) 2025-08-14T21:52:19.5756933Z 2025-08-14T21:52:19.5757057Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.5757132Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.5757234Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5757415Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5757476Z return mod(**inputs) 2025-08-14T21:52:19.5757760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5757824Z outputs = self.mobilebert( 2025-08-14T21:52:19.5758088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5758155Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5758425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5758502Z layer_outputs = layer_module( 2025-08-14T21:52:19.5758771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5758848Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5759110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.5759224Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.5759483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.5759560Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5759563Z 2025-08-14T21:52:19.5759657Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5759848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5759909Z return mod(**inputs) 2025-08-14T21:52:19.5760172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5760236Z outputs = self.mobilebert( 2025-08-14T21:52:19.5760487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5760562Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5760815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5760880Z layer_outputs = layer_module( 2025-08-14T21:52:19.5761136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.5761281Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.5761545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.5761645Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.5761896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.5761980Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.5761983Z 2025-08-14T21:52:19.5762075Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5762259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5762317Z return mod(**inputs) 2025-08-14T21:52:19.5762576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5762649Z outputs = self.mobilebert( 2025-08-14T21:52:19.5762926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5762993Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5763258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5763351Z layer_outputs = layer_module( 2025-08-14T21:52:19.5763623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5763700Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5763963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.5764085Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.5764365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.5764510Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5764775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5764860Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5764865Z 2025-08-14T21:52:19.5764968Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5765155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5765227Z return mod(**inputs) 2025-08-14T21:52:19.5765559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5765631Z outputs = self.mobilebert( 2025-08-14T21:52:19.5765900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5765971Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5766255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5766339Z layer_outputs = layer_module( 2025-08-14T21:52:19.5766636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5766749Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5767017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5767124Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5767403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5767484Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5767488Z 2025-08-14T21:52:19.5767593Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5767782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5767847Z return mod(**inputs) 2025-08-14T21:52:19.5768136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5768203Z outputs = self.mobilebert( 2025-08-14T21:52:19.5768462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5768540Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5768798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5768896Z layer_outputs = layer_module( 2025-08-14T21:52:19.5769163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5769252Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5769526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5769644Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5769910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5770014Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5770017Z 2025-08-14T21:52:19.5770112Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5770315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5770379Z return mod(**inputs) 2025-08-14T21:52:19.5770657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5770732Z outputs = self.mobilebert( 2025-08-14T21:52:19.5770990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5771065Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5771322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5771391Z layer_outputs = layer_module( 2025-08-14T21:52:19.5771655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5771740Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5772007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5772125Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5772383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.5772472Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5772476Z 2025-08-14T21:52:19.5772569Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5772753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5772823Z return mod(**inputs) 2025-08-14T21:52:19.5773086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5773158Z outputs = self.mobilebert( 2025-08-14T21:52:19.5773416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5773488Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5773754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5773823Z layer_outputs = layer_module( 2025-08-14T21:52:19.5774088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5774173Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5774434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5774556Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5774817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.5774949Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5775225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5775311Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5775331Z 2025-08-14T21:52:19.5775435Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5775620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5775683Z return mod(**inputs) 2025-08-14T21:52:19.5775955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5776021Z outputs = self.mobilebert( 2025-08-14T21:52:19.5776332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5776436Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5776698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5776772Z layer_outputs = layer_module( 2025-08-14T21:52:19.5777033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5777119Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5777387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5777488Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5777753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5777832Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5777835Z 2025-08-14T21:52:19.5777931Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5778123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5778185Z return mod(**inputs) 2025-08-14T21:52:19.5778463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5778527Z outputs = self.mobilebert( 2025-08-14T21:52:19.5778778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5778851Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5779104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5779171Z layer_outputs = layer_module( 2025-08-14T21:52:19.5779433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5779518Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5779781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5779881Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5780135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5780245Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5780249Z 2025-08-14T21:52:19.5780342Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5780527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5780607Z return mod(**inputs) 2025-08-14T21:52:19.5780867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5780941Z outputs = self.mobilebert( 2025-08-14T21:52:19.5781196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5781287Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5781541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5781608Z layer_outputs = layer_module( 2025-08-14T21:52:19.5781867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5781952Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5782218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5782355Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5782614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.5782701Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5782704Z 2025-08-14T21:52:19.5782799Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5782982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5783050Z return mod(**inputs) 2025-08-14T21:52:19.5783311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5783383Z outputs = self.mobilebert( 2025-08-14T21:52:19.5783639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5783708Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5783971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5784038Z layer_outputs = layer_module( 2025-08-14T21:52:19.5784301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5784390Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5784639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5784755Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5785009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.5785122Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5785380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5785462Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5785466Z 2025-08-14T21:52:19.5785566Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5785743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5785803Z return mod(**inputs) 2025-08-14T21:52:19.5786144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5825277Z outputs = self.mobilebert( 2025-08-14T21:52:19.5825806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5825987Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5826284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5826359Z layer_outputs = layer_module( 2025-08-14T21:52:19.5826669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5826773Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5827043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5827152Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5827457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5827542Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5827547Z 2025-08-14T21:52:19.5827686Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5827888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5827957Z return mod(**inputs) 2025-08-14T21:52:19.5828239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5828315Z outputs = self.mobilebert( 2025-08-14T21:52:19.5828584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5828658Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5828927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5829004Z layer_outputs = layer_module( 2025-08-14T21:52:19.5829266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5829357Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5829633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5829743Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5830012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5830119Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5830123Z 2025-08-14T21:52:19.5830224Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5830428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5830493Z return mod(**inputs) 2025-08-14T21:52:19.5830773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5830847Z outputs = self.mobilebert( 2025-08-14T21:52:19.5831113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5831194Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5831460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5831529Z layer_outputs = layer_module( 2025-08-14T21:52:19.5831801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5831891Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5832163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5832306Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5832573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.5832683Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5832687Z 2025-08-14T21:52:19.5832790Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5832991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5833057Z return mod(**inputs) 2025-08-14T21:52:19.5833329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5833410Z outputs = self.mobilebert( 2025-08-14T21:52:19.5833693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5833785Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5834064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5834134Z layer_outputs = layer_module( 2025-08-14T21:52:19.5834422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5834512Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5834777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5834907Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5835168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.5835296Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5835559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5835656Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5835661Z 2025-08-14T21:52:19.5835763Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5835964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5836029Z return mod(**inputs) 2025-08-14T21:52:19.5836303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5836371Z outputs = self.mobilebert( 2025-08-14T21:52:19.5836632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5836710Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5836970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5837038Z layer_outputs = layer_module( 2025-08-14T21:52:19.5837305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.5837420Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.5837929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5838015Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5838019Z 2025-08-14T21:52:19.5838118Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5838319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5838458Z return mod(**inputs) 2025-08-14T21:52:19.5838738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5838805Z outputs = self.mobilebert( 2025-08-14T21:52:19.5839067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5839169Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5839430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5839497Z layer_outputs = layer_module( 2025-08-14T21:52:19.5839765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.5839908Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.5840211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5840320Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5840324Z 2025-08-14T21:52:19.5840421Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5840646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5840710Z return mod(**inputs) 2025-08-14T21:52:19.5840993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5841062Z outputs = self.mobilebert( 2025-08-14T21:52:19.5841335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5841417Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5841690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5841760Z layer_outputs = layer_module( 2025-08-14T21:52:19.5842041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5842202Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5842480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.5842572Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.5842578Z 2025-08-14T21:52:19.5842676Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5842879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5842947Z return mod(**inputs) 2025-08-14T21:52:19.5843229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5843299Z outputs = self.mobilebert( 2025-08-14T21:52:19.5843569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5843649Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5843919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5843989Z layer_outputs = layer_module( 2025-08-14T21:52:19.5844269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5844424Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5844703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.5844855Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.5845130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5845247Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5845253Z 2025-08-14T21:52:19.5845355Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5845621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5845691Z return mod(**inputs) 2025-08-14T21:52:19.5845960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5846070Z outputs = self.mobilebert( 2025-08-14T21:52:19.5846391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5846469Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5846781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5846857Z layer_outputs = layer_module( 2025-08-14T21:52:19.5847156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5847327Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5847604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.5847736Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.5848016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.5848111Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5848115Z 2025-08-14T21:52:19.5848214Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5848406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5848482Z return mod(**inputs) 2025-08-14T21:52:19.5848754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5848840Z outputs = self.mobilebert( 2025-08-14T21:52:19.5849111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5849185Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5849469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5849542Z layer_outputs = layer_module( 2025-08-14T21:52:19.5849811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5849972Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5850241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.5850367Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.5850635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.5850753Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5851033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5851178Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5851182Z 2025-08-14T21:52:19.5851292Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5851484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5851567Z return mod(**inputs) 2025-08-14T21:52:19.5851848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5851917Z outputs = self.mobilebert( 2025-08-14T21:52:19.5852188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5852270Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5852557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5852641Z layer_outputs = layer_module( 2025-08-14T21:52:19.5852932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.5853098Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.5853384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.5853495Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.5853779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.5853862Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.5853866Z 2025-08-14T21:52:19.5853969Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5854183Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5854248Z return mod(**inputs) 2025-08-14T21:52:19.5854529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5854597Z outputs = self.mobilebert( 2025-08-14T21:52:19.5854868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5854949Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5855215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5855285Z layer_outputs = layer_module( 2025-08-14T21:52:19.5855563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.5855719Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.5855995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.5856101Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.5856370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.5856463Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.5856728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5856828Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5856832Z 2025-08-14T21:52:19.5856932Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5857127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5857220Z return mod(**inputs) 2025-08-14T21:52:19.5857496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5857564Z outputs = self.mobilebert( 2025-08-14T21:52:19.5857841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5857930Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5858210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5858283Z layer_outputs = layer_module( 2025-08-14T21:52:19.5858554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5858664Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5858958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.5859038Z self_outputs = self.self( 2025-08-14T21:52:19.5859301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.5859371Z self.query(query_tensor) 2025-08-14T21:52:19.5859374Z 2025-08-14T21:52:19.5859483Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5859667Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5859731Z return mod(**inputs) 2025-08-14T21:52:19.5860001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5860067Z outputs = self.mobilebert( 2025-08-14T21:52:19.5860338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5860411Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5860673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5860754Z layer_outputs = layer_module( 2025-08-14T21:52:19.5861017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5861111Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5861369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.5861437Z self_outputs = self.self( 2025-08-14T21:52:19.5861707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.5861772Z self.key(key_tensor) 2025-08-14T21:52:19.5861775Z 2025-08-14T21:52:19.5861874Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5862067Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5862131Z return mod(**inputs) 2025-08-14T21:52:19.5862406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5862474Z outputs = self.mobilebert( 2025-08-14T21:52:19.5862743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5862822Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5863090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5863162Z layer_outputs = layer_module( 2025-08-14T21:52:19.5863465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5863551Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5863833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.5863921Z self_outputs = self.self( 2025-08-14T21:52:19.5864196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.5864281Z self.value(value_tensor) 2025-08-14T21:52:19.5864284Z 2025-08-14T21:52:19.5864370Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.5864459Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.5864562Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5864777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5864856Z return mod(**inputs) 2025-08-14T21:52:19.5865161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5865231Z outputs = self.mobilebert( 2025-08-14T21:52:19.5865508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5865579Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5865854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5865922Z layer_outputs = layer_module( 2025-08-14T21:52:19.5866190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5866282Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5866554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.5866682Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.5866947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.5867031Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5867034Z 2025-08-14T21:52:19.5867141Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5867329Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5867393Z return mod(**inputs) 2025-08-14T21:52:19.5867670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5867740Z outputs = self.mobilebert( 2025-08-14T21:52:19.5868018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5868087Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5868355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5868436Z layer_outputs = layer_module( 2025-08-14T21:52:19.5868702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.5868862Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.5869131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.5869236Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.5869519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.5869597Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.5869601Z 2025-08-14T21:52:19.5869697Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5869889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5870765Z return mod(**inputs) 2025-08-14T21:52:19.5871047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5871114Z outputs = self.mobilebert( 2025-08-14T21:52:19.5871377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5871456Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5871737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5871838Z layer_outputs = layer_module( 2025-08-14T21:52:19.5872099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5872181Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5872456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.5872575Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.5872845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.5872983Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5873256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5873359Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5873363Z 2025-08-14T21:52:19.5873465Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5873662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5873741Z return mod(**inputs) 2025-08-14T21:52:19.5874026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5874104Z outputs = self.mobilebert( 2025-08-14T21:52:19.5874369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5874439Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5874716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5874786Z layer_outputs = layer_module( 2025-08-14T21:52:19.5875053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5875155Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5875430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5875551Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5875823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5875917Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5875920Z 2025-08-14T21:52:19.5876026Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5876214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5876303Z return mod(**inputs) 2025-08-14T21:52:19.5876577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5876646Z outputs = self.mobilebert( 2025-08-14T21:52:19.5876921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5877010Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5877286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5877363Z layer_outputs = layer_module( 2025-08-14T21:52:19.5877638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5877755Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5878042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5878149Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5878427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5878536Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5878539Z 2025-08-14T21:52:19.5878643Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5878832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5878893Z return mod(**inputs) 2025-08-14T21:52:19.5879174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5879243Z outputs = self.mobilebert( 2025-08-14T21:52:19.5879513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5879590Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5879862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5879937Z layer_outputs = layer_module( 2025-08-14T21:52:19.5880207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5880296Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5880571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5880693Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5880969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.5881054Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5881058Z 2025-08-14T21:52:19.5881155Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5881351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5881414Z return mod(**inputs) 2025-08-14T21:52:19.5881687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5881755Z outputs = self.mobilebert( 2025-08-14T21:52:19.5882021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5882097Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5882364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5882453Z layer_outputs = layer_module( 2025-08-14T21:52:19.5882726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5882816Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5883109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5883229Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5883494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.5883617Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5883895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5884008Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5884013Z 2025-08-14T21:52:19.5884111Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5884301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5884373Z return mod(**inputs) 2025-08-14T21:52:19.5884639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5884706Z outputs = self.mobilebert( 2025-08-14T21:52:19.5884981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5885048Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5885320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5885450Z layer_outputs = layer_module( 2025-08-14T21:52:19.5885737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5885836Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5886101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5886216Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5886498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5886585Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5886590Z 2025-08-14T21:52:19.5886705Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5886911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5886982Z return mod(**inputs) 2025-08-14T21:52:19.5887284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5887358Z outputs = self.mobilebert( 2025-08-14T21:52:19.5887653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5887740Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5888005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5888082Z layer_outputs = layer_module( 2025-08-14T21:52:19.5888354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5888452Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5888741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5888847Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5889119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5889247Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5889251Z 2025-08-14T21:52:19.5889356Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5889548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5889611Z return mod(**inputs) 2025-08-14T21:52:19.5889890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5889980Z outputs = self.mobilebert( 2025-08-14T21:52:19.5890280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5890351Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5890621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5890691Z layer_outputs = layer_module( 2025-08-14T21:52:19.5890958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5891054Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5891319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5891438Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5891722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.5891804Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5891808Z 2025-08-14T21:52:19.5891910Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5892091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5892154Z return mod(**inputs) 2025-08-14T21:52:19.5892424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5892489Z outputs = self.mobilebert( 2025-08-14T21:52:19.5892750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5892818Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5893077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5893153Z layer_outputs = layer_module( 2025-08-14T21:52:19.5893416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5893502Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5893771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5893887Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5894157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.5894268Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5894530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5894641Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5894646Z 2025-08-14T21:52:19.5894742Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5894934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5895010Z return mod(**inputs) 2025-08-14T21:52:19.5895277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5895348Z outputs = self.mobilebert( 2025-08-14T21:52:19.5895607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5895682Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5895960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5896030Z layer_outputs = layer_module( 2025-08-14T21:52:19.5896309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5896396Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5896654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5896764Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5897023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5897107Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5897110Z 2025-08-14T21:52:19.5897202Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5897387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5897455Z return mod(**inputs) 2025-08-14T21:52:19.5897715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5897787Z outputs = self.mobilebert( 2025-08-14T21:52:19.5898043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5898111Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5898375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5898441Z layer_outputs = layer_module( 2025-08-14T21:52:19.5898699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5898792Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5899051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5899158Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5899414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5899519Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5899523Z 2025-08-14T21:52:19.5899624Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5899806Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5899876Z return mod(**inputs) 2025-08-14T21:52:19.5900138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5900205Z outputs = self.mobilebert( 2025-08-14T21:52:19.5900491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5900559Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5900817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5900909Z layer_outputs = layer_module( 2025-08-14T21:52:19.5901166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5901260Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5901520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5901635Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5901915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.5902011Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5902015Z 2025-08-14T21:52:19.5902123Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5902310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5902375Z return mod(**inputs) 2025-08-14T21:52:19.5902647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5902713Z outputs = self.mobilebert( 2025-08-14T21:52:19.5902973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5903049Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5903313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5903390Z layer_outputs = layer_module( 2025-08-14T21:52:19.5903654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5903738Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5904007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5904122Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5904389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.5904499Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5904759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5904853Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5904858Z 2025-08-14T21:52:19.5904953Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5905145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5905208Z return mod(**inputs) 2025-08-14T21:52:19.5905470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5905544Z outputs = self.mobilebert( 2025-08-14T21:52:19.5905801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5905868Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5906136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5906221Z layer_outputs = layer_module( 2025-08-14T21:52:19.5906491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.5906603Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.5906871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5906971Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5906974Z 2025-08-14T21:52:19.5907067Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5907257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5907318Z return mod(**inputs) 2025-08-14T21:52:19.5907596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5907673Z outputs = self.mobilebert( 2025-08-14T21:52:19.5907945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5908015Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5908281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5908350Z layer_outputs = layer_module( 2025-08-14T21:52:19.5908618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.5908728Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.5908986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5909098Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5909103Z 2025-08-14T21:52:19.5909207Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5909392Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5909453Z return mod(**inputs) 2025-08-14T21:52:19.5909709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5909783Z outputs = self.mobilebert( 2025-08-14T21:52:19.5910036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5910101Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5910360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5910425Z layer_outputs = layer_module( 2025-08-14T21:52:19.5910682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5910830Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5911082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.5911176Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.5911180Z 2025-08-14T21:52:19.5911272Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5911459Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5911518Z return mod(**inputs) 2025-08-14T21:52:19.5911792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5911865Z outputs = self.mobilebert( 2025-08-14T21:52:19.5912141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5912209Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5912480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5912575Z layer_outputs = layer_module( 2025-08-14T21:52:19.5912856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5913009Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5913282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.5913409Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.5913708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5913815Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5913819Z 2025-08-14T21:52:19.5913917Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5914103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5914174Z return mod(**inputs) 2025-08-14T21:52:19.5914440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5914505Z outputs = self.mobilebert( 2025-08-14T21:52:19.5914779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5914845Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5915104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5915172Z layer_outputs = layer_module( 2025-08-14T21:52:19.5915425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5915578Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5915831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.5915948Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.5916198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.5916275Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5916279Z 2025-08-14T21:52:19.5916381Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5916561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5916628Z return mod(**inputs) 2025-08-14T21:52:19.5916883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5916950Z outputs = self.mobilebert( 2025-08-14T21:52:19.5917211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5917276Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5917529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5917602Z layer_outputs = layer_module( 2025-08-14T21:52:19.5917854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5918024Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5918283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.5918394Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.5918671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.5918780Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5919042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5919126Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5919129Z 2025-08-14T21:52:19.5919238Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5919428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5919504Z return mod(**inputs) 2025-08-14T21:52:19.5919764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5919839Z outputs = self.mobilebert( 2025-08-14T21:52:19.5920094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5920166Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5920420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5920484Z layer_outputs = layer_module( 2025-08-14T21:52:19.5920748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.5920901Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.5921172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.5921273Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.5921535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.5921620Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.5921623Z 2025-08-14T21:52:19.5921718Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5921908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5921969Z return mod(**inputs) 2025-08-14T21:52:19.5922234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5922309Z outputs = self.mobilebert( 2025-08-14T21:52:19.5922571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5922638Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5922914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5922983Z layer_outputs = layer_module( 2025-08-14T21:52:19.5923259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.5923414Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.5923692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.5923827Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.5924103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.5924195Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.5924466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5924577Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5924581Z 2025-08-14T21:52:19.5924689Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5924884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5924951Z return mod(**inputs) 2025-08-14T21:52:19.5925256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5925328Z outputs = self.mobilebert( 2025-08-14T21:52:19.5925735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5925813Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5926094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5926174Z layer_outputs = layer_module( 2025-08-14T21:52:19.5926461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5926558Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5926860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.5926935Z self_outputs = self.self( 2025-08-14T21:52:19.5927224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.5927294Z self.query(query_tensor) 2025-08-14T21:52:19.5927297Z 2025-08-14T21:52:19.5927394Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5927587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5927652Z return mod(**inputs) 2025-08-14T21:52:19.5927927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5927995Z outputs = self.mobilebert( 2025-08-14T21:52:19.5928259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5928335Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5928602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5928671Z layer_outputs = layer_module( 2025-08-14T21:52:19.5928943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5929021Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5929290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.5929356Z self_outputs = self.self( 2025-08-14T21:52:19.5929621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.5929690Z self.key(key_tensor) 2025-08-14T21:52:19.5929693Z 2025-08-14T21:52:19.5929787Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5929977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5930061Z return mod(**inputs) 2025-08-14T21:52:19.5930330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5930403Z outputs = self.mobilebert( 2025-08-14T21:52:19.5930663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5930752Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5931014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5931079Z layer_outputs = layer_module( 2025-08-14T21:52:19.5931340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5931436Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5931722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.5931798Z self_outputs = self.self( 2025-08-14T21:52:19.5932059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.5932132Z self.value(value_tensor) 2025-08-14T21:52:19.5932135Z 2025-08-14T21:52:19.5932212Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.5932289Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.5932392Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5932580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5932641Z return mod(**inputs) 2025-08-14T21:52:19.5932918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5932985Z outputs = self.mobilebert( 2025-08-14T21:52:19.5933253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5933320Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5933579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5933656Z layer_outputs = layer_module( 2025-08-14T21:52:19.5933917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5933995Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5934263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.5934379Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.5934649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.5934727Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5934731Z 2025-08-14T21:52:19.5934826Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5935017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5935079Z return mod(**inputs) 2025-08-14T21:52:19.5935350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5935416Z outputs = self.mobilebert( 2025-08-14T21:52:19.5935676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5935752Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5936013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5936110Z layer_outputs = layer_module( 2025-08-14T21:52:19.5936378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.5936526Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.5936811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.5936911Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.5937167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.5937250Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.5937253Z 2025-08-14T21:52:19.5937364Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5937574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5937837Z return mod(**inputs) 2025-08-14T21:52:19.5938113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5938193Z outputs = self.mobilebert( 2025-08-14T21:52:19.5938454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5938531Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5938793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5938860Z layer_outputs = layer_module( 2025-08-14T21:52:19.5939126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5939207Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5939468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.5939591Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.5939850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.5939975Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5940236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5940322Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5940326Z 2025-08-14T21:52:19.5940430Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5940619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5940690Z return mod(**inputs) 2025-08-14T21:52:19.5940955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5941022Z outputs = self.mobilebert( 2025-08-14T21:52:19.5941289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5941358Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5941618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5941691Z layer_outputs = layer_module( 2025-08-14T21:52:19.5942010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5942152Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5942407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5942509Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5942778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5942885Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5942888Z 2025-08-14T21:52:19.5942996Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5943185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5943249Z return mod(**inputs) 2025-08-14T21:52:19.5943546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5943614Z outputs = self.mobilebert( 2025-08-14T21:52:19.5943926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5944002Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5944270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5944346Z layer_outputs = layer_module( 2025-08-14T21:52:19.5944598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5944686Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5944946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5945048Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5945310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5945415Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5945419Z 2025-08-14T21:52:19.5945514Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5945701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5945766Z return mod(**inputs) 2025-08-14T21:52:19.5946021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5946097Z outputs = self.mobilebert( 2025-08-14T21:52:19.5946351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5946430Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5946687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5946756Z layer_outputs = layer_module( 2025-08-14T21:52:19.5947019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5947107Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5947368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5947487Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5947739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.5947825Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5947828Z 2025-08-14T21:52:19.5947928Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5948175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5948237Z return mod(**inputs) 2025-08-14T21:52:19.5948502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5948592Z outputs = self.mobilebert( 2025-08-14T21:52:19.5948858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5948926Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5949198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5949267Z layer_outputs = layer_module( 2025-08-14T21:52:19.5949552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5949641Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5949924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5950049Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5950309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.5950429Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5950685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5950769Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5950772Z 2025-08-14T21:52:19.5950875Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5951057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5951119Z return mod(**inputs) 2025-08-14T21:52:19.5951382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5951447Z outputs = self.mobilebert( 2025-08-14T21:52:19.5951715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5951781Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5952037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5952108Z layer_outputs = layer_module( 2025-08-14T21:52:19.5952375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5952472Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5952740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5952848Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5953132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5953215Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5953219Z 2025-08-14T21:52:19.5953318Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5953522Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5953586Z return mod(**inputs) 2025-08-14T21:52:19.5953874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5953963Z outputs = self.mobilebert( 2025-08-14T21:52:19.5954245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5954320Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5954580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5954672Z layer_outputs = layer_module( 2025-08-14T21:52:19.5954940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5955025Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5955282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5955397Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5955682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5955788Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5955792Z 2025-08-14T21:52:19.5955886Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5956075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5956135Z return mod(**inputs) 2025-08-14T21:52:19.5956393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5956466Z outputs = self.mobilebert( 2025-08-14T21:52:19.5956717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5956791Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5957041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5957109Z layer_outputs = layer_module( 2025-08-14T21:52:19.5957371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5957457Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5957706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5957827Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5958074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.5958157Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5958160Z 2025-08-14T21:52:19.5958254Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5958433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5958499Z return mod(**inputs) 2025-08-14T21:52:19.5958751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5958823Z outputs = self.mobilebert( 2025-08-14T21:52:19.5959074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5959140Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5959395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5959459Z layer_outputs = layer_module( 2025-08-14T21:52:19.5959710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5959822Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5960077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5960198Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5960475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.5960598Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5960858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5960940Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5960944Z 2025-08-14T21:52:19.5961057Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5961244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5961323Z return mod(**inputs) 2025-08-14T21:52:19.5961598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5961664Z outputs = self.mobilebert( 2025-08-14T21:52:19.5961934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5961999Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5962261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5962334Z layer_outputs = layer_module( 2025-08-14T21:52:19.5962597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5962684Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5962956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5963063Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5963338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5963419Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5963423Z 2025-08-14T21:52:19.5963519Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5963716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5963779Z return mod(**inputs) 2025-08-14T21:52:19.5964058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5964127Z outputs = self.mobilebert( 2025-08-14T21:52:19.5964397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5964474Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5964746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5964826Z layer_outputs = layer_module( 2025-08-14T21:52:19.5965093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5965179Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5965506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.5965619Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.5965912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5966035Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5966039Z 2025-08-14T21:52:19.5966145Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5966361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5966449Z return mod(**inputs) 2025-08-14T21:52:19.5966746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5966825Z outputs = self.mobilebert( 2025-08-14T21:52:19.5967101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5967173Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5967476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5967566Z layer_outputs = layer_module( 2025-08-14T21:52:19.5967846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5967940Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5968216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5968348Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5968632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.5968719Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5968722Z 2025-08-14T21:52:19.5968822Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5969014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5969085Z return mod(**inputs) 2025-08-14T21:52:19.5969351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5969421Z outputs = self.mobilebert( 2025-08-14T21:52:19.5969689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5969759Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5970027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5970095Z layer_outputs = layer_module( 2025-08-14T21:52:19.5970356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.5970454Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.5970718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.5970843Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.5971102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.5971215Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5971484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5971570Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5971573Z 2025-08-14T21:52:19.5971680Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5971870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5971949Z return mod(**inputs) 2025-08-14T21:52:19.5972224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5972289Z outputs = self.mobilebert( 2025-08-14T21:52:19.5972569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5972642Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5972900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5972971Z layer_outputs = layer_module( 2025-08-14T21:52:19.5973245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.5973360Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.5973663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.5973745Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.5973748Z 2025-08-14T21:52:19.5973854Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5974043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5974105Z return mod(**inputs) 2025-08-14T21:52:19.5974381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5974446Z outputs = self.mobilebert( 2025-08-14T21:52:19.5974719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5974795Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5975056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5975129Z layer_outputs = layer_module( 2025-08-14T21:52:19.5975388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.5975497Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.5975764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.5975866Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.5975870Z 2025-08-14T21:52:19.5975970Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5976156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5976218Z return mod(**inputs) 2025-08-14T21:52:19.5976486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5976551Z outputs = self.mobilebert( 2025-08-14T21:52:19.5976812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5976889Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5977149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5977221Z layer_outputs = layer_module( 2025-08-14T21:52:19.5977480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5977628Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5977914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.5978001Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.5978005Z 2025-08-14T21:52:19.5978105Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5978291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5978369Z return mod(**inputs) 2025-08-14T21:52:19.5978636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5978700Z outputs = self.mobilebert( 2025-08-14T21:52:19.5978958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5979034Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5979317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5979411Z layer_outputs = layer_module( 2025-08-14T21:52:19.5979678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5979825Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5980095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.5980210Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.5980482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5980566Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5980569Z 2025-08-14T21:52:19.5980668Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5980861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5980922Z return mod(**inputs) 2025-08-14T21:52:19.5981187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5981259Z outputs = self.mobilebert( 2025-08-14T21:52:19.5981519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5981595Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5981857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5981923Z layer_outputs = layer_module( 2025-08-14T21:52:19.5982194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5982342Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5982624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.5982745Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.5983032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.5983122Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.5983125Z 2025-08-14T21:52:19.5983222Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5983417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5983480Z return mod(**inputs) 2025-08-14T21:52:19.5983751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5983847Z outputs = self.mobilebert( 2025-08-14T21:52:19.5984125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5984203Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5984495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5984562Z layer_outputs = layer_module( 2025-08-14T21:52:19.5984829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.5984973Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.5985251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.5985399Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.5985666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.5985789Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.5986058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5986145Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5986149Z 2025-08-14T21:52:19.5986254Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5986442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5986505Z return mod(**inputs) 2025-08-14T21:52:19.5986783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5986854Z outputs = self.mobilebert( 2025-08-14T21:52:19.5987128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5987196Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5987463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5987538Z layer_outputs = layer_module( 2025-08-14T21:52:19.5987802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.5987964Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.5988232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.5988339Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.5988612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.5988691Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.5988694Z 2025-08-14T21:52:19.5988802Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5988991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5989053Z return mod(**inputs) 2025-08-14T21:52:19.5989330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5989399Z outputs = self.mobilebert( 2025-08-14T21:52:19.5989664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5989760Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5990037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5990111Z layer_outputs = layer_module( 2025-08-14T21:52:19.5990388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.5990556Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.5990828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.5990931Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.5991220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.5991305Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.5991589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.5991687Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.5991691Z 2025-08-14T21:52:19.5991789Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5991980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5992050Z return mod(**inputs) 2025-08-14T21:52:19.5992318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5992393Z outputs = self.mobilebert( 2025-08-14T21:52:19.5992668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5992742Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5993025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5993095Z layer_outputs = layer_module( 2025-08-14T21:52:19.5993377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5993462Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5993737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.5993813Z self_outputs = self.self( 2025-08-14T21:52:19.5994088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.5994158Z self.query(query_tensor) 2025-08-14T21:52:19.5994161Z 2025-08-14T21:52:19.5994272Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5994471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5994543Z return mod(**inputs) 2025-08-14T21:52:19.5994821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5994901Z outputs = self.mobilebert( 2025-08-14T21:52:19.5995177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5995247Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5995513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5995589Z layer_outputs = layer_module( 2025-08-14T21:52:19.5995857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5995985Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5996255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.5996323Z self_outputs = self.self( 2025-08-14T21:52:19.5996593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.5996681Z self.key(key_tensor) 2025-08-14T21:52:19.5996685Z 2025-08-14T21:52:19.5996791Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5996982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5997044Z return mod(**inputs) 2025-08-14T21:52:19.5997339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.5997409Z outputs = self.mobilebert( 2025-08-14T21:52:19.5997693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.5997772Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.5998040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.5998117Z layer_outputs = layer_module( 2025-08-14T21:52:19.5998381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.5998460Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.5998738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.5998807Z self_outputs = self.self( 2025-08-14T21:52:19.5999087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.5999168Z self.value(value_tensor) 2025-08-14T21:52:19.5999171Z 2025-08-14T21:52:19.5999249Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.5999331Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.5999426Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.5999611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.5999679Z return mod(**inputs) 2025-08-14T21:52:19.5999940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6000012Z outputs = self.mobilebert( 2025-08-14T21:52:19.6000271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6000340Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6000610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6000675Z layer_outputs = layer_module( 2025-08-14T21:52:19.6000937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6001023Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6001284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6001405Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6001664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.6001742Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6001747Z 2025-08-14T21:52:19.6001867Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6002053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6002121Z return mod(**inputs) 2025-08-14T21:52:19.6002382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6002474Z outputs = self.mobilebert( 2025-08-14T21:52:19.6002739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6002807Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6003063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6003137Z layer_outputs = layer_module( 2025-08-14T21:52:19.6003409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6003585Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6003846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.6003951Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.6004225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6004304Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6004307Z 2025-08-14T21:52:19.6004411Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6004600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6004663Z return mod(**inputs) 2025-08-14T21:52:19.6004937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6005008Z outputs = self.mobilebert( 2025-08-14T21:52:19.6005280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6005349Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6005698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6005778Z layer_outputs = layer_module( 2025-08-14T21:52:19.6006045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6006126Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6006413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6006528Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6006798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.6006917Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6007184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6007279Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6007283Z 2025-08-14T21:52:19.6007380Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6007576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6007641Z return mod(**inputs) 2025-08-14T21:52:19.6007910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6008011Z outputs = self.mobilebert( 2025-08-14T21:52:19.6008279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6008359Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6008627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6008715Z layer_outputs = layer_module( 2025-08-14T21:52:19.6008984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6009072Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6009331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6009459Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6009736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6009821Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6009824Z 2025-08-14T21:52:19.6009919Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6010102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6010171Z return mod(**inputs) 2025-08-14T21:52:19.6010435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6010501Z outputs = self.mobilebert( 2025-08-14T21:52:19.6010767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6010837Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6011105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6011172Z layer_outputs = layer_module( 2025-08-14T21:52:19.6011430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6011526Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6011784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6011892Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6012151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6012255Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6012258Z 2025-08-14T21:52:19.6012360Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6012545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6012608Z return mod(**inputs) 2025-08-14T21:52:19.6012884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6012953Z outputs = self.mobilebert( 2025-08-14T21:52:19.6013223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6013294Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6013572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6013647Z layer_outputs = layer_module( 2025-08-14T21:52:19.6013911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6014024Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6014288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6014409Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6014698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6014778Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6014782Z 2025-08-14T21:52:19.6014885Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6015070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6015134Z return mod(**inputs) 2025-08-14T21:52:19.6015428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6015512Z outputs = self.mobilebert( 2025-08-14T21:52:19.6015776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6015850Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6016116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6016187Z layer_outputs = layer_module( 2025-08-14T21:52:19.6016455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6016543Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6016823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6016944Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6017221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6017347Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6017623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6017718Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6017721Z 2025-08-14T21:52:19.6017819Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6018010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6018083Z return mod(**inputs) 2025-08-14T21:52:19.6018359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6018433Z outputs = self.mobilebert( 2025-08-14T21:52:19.6018706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6018775Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6019053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6019123Z layer_outputs = layer_module( 2025-08-14T21:52:19.6019401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6019489Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6019761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6019875Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6020167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6020245Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6020254Z 2025-08-14T21:52:19.6020349Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6020561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6020628Z return mod(**inputs) 2025-08-14T21:52:19.6020899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6020964Z outputs = self.mobilebert( 2025-08-14T21:52:19.6021240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6021333Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6021627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6021698Z layer_outputs = layer_module( 2025-08-14T21:52:19.6021965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6022062Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6022329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6022433Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6022705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6022811Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6022816Z 2025-08-14T21:52:19.6022924Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6023117Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6023180Z return mod(**inputs) 2025-08-14T21:52:19.6023459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6023529Z outputs = self.mobilebert( 2025-08-14T21:52:19.6023799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6023868Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6024133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6024210Z layer_outputs = layer_module( 2025-08-14T21:52:19.6024477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6024568Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6024843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6024963Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6025242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6025323Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6025326Z 2025-08-14T21:52:19.6025423Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6025621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6025684Z return mod(**inputs) 2025-08-14T21:52:19.6025964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6026058Z outputs = self.mobilebert( 2025-08-14T21:52:19.6026332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6026410Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6026695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6026763Z layer_outputs = layer_module( 2025-08-14T21:52:19.6027042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6027129Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6027407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6027526Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6027807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6027931Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6028200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6028291Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6028294Z 2025-08-14T21:52:19.6028389Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6028575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6028643Z return mod(**inputs) 2025-08-14T21:52:19.6028913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6028982Z outputs = self.mobilebert( 2025-08-14T21:52:19.6029264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6029330Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6029596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6029664Z layer_outputs = layer_module( 2025-08-14T21:52:19.6029920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6030015Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6030273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6030383Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6030642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6030719Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6030722Z 2025-08-14T21:52:19.6030826Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6031012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6031088Z return mod(**inputs) 2025-08-14T21:52:19.6031353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6031418Z outputs = self.mobilebert( 2025-08-14T21:52:19.6031683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6031754Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6032038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6032115Z layer_outputs = layer_module( 2025-08-14T21:52:19.6032376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6032487Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6032747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6032851Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6033123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6033226Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6033248Z 2025-08-14T21:52:19.6033364Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6033563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6033626Z return mod(**inputs) 2025-08-14T21:52:19.6033899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6033966Z outputs = self.mobilebert( 2025-08-14T21:52:19.6034225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6034301Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6034562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6034636Z layer_outputs = layer_module( 2025-08-14T21:52:19.6034896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6034983Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6035248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6035363Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6035631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6035710Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6035713Z 2025-08-14T21:52:19.6035809Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6035998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6036058Z return mod(**inputs) 2025-08-14T21:52:19.6036320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6036394Z outputs = self.mobilebert( 2025-08-14T21:52:19.6036654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6036728Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6036985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6037050Z layer_outputs = layer_module( 2025-08-14T21:52:19.6037314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6037397Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6037791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6037958Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6038216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6038335Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6038615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6038700Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6038711Z 2025-08-14T21:52:19.6038808Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6038990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6039059Z return mod(**inputs) 2025-08-14T21:52:19.6039387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6039457Z outputs = self.mobilebert( 2025-08-14T21:52:19.6039750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6039819Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6040087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6040153Z layer_outputs = layer_module( 2025-08-14T21:52:19.6040411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6040530Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6040789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6040868Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6040880Z 2025-08-14T21:52:19.6040976Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6041161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6041230Z return mod(**inputs) 2025-08-14T21:52:19.6041491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6041557Z outputs = self.mobilebert( 2025-08-14T21:52:19.6041824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6041892Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6042160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6042229Z layer_outputs = layer_module( 2025-08-14T21:52:19.6042495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6042617Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6042883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6042990Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6043002Z 2025-08-14T21:52:19.6043097Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6043287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6043359Z return mod(**inputs) 2025-08-14T21:52:19.6043627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6043695Z outputs = self.mobilebert( 2025-08-14T21:52:19.6043996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6044064Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6044332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6044414Z layer_outputs = layer_module( 2025-08-14T21:52:19.6044672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6044827Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6045084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.6045177Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.6045197Z 2025-08-14T21:52:19.6045295Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6045564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6045642Z return mod(**inputs) 2025-08-14T21:52:19.6045919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6045989Z outputs = self.mobilebert( 2025-08-14T21:52:19.6046284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6046355Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6046624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6046693Z layer_outputs = layer_module( 2025-08-14T21:52:19.6046966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6047126Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6047388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.6047512Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.6047776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6047863Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6047867Z 2025-08-14T21:52:19.6047969Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6048151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6048214Z return mod(**inputs) 2025-08-14T21:52:19.6048487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6048554Z outputs = self.mobilebert( 2025-08-14T21:52:19.6048823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6048893Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6049160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6049236Z layer_outputs = layer_module( 2025-08-14T21:52:19.6049502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6049658Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6049922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6050066Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6050343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.6050426Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6050444Z 2025-08-14T21:52:19.6050549Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6050743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6050806Z return mod(**inputs) 2025-08-14T21:52:19.6051080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6051148Z outputs = self.mobilebert( 2025-08-14T21:52:19.6051429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6051509Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6051795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6051871Z layer_outputs = layer_module( 2025-08-14T21:52:19.6052133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6052282Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6052550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6052663Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6052934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.6053050Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6053315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6053408Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6053413Z 2025-08-14T21:52:19.6053510Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6053698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6053766Z return mod(**inputs) 2025-08-14T21:52:19.6054033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6054106Z outputs = self.mobilebert( 2025-08-14T21:52:19.6054369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6054440Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6054713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6054781Z layer_outputs = layer_module( 2025-08-14T21:52:19.6055052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6055207Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6055471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6055582Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6055846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6055945Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6055956Z 2025-08-14T21:52:19.6056054Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6056243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6056313Z return mod(**inputs) 2025-08-14T21:52:19.6056600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6056666Z outputs = self.mobilebert( 2025-08-14T21:52:19.6056941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6057010Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6057284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6057368Z layer_outputs = layer_module( 2025-08-14T21:52:19.6057655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6057816Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6058087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6058194Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6058483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.6058560Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.6058826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6058910Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6058914Z 2025-08-14T21:52:19.6059010Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6059199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6059259Z return mod(**inputs) 2025-08-14T21:52:19.6059528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6059593Z outputs = self.mobilebert( 2025-08-14T21:52:19.6059849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6059923Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6060179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6060246Z layer_outputs = layer_module( 2025-08-14T21:52:19.6060508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6060585Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6060851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6060919Z self_outputs = self.self( 2025-08-14T21:52:19.6061175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.6061248Z self.query(query_tensor) 2025-08-14T21:52:19.6061251Z 2025-08-14T21:52:19.6061343Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6061528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6061587Z return mod(**inputs) 2025-08-14T21:52:19.6061848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6061943Z outputs = self.mobilebert( 2025-08-14T21:52:19.6062194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6062260Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6062540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6062608Z layer_outputs = layer_module( 2025-08-14T21:52:19.6062874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6062953Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6063230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6063306Z self_outputs = self.self( 2025-08-14T21:52:19.6063578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.6063651Z self.key(key_tensor) 2025-08-14T21:52:19.6063654Z 2025-08-14T21:52:19.6063751Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6063936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6064006Z return mod(**inputs) 2025-08-14T21:52:19.6064273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6064338Z outputs = self.mobilebert( 2025-08-14T21:52:19.6064622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6064689Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6064955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6065020Z layer_outputs = layer_module( 2025-08-14T21:52:19.6065275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6065360Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6065614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6065683Z self_outputs = self.self( 2025-08-14T21:52:19.6065935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.6065999Z self.value(value_tensor) 2025-08-14T21:52:19.6066002Z 2025-08-14T21:52:19.6066084Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6066158Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6066251Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6066437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6066497Z return mod(**inputs) 2025-08-14T21:52:19.6066757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6066822Z outputs = self.mobilebert( 2025-08-14T21:52:19.6067074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6067146Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6067398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6067464Z layer_outputs = layer_module( 2025-08-14T21:52:19.6067739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6067814Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6068072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6068201Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6068455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.6068538Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6068541Z 2025-08-14T21:52:19.6068633Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6068817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6068900Z return mod(**inputs) 2025-08-14T21:52:19.6069175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6069247Z outputs = self.mobilebert( 2025-08-14T21:52:19.6069499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6069566Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6069822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6069887Z layer_outputs = layer_module( 2025-08-14T21:52:19.6070141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6070287Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6070539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.6070650Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.6070900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6070993Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6070998Z 2025-08-14T21:52:19.6071090Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6071269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6071336Z return mod(**inputs) 2025-08-14T21:52:19.6071592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6071656Z outputs = self.mobilebert( 2025-08-14T21:52:19.6071915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6071985Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6072245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6072310Z layer_outputs = layer_module( 2025-08-14T21:52:19.6072562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6072645Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6072906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6073025Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6073286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.6073431Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6073695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6073779Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6073782Z 2025-08-14T21:52:19.6073898Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6074076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6074137Z return mod(**inputs) 2025-08-14T21:52:19.6074397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6074462Z outputs = self.mobilebert( 2025-08-14T21:52:19.6074736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6074814Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6075101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6075185Z layer_outputs = layer_module( 2025-08-14T21:52:19.6075435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6075523Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6075781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6075880Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6076138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6076214Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6076218Z 2025-08-14T21:52:19.6076310Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6076499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6076559Z return mod(**inputs) 2025-08-14T21:52:19.6076813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6076885Z outputs = self.mobilebert( 2025-08-14T21:52:19.6077135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6077208Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6077457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6077523Z layer_outputs = layer_module( 2025-08-14T21:52:19.6077782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6077869Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6078125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6078225Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6078475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6078584Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6078587Z 2025-08-14T21:52:19.6078677Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6078855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6078924Z return mod(**inputs) 2025-08-14T21:52:19.6079200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6079270Z outputs = self.mobilebert( 2025-08-14T21:52:19.6079521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6079606Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6079864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6079928Z layer_outputs = layer_module( 2025-08-14T21:52:19.6080185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6080270Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6080538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6080681Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6080934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6081010Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6081021Z 2025-08-14T21:52:19.6081116Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6081297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6081365Z return mod(**inputs) 2025-08-14T21:52:19.6081633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6081697Z outputs = self.mobilebert( 2025-08-14T21:52:19.6081956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6082022Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6082285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6082352Z layer_outputs = layer_module( 2025-08-14T21:52:19.6082609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6082701Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6082959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6083074Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6083342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6083457Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6083722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6083805Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6083809Z 2025-08-14T21:52:19.6083904Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6084095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6084159Z return mod(**inputs) 2025-08-14T21:52:19.6084424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6084488Z outputs = self.mobilebert( 2025-08-14T21:52:19.6084749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6084843Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6085102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6085170Z layer_outputs = layer_module( 2025-08-14T21:52:19.6085497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6085612Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6085882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6085985Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6086245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6086353Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6086358Z 2025-08-14T21:52:19.6086459Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6086673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6086740Z return mod(**inputs) 2025-08-14T21:52:19.6087020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6087096Z outputs = self.mobilebert( 2025-08-14T21:52:19.6087357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6087431Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6087690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6087760Z layer_outputs = layer_module( 2025-08-14T21:52:19.6088026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6088113Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6088376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6088489Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6088750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6088859Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6088863Z 2025-08-14T21:52:19.6088957Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6089142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6089213Z return mod(**inputs) 2025-08-14T21:52:19.6089479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6089549Z outputs = self.mobilebert( 2025-08-14T21:52:19.6089809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6089877Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6090148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6090215Z layer_outputs = layer_module( 2025-08-14T21:52:19.6090476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6090570Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6090829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6090972Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6091233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6091313Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6091341Z 2025-08-14T21:52:19.6091445Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6091631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6091701Z return mod(**inputs) 2025-08-14T21:52:19.6091965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6092032Z outputs = self.mobilebert( 2025-08-14T21:52:19.6092317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6092388Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6092661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6092736Z layer_outputs = layer_module( 2025-08-14T21:52:19.6092995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6093089Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6093349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6093464Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6093733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6093849Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6094117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6094203Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6094207Z 2025-08-14T21:52:19.6094303Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6094497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6094559Z return mod(**inputs) 2025-08-14T21:52:19.6094821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6094896Z outputs = self.mobilebert( 2025-08-14T21:52:19.6095157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6095233Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6095492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6095559Z layer_outputs = layer_module( 2025-08-14T21:52:19.6095829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6095916Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6096185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6096286Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6096546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6096630Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6096652Z 2025-08-14T21:52:19.6096750Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6096943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6097003Z return mod(**inputs) 2025-08-14T21:52:19.6097271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6097363Z outputs = self.mobilebert( 2025-08-14T21:52:19.6097620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6097689Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6097955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6098021Z layer_outputs = layer_module( 2025-08-14T21:52:19.6098305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6098411Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6098674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6098782Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6099043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6099152Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6099155Z 2025-08-14T21:52:19.6099250Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6099435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6099504Z return mod(**inputs) 2025-08-14T21:52:19.6099772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6099838Z outputs = self.mobilebert( 2025-08-14T21:52:19.6100114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6100181Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6100442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6100506Z layer_outputs = layer_module( 2025-08-14T21:52:19.6100760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6100848Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6101105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6101226Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6101481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6101559Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6101563Z 2025-08-14T21:52:19.6101662Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6101844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6101906Z return mod(**inputs) 2025-08-14T21:52:19.6102177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6102243Z outputs = self.mobilebert( 2025-08-14T21:52:19.6102511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6102614Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6102882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6102957Z layer_outputs = layer_module( 2025-08-14T21:52:19.6103216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6103329Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6103589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6103704Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6103989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6104106Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6104384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6104478Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6104482Z 2025-08-14T21:52:19.6104578Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6104771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6104833Z return mod(**inputs) 2025-08-14T21:52:19.6105096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6105167Z outputs = self.mobilebert( 2025-08-14T21:52:19.6105431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6105507Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6105772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6105840Z layer_outputs = layer_module( 2025-08-14T21:52:19.6106105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6106219Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6106479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6106564Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6106567Z 2025-08-14T21:52:19.6106662Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6106855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6106917Z return mod(**inputs) 2025-08-14T21:52:19.6107183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6107258Z outputs = self.mobilebert( 2025-08-14T21:52:19.6107518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6107593Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6107851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6107918Z layer_outputs = layer_module( 2025-08-14T21:52:19.6108185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6108299Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6108561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6108695Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6108699Z 2025-08-14T21:52:19.6108794Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6108984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6109062Z return mod(**inputs) 2025-08-14T21:52:19.6109329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6109403Z outputs = self.mobilebert( 2025-08-14T21:52:19.6109665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6109738Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6110015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6110101Z layer_outputs = layer_module( 2025-08-14T21:52:19.6110371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6110519Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6110777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.6110872Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.6110875Z 2025-08-14T21:52:19.6110970Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6111156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6111216Z return mod(**inputs) 2025-08-14T21:52:19.6111478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6111551Z outputs = self.mobilebert( 2025-08-14T21:52:19.6111809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6111885Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6112142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6112208Z layer_outputs = layer_module( 2025-08-14T21:52:19.6112475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6112624Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6112898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.6113018Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.6113290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6113389Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6113395Z 2025-08-14T21:52:19.6113496Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6113691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6113764Z return mod(**inputs) 2025-08-14T21:52:19.6114039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6114113Z outputs = self.mobilebert( 2025-08-14T21:52:19.6114383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6114495Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6114764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6114829Z layer_outputs = layer_module( 2025-08-14T21:52:19.6115117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6115263Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6115528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6115650Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6115934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.6116015Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6116043Z 2025-08-14T21:52:19.6116141Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6116325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6116396Z return mod(**inputs) 2025-08-14T21:52:19.6116658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6116724Z outputs = self.mobilebert( 2025-08-14T21:52:19.6116998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6117063Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6117324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6117390Z layer_outputs = layer_module( 2025-08-14T21:52:19.6117642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6117793Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6118047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6118160Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6118423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.6118534Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6118798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6118883Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6118886Z 2025-08-14T21:52:19.6118981Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6119168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6119230Z return mod(**inputs) 2025-08-14T21:52:19.6119496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6119559Z outputs = self.mobilebert( 2025-08-14T21:52:19.6119811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6119885Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6120141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6120230Z layer_outputs = layer_module( 2025-08-14T21:52:19.6120484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6120629Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6120889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6121006Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6121259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6121339Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6121343Z 2025-08-14T21:52:19.6121435Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6121636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6121699Z return mod(**inputs) 2025-08-14T21:52:19.6121974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6122047Z outputs = self.mobilebert( 2025-08-14T21:52:19.6122301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6122377Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6122634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6122700Z layer_outputs = layer_module( 2025-08-14T21:52:19.6122969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6123119Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6123383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6123490Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6123747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.6123836Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.6124102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6124188Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6124191Z 2025-08-14T21:52:19.6124296Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6124480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6124551Z return mod(**inputs) 2025-08-14T21:52:19.6124816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6124882Z outputs = self.mobilebert( 2025-08-14T21:52:19.6125151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6125220Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6125560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6125643Z layer_outputs = layer_module( 2025-08-14T21:52:19.6125933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6126032Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6126332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6126431Z self_outputs = self.self( 2025-08-14T21:52:19.6126714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.6126784Z self.query(query_tensor) 2025-08-14T21:52:19.6126806Z 2025-08-14T21:52:19.6126916Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6127109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6127176Z return mod(**inputs) 2025-08-14T21:52:19.6127458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6127529Z outputs = self.mobilebert( 2025-08-14T21:52:19.6127821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6127897Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6128167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6128239Z layer_outputs = layer_module( 2025-08-14T21:52:19.6128492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6128569Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6128825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6128888Z self_outputs = self.self( 2025-08-14T21:52:19.6129142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.6129205Z self.key(key_tensor) 2025-08-14T21:52:19.6129209Z 2025-08-14T21:52:19.6129301Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6129485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6129545Z return mod(**inputs) 2025-08-14T21:52:19.6129798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6129870Z outputs = self.mobilebert( 2025-08-14T21:52:19.6130119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6130191Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6130440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6130506Z layer_outputs = layer_module( 2025-08-14T21:52:19.6130764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6130841Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6131098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6131161Z self_outputs = self.self( 2025-08-14T21:52:19.6131415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.6131485Z self.value(value_tensor) 2025-08-14T21:52:19.6131488Z 2025-08-14T21:52:19.6131561Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6131632Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6131733Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6131913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6131998Z return mod(**inputs) 2025-08-14T21:52:19.6132258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6132324Z outputs = self.mobilebert( 2025-08-14T21:52:19.6132583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6132694Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6132949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6133021Z layer_outputs = layer_module( 2025-08-14T21:52:19.6133272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6133355Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6133622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6133761Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6134021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.6134099Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6134102Z 2025-08-14T21:52:19.6134202Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6134381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6134442Z return mod(**inputs) 2025-08-14T21:52:19.6134703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6134767Z outputs = self.mobilebert( 2025-08-14T21:52:19.6135022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6135097Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6135351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6135422Z layer_outputs = layer_module( 2025-08-14T21:52:19.6135685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6135830Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6136093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.6136191Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.6136452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6136528Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6136531Z 2025-08-14T21:52:19.6136623Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6136813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6136876Z return mod(**inputs) 2025-08-14T21:52:19.6137131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6137200Z outputs = self.mobilebert( 2025-08-14T21:52:19.6137453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6137526Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6137986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6138102Z layer_outputs = layer_module( 2025-08-14T21:52:19.6138367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6138442Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6138704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6138838Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6139090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.6139209Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6139488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6139584Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6139588Z 2025-08-14T21:52:19.6139703Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6139889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6139962Z return mod(**inputs) 2025-08-14T21:52:19.6140228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6140297Z outputs = self.mobilebert( 2025-08-14T21:52:19.6140562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6140632Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6140894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6140963Z layer_outputs = layer_module( 2025-08-14T21:52:19.6141221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6141318Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6141577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6141690Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6141946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6142025Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6142028Z 2025-08-14T21:52:19.6142131Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6142316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6142379Z return mod(**inputs) 2025-08-14T21:52:19.6142647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6142714Z outputs = self.mobilebert( 2025-08-14T21:52:19.6142975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6143043Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6143301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6143374Z layer_outputs = layer_module( 2025-08-14T21:52:19.6143630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6143724Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6143981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6144102Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6144364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6144484Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6144488Z 2025-08-14T21:52:19.6144579Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6144767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6144826Z return mod(**inputs) 2025-08-14T21:52:19.6145088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6145151Z outputs = self.mobilebert( 2025-08-14T21:52:19.6145420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6145510Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6145764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6145835Z layer_outputs = layer_module( 2025-08-14T21:52:19.6146091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6146175Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6146434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6146551Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6146801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6146886Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6146890Z 2025-08-14T21:52:19.6146984Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6147171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6147232Z return mod(**inputs) 2025-08-14T21:52:19.6147486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6147558Z outputs = self.mobilebert( 2025-08-14T21:52:19.6147809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6147882Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6148138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6148203Z layer_outputs = layer_module( 2025-08-14T21:52:19.6148464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6148550Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6148802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6148923Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6149174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6149289Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6149545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6149648Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6149651Z 2025-08-14T21:52:19.6149752Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6149934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6150001Z return mod(**inputs) 2025-08-14T21:52:19.6150276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6150339Z outputs = self.mobilebert( 2025-08-14T21:52:19.6150598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6150664Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6150919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6151004Z layer_outputs = layer_module( 2025-08-14T21:52:19.6151270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6151362Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6151613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6151714Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6151972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6152049Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6152052Z 2025-08-14T21:52:19.6152157Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6152346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6152409Z return mod(**inputs) 2025-08-14T21:52:19.6152685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6152751Z outputs = self.mobilebert( 2025-08-14T21:52:19.6153013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6153090Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6153364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6153437Z layer_outputs = layer_module( 2025-08-14T21:52:19.6153695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6153778Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6154046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6154148Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6154411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6154511Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6154516Z 2025-08-14T21:52:19.6154609Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6154801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6154862Z return mod(**inputs) 2025-08-14T21:52:19.6155130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6155195Z outputs = self.mobilebert( 2025-08-14T21:52:19.6155452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6155545Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6155800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6155865Z layer_outputs = layer_module( 2025-08-14T21:52:19.6156148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6156231Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6156489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6156601Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6156876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6156961Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6156978Z 2025-08-14T21:52:19.6157073Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6157259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6157319Z return mod(**inputs) 2025-08-14T21:52:19.6157574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6157645Z outputs = self.mobilebert( 2025-08-14T21:52:19.6157897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6157961Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6158222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6158285Z layer_outputs = layer_module( 2025-08-14T21:52:19.6158544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6158626Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6158877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6158997Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6159250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6159363Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6159616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6159700Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6159704Z 2025-08-14T21:52:19.6159806Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6159983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6160043Z return mod(**inputs) 2025-08-14T21:52:19.6160304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6160368Z outputs = self.mobilebert( 2025-08-14T21:52:19.6160626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6160691Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6160941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6161014Z layer_outputs = layer_module( 2025-08-14T21:52:19.6161284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6161375Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6161626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6161740Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6161998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6162074Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6162077Z 2025-08-14T21:52:19.6162176Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6162372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6162434Z return mod(**inputs) 2025-08-14T21:52:19.6162723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6162791Z outputs = self.mobilebert( 2025-08-14T21:52:19.6163049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6163127Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6163384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6163458Z layer_outputs = layer_module( 2025-08-14T21:52:19.6163716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6163801Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6164065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6164170Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6164426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6164536Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6164541Z 2025-08-14T21:52:19.6164634Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6164822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6164884Z return mod(**inputs) 2025-08-14T21:52:19.6165143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6165215Z outputs = self.mobilebert( 2025-08-14T21:52:19.6165540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6165626Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6165893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6165962Z layer_outputs = layer_module( 2025-08-14T21:52:19.6166235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6166325Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6166588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6166713Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6166977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6167089Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6167093Z 2025-08-14T21:52:19.6167189Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6167374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6167445Z return mod(**inputs) 2025-08-14T21:52:19.6167727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6167802Z outputs = self.mobilebert( 2025-08-14T21:52:19.6168060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6168129Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6168414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6168484Z layer_outputs = layer_module( 2025-08-14T21:52:19.6168761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6168856Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6169114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6169237Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6169495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6169609Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6169875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6169962Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6169965Z 2025-08-14T21:52:19.6170070Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6170255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6170315Z return mod(**inputs) 2025-08-14T21:52:19.6170584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6170651Z outputs = self.mobilebert( 2025-08-14T21:52:19.6170916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6170982Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6171240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6171312Z layer_outputs = layer_module( 2025-08-14T21:52:19.6171572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6171682Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6171949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6172027Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6172030Z 2025-08-14T21:52:19.6172132Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6172314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6172373Z return mod(**inputs) 2025-08-14T21:52:19.6172642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6172708Z outputs = self.mobilebert( 2025-08-14T21:52:19.6172989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6173056Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6173313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6173405Z layer_outputs = layer_module( 2025-08-14T21:52:19.6173667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6173774Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6174042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6174144Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6174164Z 2025-08-14T21:52:19.6174271Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6174471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6174534Z return mod(**inputs) 2025-08-14T21:52:19.6174802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6174868Z outputs = self.mobilebert( 2025-08-14T21:52:19.6175133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6175201Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6175457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6175532Z layer_outputs = layer_module( 2025-08-14T21:52:19.6175789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6175941Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6176211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.6176296Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.6176301Z 2025-08-14T21:52:19.6176401Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6176580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6176640Z return mod(**inputs) 2025-08-14T21:52:19.6176902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6176965Z outputs = self.mobilebert( 2025-08-14T21:52:19.6177222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6177291Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6177542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6177615Z layer_outputs = layer_module( 2025-08-14T21:52:19.6177865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6178009Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6178268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.6178379Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.6178638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6178742Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6178745Z 2025-08-14T21:52:19.6178840Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6179031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6179117Z return mod(**inputs) 2025-08-14T21:52:19.6179382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6179445Z outputs = self.mobilebert( 2025-08-14T21:52:19.6179704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6179778Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6180057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6180130Z layer_outputs = layer_module( 2025-08-14T21:52:19.6180400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6180545Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6180808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6180919Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6181174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.6181259Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6181263Z 2025-08-14T21:52:19.6181356Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6181542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6181604Z return mod(**inputs) 2025-08-14T21:52:19.6181861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6181932Z outputs = self.mobilebert( 2025-08-14T21:52:19.6182189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6182264Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6182524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6182590Z layer_outputs = layer_module( 2025-08-14T21:52:19.6182858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6183004Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6183268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6183392Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6183654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.6183774Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6184046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6184129Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6184133Z 2025-08-14T21:52:19.6184235Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6184419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6184503Z return mod(**inputs) 2025-08-14T21:52:19.6184763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6184828Z outputs = self.mobilebert( 2025-08-14T21:52:19.6185089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6185173Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6185425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6185498Z layer_outputs = layer_module( 2025-08-14T21:52:19.6185748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6185922Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6186194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6186297Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6186556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6186632Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6186636Z 2025-08-14T21:52:19.6186735Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6186915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6186975Z return mod(**inputs) 2025-08-14T21:52:19.6187239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6187305Z outputs = self.mobilebert( 2025-08-14T21:52:19.6187564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6187631Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6187883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6187955Z layer_outputs = layer_module( 2025-08-14T21:52:19.6188207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6188352Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6188611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6188711Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6188971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.6189049Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.6189302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6189394Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6189397Z 2025-08-14T21:52:19.6189489Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6189676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6189737Z return mod(**inputs) 2025-08-14T21:52:19.6189993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6190065Z outputs = self.mobilebert( 2025-08-14T21:52:19.6190336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6190402Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6190665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6190745Z layer_outputs = layer_module( 2025-08-14T21:52:19.6191001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6191079Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6191331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6191404Z self_outputs = self.self( 2025-08-14T21:52:19.6191671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.6191746Z self.query(query_tensor) 2025-08-14T21:52:19.6191765Z 2025-08-14T21:52:19.6191861Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6192041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6192107Z return mod(**inputs) 2025-08-14T21:52:19.6192369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6192436Z outputs = self.mobilebert( 2025-08-14T21:52:19.6192714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6192785Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6193066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6193136Z layer_outputs = layer_module( 2025-08-14T21:52:19.6193406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6193498Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6193764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6193841Z self_outputs = self.self( 2025-08-14T21:52:19.6194109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.6194174Z self.key(key_tensor) 2025-08-14T21:52:19.6194177Z 2025-08-14T21:52:19.6194281Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6194470Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6194544Z return mod(**inputs) 2025-08-14T21:52:19.6194810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6194875Z outputs = self.mobilebert( 2025-08-14T21:52:19.6195138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6195207Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6195464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6195537Z layer_outputs = layer_module( 2025-08-14T21:52:19.6195801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6195889Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6196159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6196248Z self_outputs = self.self( 2025-08-14T21:52:19.6196522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.6196590Z self.value(value_tensor) 2025-08-14T21:52:19.6196611Z 2025-08-14T21:52:19.6196691Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6196775Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6196874Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6197070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6197133Z return mod(**inputs) 2025-08-14T21:52:19.6197408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6197518Z outputs = self.mobilebert( 2025-08-14T21:52:19.6197807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6197878Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6198152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6198222Z layer_outputs = layer_module( 2025-08-14T21:52:19.6198496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6198575Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6198841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6198964Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6199231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.6199320Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6199324Z 2025-08-14T21:52:19.6199421Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6199609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6199683Z return mod(**inputs) 2025-08-14T21:52:19.6199954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6200020Z outputs = self.mobilebert( 2025-08-14T21:52:19.6200292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6200361Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6200640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6200707Z layer_outputs = layer_module( 2025-08-14T21:52:19.6200975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6201140Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6201418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.6201534Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.6201811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6201890Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6201893Z 2025-08-14T21:52:19.6202001Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6202224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6202291Z return mod(**inputs) 2025-08-14T21:52:19.6202589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6202660Z outputs = self.mobilebert( 2025-08-14T21:52:19.6202978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6203055Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6203353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6203435Z layer_outputs = layer_module( 2025-08-14T21:52:19.6203749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6203847Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6204163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6204295Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6204598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.6204734Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6205035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6205140Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6205144Z 2025-08-14T21:52:19.6205252Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6205556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6205636Z return mod(**inputs) 2025-08-14T21:52:19.6205945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6206029Z outputs = self.mobilebert( 2025-08-14T21:52:19.6206329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6206418Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6206717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6206804Z layer_outputs = layer_module( 2025-08-14T21:52:19.6207082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6207176Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6207447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6207563Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6207831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6207922Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6207925Z 2025-08-14T21:52:19.6208024Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6208213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6208284Z return mod(**inputs) 2025-08-14T21:52:19.6208554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6208630Z outputs = self.mobilebert( 2025-08-14T21:52:19.6208926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6208996Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6209271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6209356Z layer_outputs = layer_module( 2025-08-14T21:52:19.6209624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6209723Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6209988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6210107Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6210415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6210552Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6210557Z 2025-08-14T21:52:19.6210675Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6210880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6210958Z return mod(**inputs) 2025-08-14T21:52:19.6211253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6211326Z outputs = self.mobilebert( 2025-08-14T21:52:19.6211631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6211704Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6212001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6212077Z layer_outputs = layer_module( 2025-08-14T21:52:19.6212370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6212473Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6212764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6212895Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6213193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6213281Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6213286Z 2025-08-14T21:52:19.6213396Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6213602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6213673Z return mod(**inputs) 2025-08-14T21:52:19.6213975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6214046Z outputs = self.mobilebert( 2025-08-14T21:52:19.6214342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6214416Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6214706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6214789Z layer_outputs = layer_module( 2025-08-14T21:52:19.6215079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6215175Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6215489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6215619Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6215914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6216059Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6216359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6216461Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6216465Z 2025-08-14T21:52:19.6216572Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6216805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6216877Z return mod(**inputs) 2025-08-14T21:52:19.6217190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6217273Z outputs = self.mobilebert( 2025-08-14T21:52:19.6217565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6217642Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6217940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6218014Z layer_outputs = layer_module( 2025-08-14T21:52:19.6218308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6218406Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6218702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6218825Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6219115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6219211Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6219214Z 2025-08-14T21:52:19.6219322Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6219526Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6219601Z return mod(**inputs) 2025-08-14T21:52:19.6219896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6219968Z outputs = self.mobilebert( 2025-08-14T21:52:19.6220268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6220345Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6220642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6220718Z layer_outputs = layer_module( 2025-08-14T21:52:19.6221006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6221108Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6221400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6221517Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6221806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6221940Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6221944Z 2025-08-14T21:52:19.6222057Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6222263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6222354Z return mod(**inputs) 2025-08-14T21:52:19.6222649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6222722Z outputs = self.mobilebert( 2025-08-14T21:52:19.6223020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6223095Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6223404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6223488Z layer_outputs = layer_module( 2025-08-14T21:52:19.6223804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6223910Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6224200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6224329Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6224624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6224712Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6224716Z 2025-08-14T21:52:19.6224828Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6225035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6225106Z return mod(**inputs) 2025-08-14T21:52:19.6225405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6225476Z outputs = self.mobilebert( 2025-08-14T21:52:19.6225765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6225848Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6226138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6226217Z layer_outputs = layer_module( 2025-08-14T21:52:19.6226506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6226604Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6226907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6227035Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6227332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6227458Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6227749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6227852Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6227856Z 2025-08-14T21:52:19.6227962Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6228170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6228287Z return mod(**inputs) 2025-08-14T21:52:19.6228585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6228665Z outputs = self.mobilebert( 2025-08-14T21:52:19.6228954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6229047Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6229353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6229426Z layer_outputs = layer_module( 2025-08-14T21:52:19.6229729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6229842Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6230153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6230280Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6230568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6230656Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6230668Z 2025-08-14T21:52:19.6230772Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6230979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6231054Z return mod(**inputs) 2025-08-14T21:52:19.6231344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6231418Z outputs = self.mobilebert( 2025-08-14T21:52:19.6231720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6231795Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6232094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6232169Z layer_outputs = layer_module( 2025-08-14T21:52:19.6232459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6232562Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6232850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6232964Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6233262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6233380Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6233383Z 2025-08-14T21:52:19.6233496Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6233704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6233773Z return mod(**inputs) 2025-08-14T21:52:19.6234076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6234150Z outputs = self.mobilebert( 2025-08-14T21:52:19.6234447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6234522Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6234812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6234916Z layer_outputs = layer_module( 2025-08-14T21:52:19.6235212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6235308Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6235623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6235751Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6236046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6236133Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6236137Z 2025-08-14T21:52:19.6236242Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6236466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6236555Z return mod(**inputs) 2025-08-14T21:52:19.6236855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6236928Z outputs = self.mobilebert( 2025-08-14T21:52:19.6237219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6237299Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6237586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6237798Z layer_outputs = layer_module( 2025-08-14T21:52:19.6238104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6238203Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6238517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6238647Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6238947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6239085Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6239391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6239496Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6239500Z 2025-08-14T21:52:19.6239608Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6239828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6239910Z return mod(**inputs) 2025-08-14T21:52:19.6240219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6240299Z outputs = self.mobilebert( 2025-08-14T21:52:19.6240603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6240680Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6240990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6241064Z layer_outputs = layer_module( 2025-08-14T21:52:19.6241371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6241507Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6241860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6241955Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6241959Z 2025-08-14T21:52:19.6242071Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6242288Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6242391Z return mod(**inputs) 2025-08-14T21:52:19.6242700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6242780Z outputs = self.mobilebert( 2025-08-14T21:52:19.6243083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6243159Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6243490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6243590Z layer_outputs = layer_module( 2025-08-14T21:52:19.6243898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6244030Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6244334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6244457Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6244461Z 2025-08-14T21:52:19.6244568Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6244783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6244859Z return mod(**inputs) 2025-08-14T21:52:19.6245166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6245250Z outputs = self.mobilebert( 2025-08-14T21:52:19.6245611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6245698Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6246011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6246087Z layer_outputs = layer_module( 2025-08-14T21:52:19.6246392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6246558Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6246837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.6246940Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.6246945Z 2025-08-14T21:52:19.6247044Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6247236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6247311Z return mod(**inputs) 2025-08-14T21:52:19.6247590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6247667Z outputs = self.mobilebert( 2025-08-14T21:52:19.6247942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6248012Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6248297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6248389Z layer_outputs = layer_module( 2025-08-14T21:52:19.6248667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6248829Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6249122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.6249253Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.6249527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6249617Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6249620Z 2025-08-14T21:52:19.6249750Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6249949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6250037Z return mod(**inputs) 2025-08-14T21:52:19.6250318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6250387Z outputs = self.mobilebert( 2025-08-14T21:52:19.6250671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6250742Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6251017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6251094Z layer_outputs = layer_module( 2025-08-14T21:52:19.6251373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6251533Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6251811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6251929Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6252215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.6252297Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6252301Z 2025-08-14T21:52:19.6252409Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6252606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6252671Z return mod(**inputs) 2025-08-14T21:52:19.6252956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6253026Z outputs = self.mobilebert( 2025-08-14T21:52:19.6253327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6253393Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6253653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6253727Z layer_outputs = layer_module( 2025-08-14T21:52:19.6253984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6254130Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6254400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6254510Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6254800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.6254910Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6255173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6255284Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6255287Z 2025-08-14T21:52:19.6255382Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6255573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6255634Z return mod(**inputs) 2025-08-14T21:52:19.6255914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6255990Z outputs = self.mobilebert( 2025-08-14T21:52:19.6256261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6256329Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6256599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6256667Z layer_outputs = layer_module( 2025-08-14T21:52:19.6256935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6257085Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6257347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6257458Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6257721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6257806Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6257809Z 2025-08-14T21:52:19.6257903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6258090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6258159Z return mod(**inputs) 2025-08-14T21:52:19.6258419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6258491Z outputs = self.mobilebert( 2025-08-14T21:52:19.6258751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6258820Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6259091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6259157Z layer_outputs = layer_module( 2025-08-14T21:52:19.6259417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6259575Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6259839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6259948Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6260205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.6260288Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.6260571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6260656Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6260659Z 2025-08-14T21:52:19.6260762Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6260949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6261030Z return mod(**inputs) 2025-08-14T21:52:19.6261299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6261365Z outputs = self.mobilebert( 2025-08-14T21:52:19.6261623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6261698Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6261976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6262071Z layer_outputs = layer_module( 2025-08-14T21:52:19.6262333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6262412Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6262685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6262752Z self_outputs = self.self( 2025-08-14T21:52:19.6263020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.6263087Z self.query(query_tensor) 2025-08-14T21:52:19.6263091Z 2025-08-14T21:52:19.6263185Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6263378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6263443Z return mod(**inputs) 2025-08-14T21:52:19.6263705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6263777Z outputs = self.mobilebert( 2025-08-14T21:52:19.6264034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6264108Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6264364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6264430Z layer_outputs = layer_module( 2025-08-14T21:52:19.6264695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6264776Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6265042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6265109Z self_outputs = self.self( 2025-08-14T21:52:19.6265368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.6265440Z self.key(key_tensor) 2025-08-14T21:52:19.6265443Z 2025-08-14T21:52:19.6265537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6265720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6265790Z return mod(**inputs) 2025-08-14T21:52:19.6266053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6266127Z outputs = self.mobilebert( 2025-08-14T21:52:19.6266395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6266482Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6266760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6266847Z layer_outputs = layer_module( 2025-08-14T21:52:19.6267114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6267203Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6267470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6267544Z self_outputs = self.self( 2025-08-14T21:52:19.6267826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.6267896Z self.value(value_tensor) 2025-08-14T21:52:19.6267899Z 2025-08-14T21:52:19.6268003Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6268081Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6268185Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6268378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6268441Z return mod(**inputs) 2025-08-14T21:52:19.6268730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6268795Z outputs = self.mobilebert( 2025-08-14T21:52:19.6269067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6269141Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6269410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6269486Z layer_outputs = layer_module( 2025-08-14T21:52:19.6269751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6269828Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6270103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6270218Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6270489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.6270568Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6270572Z 2025-08-14T21:52:19.6270669Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6270865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6270927Z return mod(**inputs) 2025-08-14T21:52:19.6271195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6271269Z outputs = self.mobilebert( 2025-08-14T21:52:19.6271535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6271610Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6271883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6271948Z layer_outputs = layer_module( 2025-08-14T21:52:19.6272224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6272391Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6272661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.6272768Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.6273062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6273147Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6273151Z 2025-08-14T21:52:19.6273248Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6273435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6273505Z return mod(**inputs) 2025-08-14T21:52:19.6273791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6273869Z outputs = self.mobilebert( 2025-08-14T21:52:19.6274153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6274225Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6274502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6274572Z layer_outputs = layer_module( 2025-08-14T21:52:19.6274855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6274933Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6275194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6275319Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6275593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.6275715Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6275991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6276078Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6276082Z 2025-08-14T21:52:19.6276185Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6276374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6276437Z return mod(**inputs) 2025-08-14T21:52:19.6276717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6276785Z outputs = self.mobilebert( 2025-08-14T21:52:19.6277060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6277130Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6277398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6277474Z layer_outputs = layer_module( 2025-08-14T21:52:19.6277740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6277831Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6278105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6278210Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6278503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6278583Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6278586Z 2025-08-14T21:52:19.6278683Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6278880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6278961Z return mod(**inputs) 2025-08-14T21:52:19.6279238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6279306Z outputs = self.mobilebert( 2025-08-14T21:52:19.6279572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6279651Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6279937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6280023Z layer_outputs = layer_module( 2025-08-14T21:52:19.6280298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6280389Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6280664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6280771Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6281039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6281153Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6281157Z 2025-08-14T21:52:19.6281255Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6281455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6281517Z return mod(**inputs) 2025-08-14T21:52:19.6281783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6281859Z outputs = self.mobilebert( 2025-08-14T21:52:19.6282128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6282202Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6282470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6282538Z layer_outputs = layer_module( 2025-08-14T21:52:19.6282815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6282906Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6283172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6283299Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6283563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6283647Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6283650Z 2025-08-14T21:52:19.6283749Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6283938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6284007Z return mod(**inputs) 2025-08-14T21:52:19.6284275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6284366Z outputs = self.mobilebert( 2025-08-14T21:52:19.6284640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6284710Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6284988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6285071Z layer_outputs = layer_module( 2025-08-14T21:52:19.6285340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6285502Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6285778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6285926Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6286246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6286380Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6286692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6286802Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6286805Z 2025-08-14T21:52:19.6286915Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6287109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6287175Z return mod(**inputs) 2025-08-14T21:52:19.6287477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6287547Z outputs = self.mobilebert( 2025-08-14T21:52:19.6287819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6287897Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6288167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6288244Z layer_outputs = layer_module( 2025-08-14T21:52:19.6288518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6288609Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6288885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6288990Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6289274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6289355Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6289358Z 2025-08-14T21:52:19.6289456Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6289663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6289726Z return mod(**inputs) 2025-08-14T21:52:19.6289993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6290066Z outputs = self.mobilebert( 2025-08-14T21:52:19.6290331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6290408Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6290679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6290767Z layer_outputs = layer_module( 2025-08-14T21:52:19.6291041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6291129Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6291426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6291531Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6291797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6291912Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6291916Z 2025-08-14T21:52:19.6292028Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6292252Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6292315Z return mod(**inputs) 2025-08-14T21:52:19.6292578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6292654Z outputs = self.mobilebert( 2025-08-14T21:52:19.6292921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6292990Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6293261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6293330Z layer_outputs = layer_module( 2025-08-14T21:52:19.6293604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6293695Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6293962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6294090Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6294360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6294445Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6294449Z 2025-08-14T21:52:19.6294548Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6294738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6294808Z return mod(**inputs) 2025-08-14T21:52:19.6295080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6295149Z outputs = self.mobilebert( 2025-08-14T21:52:19.6295427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6295497Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6295774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6295844Z layer_outputs = layer_module( 2025-08-14T21:52:19.6296109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6296205Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6296481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6296607Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6296900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6297013Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6297278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6297385Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6297389Z 2025-08-14T21:52:19.6297487Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6297684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6297750Z return mod(**inputs) 2025-08-14T21:52:19.6298037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6298104Z outputs = self.mobilebert( 2025-08-14T21:52:19.6298381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6298460Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6298719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6298797Z layer_outputs = layer_module( 2025-08-14T21:52:19.6299056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6299141Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6299400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6299502Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6299761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6299847Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6299851Z 2025-08-14T21:52:19.6299943Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6300132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6300194Z return mod(**inputs) 2025-08-14T21:52:19.6300458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6300532Z outputs = self.mobilebert( 2025-08-14T21:52:19.6300793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6300867Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6301124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6301194Z layer_outputs = layer_module( 2025-08-14T21:52:19.6301456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6301541Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6301799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6301907Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6302166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6302272Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6302275Z 2025-08-14T21:52:19.6302371Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6302572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6302643Z return mod(**inputs) 2025-08-14T21:52:19.6302911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6302985Z outputs = self.mobilebert( 2025-08-14T21:52:19.6303272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6303340Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6303613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6303680Z layer_outputs = layer_module( 2025-08-14T21:52:19.6303960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6304058Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6304336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6304467Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6304735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6304817Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6304821Z 2025-08-14T21:52:19.6304926Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6305116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6305189Z return mod(**inputs) 2025-08-14T21:52:19.6305469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6305540Z outputs = self.mobilebert( 2025-08-14T21:52:19.6305836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6305906Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6306175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6306252Z layer_outputs = layer_module( 2025-08-14T21:52:19.6306525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6306624Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6306898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6307021Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6307306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6307424Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6307709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6307799Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6307803Z 2025-08-14T21:52:19.6307904Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6308108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6308173Z return mod(**inputs) 2025-08-14T21:52:19.6308457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6308525Z outputs = self.mobilebert( 2025-08-14T21:52:19.6308821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6308899Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6309175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6309262Z layer_outputs = layer_module( 2025-08-14T21:52:19.6309540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6309658Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6309933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6310014Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6310036Z 2025-08-14T21:52:19.6310141Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6310357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6310424Z return mod(**inputs) 2025-08-14T21:52:19.6310706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6310776Z outputs = self.mobilebert( 2025-08-14T21:52:19.6311049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6311126Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6311400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6311471Z layer_outputs = layer_module( 2025-08-14T21:52:19.6311751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6311869Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6312150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6312258Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6312264Z 2025-08-14T21:52:19.6312363Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6312565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6312630Z return mod(**inputs) 2025-08-14T21:52:19.6312911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6312981Z outputs = self.mobilebert( 2025-08-14T21:52:19.6313257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6313338Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6313609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6313678Z layer_outputs = layer_module( 2025-08-14T21:52:19.6313961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6314119Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6314396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.6314489Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.6314494Z 2025-08-14T21:52:19.6314594Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6314816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6314881Z return mod(**inputs) 2025-08-14T21:52:19.6315165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6315233Z outputs = self.mobilebert( 2025-08-14T21:52:19.6315528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6315607Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6315879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6315948Z layer_outputs = layer_module( 2025-08-14T21:52:19.6316245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6316405Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6316718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.6316839Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.6317123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6317221Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6317225Z 2025-08-14T21:52:19.6317326Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6317528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6317593Z return mod(**inputs) 2025-08-14T21:52:19.6317876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6317951Z outputs = self.mobilebert( 2025-08-14T21:52:19.6318240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6318307Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6318574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6318642Z layer_outputs = layer_module( 2025-08-14T21:52:19.6318912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6319057Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6319321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6319444Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6319710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.6319793Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6319796Z 2025-08-14T21:52:19.6319891Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6320076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6320146Z return mod(**inputs) 2025-08-14T21:52:19.6320411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6320484Z outputs = self.mobilebert( 2025-08-14T21:52:19.6320749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6320833Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6321103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6321168Z layer_outputs = layer_module( 2025-08-14T21:52:19.6321427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6321600Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6321860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6321979Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6322238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.6322365Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6322676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6322766Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6322770Z 2025-08-14T21:52:19.6322875Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6323066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6323131Z return mod(**inputs) 2025-08-14T21:52:19.6323406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6323474Z outputs = self.mobilebert( 2025-08-14T21:52:19.6323744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6323822Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6324090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6324166Z layer_outputs = layer_module( 2025-08-14T21:52:19.6324433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6324589Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6324864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6324968Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6325250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6325330Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6325335Z 2025-08-14T21:52:19.6325495Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6325710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6325774Z return mod(**inputs) 2025-08-14T21:52:19.6326055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6326125Z outputs = self.mobilebert( 2025-08-14T21:52:19.6326407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6326490Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6326781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6326855Z layer_outputs = layer_module( 2025-08-14T21:52:19.6327158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6327348Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6327656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6327781Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6328057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.6328148Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.6328423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6328520Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6328540Z 2025-08-14T21:52:19.6328644Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6328855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6328927Z return mod(**inputs) 2025-08-14T21:52:19.6329210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6329281Z outputs = self.mobilebert( 2025-08-14T21:52:19.6329568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6329638Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6329911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6329981Z layer_outputs = layer_module( 2025-08-14T21:52:19.6330246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6330339Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6330606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6330682Z self_outputs = self.self( 2025-08-14T21:52:19.6330958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.6331028Z self.query(query_tensor) 2025-08-14T21:52:19.6331032Z 2025-08-14T21:52:19.6331139Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6331332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6331396Z return mod(**inputs) 2025-08-14T21:52:19.6331681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6331749Z outputs = self.mobilebert( 2025-08-14T21:52:19.6332026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6332097Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6332370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6332452Z layer_outputs = layer_module( 2025-08-14T21:52:19.6332739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6332833Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6333120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6333196Z self_outputs = self.self( 2025-08-14T21:52:19.6333511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.6333579Z self.key(key_tensor) 2025-08-14T21:52:19.6333583Z 2025-08-14T21:52:19.6333688Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6333898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6333984Z return mod(**inputs) 2025-08-14T21:52:19.6334291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6334360Z outputs = self.mobilebert( 2025-08-14T21:52:19.6334633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6334711Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6335012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6335923Z layer_outputs = layer_module( 2025-08-14T21:52:19.6336232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6336320Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6336617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6336688Z self_outputs = self.self( 2025-08-14T21:52:19.6336974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.6337056Z self.value(value_tensor) 2025-08-14T21:52:19.6337060Z 2025-08-14T21:52:19.6337146Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6337237Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6337345Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6337564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6337789Z return mod(**inputs) 2025-08-14T21:52:19.6338096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6338173Z outputs = self.mobilebert( 2025-08-14T21:52:19.6338481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6338557Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6338854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6338928Z layer_outputs = layer_module( 2025-08-14T21:52:19.6339223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6339321Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6339611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6339750Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6340043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.6340133Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6340137Z 2025-08-14T21:52:19.6340251Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6340477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6340546Z return mod(**inputs) 2025-08-14T21:52:19.6340850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6340968Z outputs = self.mobilebert( 2025-08-14T21:52:19.6341269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6341346Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6341661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6341742Z layer_outputs = layer_module( 2025-08-14T21:52:19.6342030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6342202Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6342526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.6342644Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.6342971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6343059Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6343063Z 2025-08-14T21:52:19.6343174Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6343402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6343471Z return mod(**inputs) 2025-08-14T21:52:19.6343773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6343854Z outputs = self.mobilebert( 2025-08-14T21:52:19.6344115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6344192Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6344452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6344525Z layer_outputs = layer_module( 2025-08-14T21:52:19.6344785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6344864Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6345131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6345245Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6345507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.6345632Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6345896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6345988Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6345991Z 2025-08-14T21:52:19.6346086Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6346271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6346340Z return mod(**inputs) 2025-08-14T21:52:19.6346602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6346678Z outputs = self.mobilebert( 2025-08-14T21:52:19.6346938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6347007Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6347299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6347366Z layer_outputs = layer_module( 2025-08-14T21:52:19.6347626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6347743Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6348001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6348111Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6348369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6348447Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6348464Z 2025-08-14T21:52:19.6348569Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6348770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6348840Z return mod(**inputs) 2025-08-14T21:52:19.6349107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6349175Z outputs = self.mobilebert( 2025-08-14T21:52:19.6349441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6349509Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6349772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6349846Z layer_outputs = layer_module( 2025-08-14T21:52:19.6350106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6350203Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6350463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6350566Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6350833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6350935Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6350939Z 2025-08-14T21:52:19.6351038Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6351223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6351284Z return mod(**inputs) 2025-08-14T21:52:19.6351555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6351624Z outputs = self.mobilebert( 2025-08-14T21:52:19.6351884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6351957Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6352217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6352290Z layer_outputs = layer_module( 2025-08-14T21:52:19.6352547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6352635Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6352903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6353043Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6353312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6353390Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6353393Z 2025-08-14T21:52:19.6353507Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6353698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6353762Z return mod(**inputs) 2025-08-14T21:52:19.6354035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6354103Z outputs = self.mobilebert( 2025-08-14T21:52:19.6354377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6354454Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6354726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6354794Z layer_outputs = layer_module( 2025-08-14T21:52:19.6355065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6355152Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6355424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6355542Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6355803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6355925Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6356187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6356282Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6356285Z 2025-08-14T21:52:19.6356379Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6356569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6356637Z return mod(**inputs) 2025-08-14T21:52:19.6356899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6356966Z outputs = self.mobilebert( 2025-08-14T21:52:19.6357234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6357303Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6357574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6357639Z layer_outputs = layer_module( 2025-08-14T21:52:19.6357903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6357999Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6358263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6358372Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6358634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6358712Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6358717Z 2025-08-14T21:52:19.6358838Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6359025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6359086Z return mod(**inputs) 2025-08-14T21:52:19.6359354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6359434Z outputs = self.mobilebert( 2025-08-14T21:52:19.6359697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6359765Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6360030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6360103Z layer_outputs = layer_module( 2025-08-14T21:52:19.6360371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6360481Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6360736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6360833Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6361094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6361194Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6361197Z 2025-08-14T21:52:19.6361297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6361476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6361537Z return mod(**inputs) 2025-08-14T21:52:19.6361799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6361864Z outputs = self.mobilebert( 2025-08-14T21:52:19.6362117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6362192Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6362450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6362523Z layer_outputs = layer_module( 2025-08-14T21:52:19.6362782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6362870Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6363142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6363260Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6363526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6363615Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6363619Z 2025-08-14T21:52:19.6363718Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6363913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6363975Z return mod(**inputs) 2025-08-14T21:52:19.6364245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6364318Z outputs = self.mobilebert( 2025-08-14T21:52:19.6364584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6364680Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6364948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6365015Z layer_outputs = layer_module( 2025-08-14T21:52:19.6365287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6365458Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6365732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6365858Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6366124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6366268Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6366560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6366645Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6366649Z 2025-08-14T21:52:19.6366750Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6366933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6367005Z return mod(**inputs) 2025-08-14T21:52:19.6367265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6367333Z outputs = self.mobilebert( 2025-08-14T21:52:19.6367600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6367674Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6367948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6368020Z layer_outputs = layer_module( 2025-08-14T21:52:19.6368284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6368384Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6368645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6368750Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6369029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6369109Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6369112Z 2025-08-14T21:52:19.6369216Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6369403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6369466Z return mod(**inputs) 2025-08-14T21:52:19.6369731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6369803Z outputs = self.mobilebert( 2025-08-14T21:52:19.6370065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6370135Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6370392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6370465Z layer_outputs = layer_module( 2025-08-14T21:52:19.6370722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6370825Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6371088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6371190Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6371465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6371570Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6371573Z 2025-08-14T21:52:19.6371671Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6371861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6371924Z return mod(**inputs) 2025-08-14T21:52:19.6372206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6372290Z outputs = self.mobilebert( 2025-08-14T21:52:19.6372550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6372624Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6372882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6372948Z layer_outputs = layer_module( 2025-08-14T21:52:19.6373216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6373303Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6373569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6373687Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6373947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6374033Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6374036Z 2025-08-14T21:52:19.6374130Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6374331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6374390Z return mod(**inputs) 2025-08-14T21:52:19.6374645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6374717Z outputs = self.mobilebert( 2025-08-14T21:52:19.6374972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6375040Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6375300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6375366Z layer_outputs = layer_module( 2025-08-14T21:52:19.6375621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6375706Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6375976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6376097Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6376354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6376473Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6376756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6376841Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6376844Z 2025-08-14T21:52:19.6376950Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6377154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6377223Z return mod(**inputs) 2025-08-14T21:52:19.6377492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6377558Z outputs = self.mobilebert( 2025-08-14T21:52:19.6377830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6377913Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6378197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6378284Z layer_outputs = layer_module( 2025-08-14T21:52:19.6378539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6378657Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6378910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6378988Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6378991Z 2025-08-14T21:52:19.6379090Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6379310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6379382Z return mod(**inputs) 2025-08-14T21:52:19.6379653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6379718Z outputs = self.mobilebert( 2025-08-14T21:52:19.6379984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6380053Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6380313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6380384Z layer_outputs = layer_module( 2025-08-14T21:52:19.6380641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6380756Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6381018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6381123Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6381127Z 2025-08-14T21:52:19.6381227Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6381413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6381484Z return mod(**inputs) 2025-08-14T21:52:19.6381747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6381812Z outputs = self.mobilebert( 2025-08-14T21:52:19.6382076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6382145Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6382405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6382509Z layer_outputs = layer_module( 2025-08-14T21:52:19.6382773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6382927Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6383205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.6383293Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.6383300Z 2025-08-14T21:52:19.6383403Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6383584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6383654Z return mod(**inputs) 2025-08-14T21:52:19.6383930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6383999Z outputs = self.mobilebert( 2025-08-14T21:52:19.6384281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6384350Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6384611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6384686Z layer_outputs = layer_module( 2025-08-14T21:52:19.6384947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6385104Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6385362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.6385479Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.6385748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6385835Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6385838Z 2025-08-14T21:52:19.6385940Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6386123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6386185Z return mod(**inputs) 2025-08-14T21:52:19.6386456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6386520Z outputs = self.mobilebert( 2025-08-14T21:52:19.6386779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6386856Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6387116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6387189Z layer_outputs = layer_module( 2025-08-14T21:52:19.6387445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6387592Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6387859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6387972Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6388239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.6388348Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6388351Z 2025-08-14T21:52:19.6388448Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6388647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6388710Z return mod(**inputs) 2025-08-14T21:52:19.6388979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6389061Z outputs = self.mobilebert( 2025-08-14T21:52:19.6389331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6389405Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6389676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6389758Z layer_outputs = layer_module( 2025-08-14T21:52:19.6390043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6390193Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6390480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6390595Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6390856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.6390977Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6391236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6391329Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6391333Z 2025-08-14T21:52:19.6391431Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6391615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6391684Z return mod(**inputs) 2025-08-14T21:52:19.6391947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6392014Z outputs = self.mobilebert( 2025-08-14T21:52:19.6392283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6392355Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6392638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6392710Z layer_outputs = layer_module( 2025-08-14T21:52:19.6392994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6393157Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6393427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6393544Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6393811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6393890Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6393894Z 2025-08-14T21:52:19.6393998Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6394201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6394269Z return mod(**inputs) 2025-08-14T21:52:19.6394550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6394616Z outputs = self.mobilebert( 2025-08-14T21:52:19.6394879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6394965Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6395224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6395297Z layer_outputs = layer_module( 2025-08-14T21:52:19.6395553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6395707Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6395981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6396103Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6396372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.6396453Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.6396716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6396801Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6396804Z 2025-08-14T21:52:19.6396898Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6397087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6397149Z return mod(**inputs) 2025-08-14T21:52:19.6397413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6397484Z outputs = self.mobilebert( 2025-08-14T21:52:19.6397740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6397817Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6398076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6398143Z layer_outputs = layer_module( 2025-08-14T21:52:19.6398407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6398486Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6398751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6398819Z self_outputs = self.self( 2025-08-14T21:52:19.6399077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.6399149Z self.query(query_tensor) 2025-08-14T21:52:19.6399152Z 2025-08-14T21:52:19.6399249Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6399433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6399503Z return mod(**inputs) 2025-08-14T21:52:19.6399764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6399837Z outputs = self.mobilebert( 2025-08-14T21:52:19.6400096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6400184Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6400456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6400522Z layer_outputs = layer_module( 2025-08-14T21:52:19.6400791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6400888Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6401145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6401218Z self_outputs = self.self( 2025-08-14T21:52:19.6401477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.6401539Z self.key(key_tensor) 2025-08-14T21:52:19.6401543Z 2025-08-14T21:52:19.6401661Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6401865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6401936Z return mod(**inputs) 2025-08-14T21:52:19.6402199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6402266Z outputs = self.mobilebert( 2025-08-14T21:52:19.6402531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6402599Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6402856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6402933Z layer_outputs = layer_module( 2025-08-14T21:52:19.6403198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6403285Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6403551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6403618Z self_outputs = self.self( 2025-08-14T21:52:19.6403889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.6403959Z self.value(value_tensor) 2025-08-14T21:52:19.6403962Z 2025-08-14T21:52:19.6404047Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6404121Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6404219Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6404412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6404478Z return mod(**inputs) 2025-08-14T21:52:19.6404747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6404820Z outputs = self.mobilebert( 2025-08-14T21:52:19.6405088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6405165Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6405494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6405574Z layer_outputs = layer_module( 2025-08-14T21:52:19.6405874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6405964Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6406265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6406434Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6406713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.6406806Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6406827Z 2025-08-14T21:52:19.6406929Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6407125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6407196Z return mod(**inputs) 2025-08-14T21:52:19.6407458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6407532Z outputs = self.mobilebert( 2025-08-14T21:52:19.6407820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6407890Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6408173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6408239Z layer_outputs = layer_module( 2025-08-14T21:52:19.6408494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6408652Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6408907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.6409014Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.6409267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6409343Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6409355Z 2025-08-14T21:52:19.6409449Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6409628Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6409696Z return mod(**inputs) 2025-08-14T21:52:19.6409949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6410012Z outputs = self.mobilebert( 2025-08-14T21:52:19.6410267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6410331Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6410589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6410653Z layer_outputs = layer_module( 2025-08-14T21:52:19.6410904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6410986Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6411236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6411346Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6411603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.6411717Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6411974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6412059Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6412080Z 2025-08-14T21:52:19.6412174Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6412360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6412421Z return mod(**inputs) 2025-08-14T21:52:19.6412685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6412767Z outputs = self.mobilebert( 2025-08-14T21:52:19.6413029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6413103Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6413366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6413433Z layer_outputs = layer_module( 2025-08-14T21:52:19.6413723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6413829Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6414092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6414194Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6414446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6414529Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6414533Z 2025-08-14T21:52:19.6414625Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6414812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6414874Z return mod(**inputs) 2025-08-14T21:52:19.6415130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6415204Z outputs = self.mobilebert( 2025-08-14T21:52:19.6415456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6415524Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6415782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6415849Z layer_outputs = layer_module( 2025-08-14T21:52:19.6416108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6416196Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6416455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6416568Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6416827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6416940Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6416946Z 2025-08-14T21:52:19.6417040Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6417223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6417293Z return mod(**inputs) 2025-08-14T21:52:19.6417553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6417619Z outputs = self.mobilebert( 2025-08-14T21:52:19.6417887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6417972Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6418236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6418302Z layer_outputs = layer_module( 2025-08-14T21:52:19.6418557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6418666Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6418925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6419051Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6419353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6419437Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6419442Z 2025-08-14T21:52:19.6419558Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6419742Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6419809Z return mod(**inputs) 2025-08-14T21:52:19.6420069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6420136Z outputs = self.mobilebert( 2025-08-14T21:52:19.6420403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6420472Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6420733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6420807Z layer_outputs = layer_module( 2025-08-14T21:52:19.6421069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6421162Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6421418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6421537Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6421801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6421915Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6422185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6422274Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6422278Z 2025-08-14T21:52:19.6422375Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6422567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6422631Z return mod(**inputs) 2025-08-14T21:52:19.6422892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6422968Z outputs = self.mobilebert( 2025-08-14T21:52:19.6423230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6423303Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6423565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6423633Z layer_outputs = layer_module( 2025-08-14T21:52:19.6423899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6424008Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6424276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6424395Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6424654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6424740Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6424743Z 2025-08-14T21:52:19.6424839Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6425022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6425092Z return mod(**inputs) 2025-08-14T21:52:19.6425366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6425456Z outputs = self.mobilebert( 2025-08-14T21:52:19.6425716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6425783Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6426052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6426119Z layer_outputs = layer_module( 2025-08-14T21:52:19.6426383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6426467Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6426725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6426834Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6427094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6427198Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6427209Z 2025-08-14T21:52:19.6427304Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6427487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6427552Z return mod(**inputs) 2025-08-14T21:52:19.6427814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6427880Z outputs = self.mobilebert( 2025-08-14T21:52:19.6428145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6428213Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6428478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6428545Z layer_outputs = layer_module( 2025-08-14T21:52:19.6428801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6428894Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6429151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6429267Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6429532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6429626Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6429630Z 2025-08-14T21:52:19.6429732Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6429918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6429981Z return mod(**inputs) 2025-08-14T21:52:19.6430252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6430334Z outputs = self.mobilebert( 2025-08-14T21:52:19.6430602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6430669Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6430929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6431027Z layer_outputs = layer_module( 2025-08-14T21:52:19.6431304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6431393Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6431661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6431778Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6432047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6432161Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6432419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6432514Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6432518Z 2025-08-14T21:52:19.6432614Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6432810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6432876Z return mod(**inputs) 2025-08-14T21:52:19.6433153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6433232Z outputs = self.mobilebert( 2025-08-14T21:52:19.6433516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6433593Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6433860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6433929Z layer_outputs = layer_module( 2025-08-14T21:52:19.6434206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6434296Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6434564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6434688Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6434951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6435038Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6435041Z 2025-08-14T21:52:19.6435135Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6435319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6435388Z return mod(**inputs) 2025-08-14T21:52:19.6435715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6435818Z outputs = self.mobilebert( 2025-08-14T21:52:19.6436084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6436153Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6436448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6436517Z layer_outputs = layer_module( 2025-08-14T21:52:19.6436784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6436880Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6437160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6437274Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6437557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6437792Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6437801Z 2025-08-14T21:52:19.6437915Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6438109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6438181Z return mod(**inputs) 2025-08-14T21:52:19.6438464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6438533Z outputs = self.mobilebert( 2025-08-14T21:52:19.6438817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6438891Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6439168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6439246Z layer_outputs = layer_module( 2025-08-14T21:52:19.6439565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6439662Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6439931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6440051Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6440331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6440412Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6440417Z 2025-08-14T21:52:19.6440523Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6440713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6440776Z return mod(**inputs) 2025-08-14T21:52:19.6441052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6441121Z outputs = self.mobilebert( 2025-08-14T21:52:19.6441388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6441464Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6441732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6441807Z layer_outputs = layer_module( 2025-08-14T21:52:19.6442129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6442217Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6442489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6442629Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6442902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6443018Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6443283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6443406Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6443411Z 2025-08-14T21:52:19.6443511Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6443732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6443797Z return mod(**inputs) 2025-08-14T21:52:19.6444070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6444146Z outputs = self.mobilebert( 2025-08-14T21:52:19.6444416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6444485Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6444762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6444829Z layer_outputs = layer_module( 2025-08-14T21:52:19.6445106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6445223Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6445616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6445714Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6445718Z 2025-08-14T21:52:19.6445822Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6446036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6446106Z return mod(**inputs) 2025-08-14T21:52:19.6446396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6446485Z outputs = self.mobilebert( 2025-08-14T21:52:19.6446750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6446823Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6447097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6447167Z layer_outputs = layer_module( 2025-08-14T21:52:19.6447441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6447555Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6447818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6447932Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6447935Z 2025-08-14T21:52:19.6448034Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6448254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6448321Z return mod(**inputs) 2025-08-14T21:52:19.6448589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6448667Z outputs = self.mobilebert( 2025-08-14T21:52:19.6448960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6449029Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6449294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6449361Z layer_outputs = layer_module( 2025-08-14T21:52:19.6449639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6449793Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6450071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.6450169Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.6450173Z 2025-08-14T21:52:19.6450269Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6450465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6450526Z return mod(**inputs) 2025-08-14T21:52:19.6450786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6450858Z outputs = self.mobilebert( 2025-08-14T21:52:19.6451116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6451185Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6451451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6451519Z layer_outputs = layer_module( 2025-08-14T21:52:19.6451781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6451930Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6452186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.6452307Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.6452567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6452662Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6452666Z 2025-08-14T21:52:19.6452762Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6452946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6453017Z return mod(**inputs) 2025-08-14T21:52:19.6453279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6453343Z outputs = self.mobilebert( 2025-08-14T21:52:19.6453606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6453675Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6453938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6454007Z layer_outputs = layer_module( 2025-08-14T21:52:19.6454289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6454442Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6454701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6454850Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6455112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.6455190Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6455194Z 2025-08-14T21:52:19.6455297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6455493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6455565Z return mod(**inputs) 2025-08-14T21:52:19.6455847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6455913Z outputs = self.mobilebert( 2025-08-14T21:52:19.6456178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6456247Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6456503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6456577Z layer_outputs = layer_module( 2025-08-14T21:52:19.6456840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6456994Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6457257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6457372Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6457642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.6457755Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6458026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6458111Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6458114Z 2025-08-14T21:52:19.6458209Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6458400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6458463Z return mod(**inputs) 2025-08-14T21:52:19.6458729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6458801Z outputs = self.mobilebert( 2025-08-14T21:52:19.6459063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6459140Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6459405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6459473Z layer_outputs = layer_module( 2025-08-14T21:52:19.6459745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6459897Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6460184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6460287Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6460544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6460656Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6460659Z 2025-08-14T21:52:19.6460753Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6460940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6461002Z return mod(**inputs) 2025-08-14T21:52:19.6461259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6461345Z outputs = self.mobilebert( 2025-08-14T21:52:19.6461619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6461687Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6461948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6462015Z layer_outputs = layer_module( 2025-08-14T21:52:19.6462274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6462417Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6462680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6462790Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6463051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.6463143Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.6463401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6463486Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6463491Z 2025-08-14T21:52:19.6463596Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6463779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6463840Z return mod(**inputs) 2025-08-14T21:52:19.6464111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6464176Z outputs = self.mobilebert( 2025-08-14T21:52:19.6464448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6464516Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6464766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6464838Z layer_outputs = layer_module( 2025-08-14T21:52:19.6465090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6465175Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6465423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6465488Z self_outputs = self.self( 2025-08-14T21:52:19.6465751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.6465831Z self.query(query_tensor) 2025-08-14T21:52:19.6465834Z 2025-08-14T21:52:19.6465927Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6466113Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6466173Z return mod(**inputs) 2025-08-14T21:52:19.6466431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6466512Z outputs = self.mobilebert( 2025-08-14T21:52:19.6466766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6466839Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6467092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6467173Z layer_outputs = layer_module( 2025-08-14T21:52:19.6467454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6467533Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6467792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6467858Z self_outputs = self.self( 2025-08-14T21:52:19.6468107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.6468175Z self.key(key_tensor) 2025-08-14T21:52:19.6468179Z 2025-08-14T21:52:19.6468271Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6468456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6468516Z return mod(**inputs) 2025-08-14T21:52:19.6468768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6468843Z outputs = self.mobilebert( 2025-08-14T21:52:19.6469096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6469163Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6469427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6469491Z layer_outputs = layer_module( 2025-08-14T21:52:19.6469752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6469828Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6470080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6470154Z self_outputs = self.self( 2025-08-14T21:52:19.6470405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.6470478Z self.value(value_tensor) 2025-08-14T21:52:19.6470481Z 2025-08-14T21:52:19.6470555Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6470629Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6470729Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6470903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6470961Z return mod(**inputs) 2025-08-14T21:52:19.6471223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6471286Z outputs = self.mobilebert( 2025-08-14T21:52:19.6471545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6471635Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6471896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6471969Z layer_outputs = layer_module( 2025-08-14T21:52:19.6472251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6472330Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6472606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6472723Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6473015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.6473099Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6473102Z 2025-08-14T21:52:19.6473214Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6473414Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6473477Z return mod(**inputs) 2025-08-14T21:52:19.6473760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6473822Z outputs = self.mobilebert( 2025-08-14T21:52:19.6474073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6474145Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6474398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6474465Z layer_outputs = layer_module( 2025-08-14T21:52:19.6474721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6474868Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6475127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.6475227Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.6475480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6475562Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6475566Z 2025-08-14T21:52:19.6475659Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6475845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6475906Z return mod(**inputs) 2025-08-14T21:52:19.6476162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6476232Z outputs = self.mobilebert( 2025-08-14T21:52:19.6476485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6476559Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6476813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6476878Z layer_outputs = layer_module( 2025-08-14T21:52:19.6477138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6477215Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6477496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6477617Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6477866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.6478005Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6478264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6478348Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6478352Z 2025-08-14T21:52:19.6478454Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6478649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6478719Z return mod(**inputs) 2025-08-14T21:52:19.6478994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6479060Z outputs = self.mobilebert( 2025-08-14T21:52:19.6479318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6479385Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6479636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6479709Z layer_outputs = layer_module( 2025-08-14T21:52:19.6479959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6480052Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6480303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6480407Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6480665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6480741Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6480746Z 2025-08-14T21:52:19.6480843Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6481023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6481084Z return mod(**inputs) 2025-08-14T21:52:19.6481345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6481409Z outputs = self.mobilebert( 2025-08-14T21:52:19.6481660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6481735Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6481984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6482056Z layer_outputs = layer_module( 2025-08-14T21:52:19.6482308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6482396Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6482664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6482766Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6483032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6483157Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6483162Z 2025-08-14T21:52:19.6483256Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6483448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6483509Z return mod(**inputs) 2025-08-14T21:52:19.6483788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6483862Z outputs = self.mobilebert( 2025-08-14T21:52:19.6484120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6484196Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6484466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6484536Z layer_outputs = layer_module( 2025-08-14T21:52:19.6484823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6484912Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6485179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6485299Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6485630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6485723Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6485727Z 2025-08-14T21:52:19.6485825Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6486028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6486096Z return mod(**inputs) 2025-08-14T21:52:19.6486373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6486450Z outputs = self.mobilebert( 2025-08-14T21:52:19.6486722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6486798Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6487079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6487157Z layer_outputs = layer_module( 2025-08-14T21:52:19.6487423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6487510Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6487771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6487895Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6488154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6488275Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6488536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6488621Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6488624Z 2025-08-14T21:52:19.6488727Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6488909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6488972Z return mod(**inputs) 2025-08-14T21:52:19.6489265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6489330Z outputs = self.mobilebert( 2025-08-14T21:52:19.6489593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6489676Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6489944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6490018Z layer_outputs = layer_module( 2025-08-14T21:52:19.6490285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6490378Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6490660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6490782Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6491051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6491128Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6491133Z 2025-08-14T21:52:19.6491229Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6491421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6491482Z return mod(**inputs) 2025-08-14T21:52:19.6491750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6491815Z outputs = self.mobilebert( 2025-08-14T21:52:19.6492075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6492154Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6492413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6492486Z layer_outputs = layer_module( 2025-08-14T21:52:19.6492746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6492832Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6493099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6493199Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6493465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6493577Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6493580Z 2025-08-14T21:52:19.6493678Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6493870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6493931Z return mod(**inputs) 2025-08-14T21:52:19.6494195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6494267Z outputs = self.mobilebert( 2025-08-14T21:52:19.6494530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6494606Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6494869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6494957Z layer_outputs = layer_module( 2025-08-14T21:52:19.6495234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6495320Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6495587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6495732Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6495990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6496075Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6496078Z 2025-08-14T21:52:19.6496173Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6496374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6496446Z return mod(**inputs) 2025-08-14T21:52:19.6496737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6496811Z outputs = self.mobilebert( 2025-08-14T21:52:19.6497079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6497149Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6497420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6497486Z layer_outputs = layer_module( 2025-08-14T21:52:19.6497749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6497843Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6498111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6498235Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6498503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6498618Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6498895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6498980Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6498983Z 2025-08-14T21:52:19.6499086Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6499279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6499339Z return mod(**inputs) 2025-08-14T21:52:19.6499611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6499675Z outputs = self.mobilebert( 2025-08-14T21:52:19.6499934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6500008Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6500264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6500334Z layer_outputs = layer_module( 2025-08-14T21:52:19.6500590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6500673Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6500937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6501060Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6501323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6501398Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6501420Z 2025-08-14T21:52:19.6501516Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6501705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6501765Z return mod(**inputs) 2025-08-14T21:52:19.6502030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6502093Z outputs = self.mobilebert( 2025-08-14T21:52:19.6502364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6502457Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6502715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6502781Z layer_outputs = layer_module( 2025-08-14T21:52:19.6503051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6503135Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6503401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6503499Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6503758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6503869Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6503872Z 2025-08-14T21:52:19.6503968Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6504158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6504220Z return mod(**inputs) 2025-08-14T21:52:19.6504494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6504564Z outputs = self.mobilebert( 2025-08-14T21:52:19.6504817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6504882Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6505142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6505206Z layer_outputs = layer_module( 2025-08-14T21:52:19.6505466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6505550Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6505801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6505922Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6506174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6506257Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6506261Z 2025-08-14T21:52:19.6506354Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6506534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6506623Z return mod(**inputs) 2025-08-14T21:52:19.6506889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6506956Z outputs = self.mobilebert( 2025-08-14T21:52:19.6507222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6507309Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6507578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6507644Z layer_outputs = layer_module( 2025-08-14T21:52:19.6507902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6508010Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6508298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6508424Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6508683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6508796Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6509062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6509147Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6509151Z 2025-08-14T21:52:19.6509255Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6509442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6509506Z return mod(**inputs) 2025-08-14T21:52:19.6509779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6509845Z outputs = self.mobilebert( 2025-08-14T21:52:19.6510105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6510182Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6510441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6510514Z layer_outputs = layer_module( 2025-08-14T21:52:19.6510773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6510895Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6511162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6511243Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6511246Z 2025-08-14T21:52:19.6511346Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6511531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6511594Z return mod(**inputs) 2025-08-14T21:52:19.6511864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6511929Z outputs = self.mobilebert( 2025-08-14T21:52:19.6512189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6512266Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6512526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6512621Z layer_outputs = layer_module( 2025-08-14T21:52:19.6512888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6513002Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6513326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6513441Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6513445Z 2025-08-14T21:52:19.6513546Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6513735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6513797Z return mod(**inputs) 2025-08-14T21:52:19.6514080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6514150Z outputs = self.mobilebert( 2025-08-14T21:52:19.6514423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6514502Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6514768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6514842Z layer_outputs = layer_module( 2025-08-14T21:52:19.6515094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6515237Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6515497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.6515585Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.6515589Z 2025-08-14T21:52:19.6515692Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6515877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6515938Z return mod(**inputs) 2025-08-14T21:52:19.6516207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6516272Z outputs = self.mobilebert( 2025-08-14T21:52:19.6516529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6516603Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6516859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6516930Z layer_outputs = layer_module( 2025-08-14T21:52:19.6517190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6517337Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6517600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.6517716Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.6517977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6518062Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6518065Z 2025-08-14T21:52:19.6518158Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6518350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6518431Z return mod(**inputs) 2025-08-14T21:52:19.6518694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6518766Z outputs = self.mobilebert( 2025-08-14T21:52:19.6519023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6519113Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6519372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6519438Z layer_outputs = layer_module( 2025-08-14T21:52:19.6519705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6519872Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6520161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6520277Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6520537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.6520624Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6520627Z 2025-08-14T21:52:19.6520722Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6520913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6520975Z return mod(**inputs) 2025-08-14T21:52:19.6521239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6521313Z outputs = self.mobilebert( 2025-08-14T21:52:19.6521582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6521650Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6521928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6521994Z layer_outputs = layer_module( 2025-08-14T21:52:19.6522268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6522414Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6522682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6522804Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6523077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.6523198Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6523466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6523554Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6523557Z 2025-08-14T21:52:19.6523664Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6523856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6523920Z return mod(**inputs) 2025-08-14T21:52:19.6524207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6524275Z outputs = self.mobilebert( 2025-08-14T21:52:19.6524566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6524636Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6524912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6525003Z layer_outputs = layer_module( 2025-08-14T21:52:19.6525264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6525484Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6525754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6525879Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6526175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6526267Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6526271Z 2025-08-14T21:52:19.6526385Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6526593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6526664Z return mod(**inputs) 2025-08-14T21:52:19.6526967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6527042Z outputs = self.mobilebert( 2025-08-14T21:52:19.6527333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6527419Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6527698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6527777Z layer_outputs = layer_module( 2025-08-14T21:52:19.6528037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6528185Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6528451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6528553Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6528818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.6528899Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.6529158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6529252Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6529255Z 2025-08-14T21:52:19.6529351Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6529535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6529604Z return mod(**inputs) 2025-08-14T21:52:19.6529867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6529941Z outputs = self.mobilebert( 2025-08-14T21:52:19.6530197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6530265Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6530538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6530627Z layer_outputs = layer_module( 2025-08-14T21:52:19.6530898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6530978Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6531240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6531334Z self_outputs = self.self( 2025-08-14T21:52:19.6531622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.6531689Z self.query(query_tensor) 2025-08-14T21:52:19.6531692Z 2025-08-14T21:52:19.6531794Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6531996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6532066Z return mod(**inputs) 2025-08-14T21:52:19.6532349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6532416Z outputs = self.mobilebert( 2025-08-14T21:52:19.6532687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6532757Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6533020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6533095Z layer_outputs = layer_module( 2025-08-14T21:52:19.6533359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6533447Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6533710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6533778Z self_outputs = self.self( 2025-08-14T21:52:19.6534046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.6534112Z self.key(key_tensor) 2025-08-14T21:52:19.6534116Z 2025-08-14T21:52:19.6534219Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6534403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6534466Z return mod(**inputs) 2025-08-14T21:52:19.6534740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6534806Z outputs = self.mobilebert( 2025-08-14T21:52:19.6535071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6535148Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6535412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6535488Z layer_outputs = layer_module( 2025-08-14T21:52:19.6535752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6535834Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6536113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6536181Z self_outputs = self.self( 2025-08-14T21:52:19.6536469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.6536538Z self.value(value_tensor) 2025-08-14T21:52:19.6536557Z 2025-08-14T21:52:19.6536638Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6536723Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6536823Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6537010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6537100Z return mod(**inputs) 2025-08-14T21:52:19.6537369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6537444Z outputs = self.mobilebert( 2025-08-14T21:52:19.6537872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6537956Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6538297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6538375Z layer_outputs = layer_module( 2025-08-14T21:52:19.6538690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6538789Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6539079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6539209Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6539476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.6539556Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6539559Z 2025-08-14T21:52:19.6539665Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6539855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6539927Z return mod(**inputs) 2025-08-14T21:52:19.6540196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6540262Z outputs = self.mobilebert( 2025-08-14T21:52:19.6540535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6540607Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6540873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6540950Z layer_outputs = layer_module( 2025-08-14T21:52:19.6541218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6541381Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6541653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.6541756Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.6542031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6542114Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6542117Z 2025-08-14T21:52:19.6542222Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6542413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6542476Z return mod(**inputs) 2025-08-14T21:52:19.6542753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6542848Z outputs = self.mobilebert( 2025-08-14T21:52:19.6543125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6543194Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6543463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6543576Z layer_outputs = layer_module( 2025-08-14T21:52:19.6543847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6543927Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6544204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6544335Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6544627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.6544749Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6545020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6545116Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6545120Z 2025-08-14T21:52:19.6545217Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6545413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6545475Z return mod(**inputs) 2025-08-14T21:52:19.6545745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6545823Z outputs = self.mobilebert( 2025-08-14T21:52:19.6546092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6546161Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6546438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6546508Z layer_outputs = layer_module( 2025-08-14T21:52:19.6546783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6546873Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6547135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6547249Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6547515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6547604Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6547607Z 2025-08-14T21:52:19.6547705Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6547895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6547967Z return mod(**inputs) 2025-08-14T21:52:19.6548247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6548313Z outputs = self.mobilebert( 2025-08-14T21:52:19.6548581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6548648Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6548917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6548999Z layer_outputs = layer_module( 2025-08-14T21:52:19.6549258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6549353Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6549607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6549733Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6549993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6550098Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6550102Z 2025-08-14T21:52:19.6550203Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6550404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6550466Z return mod(**inputs) 2025-08-14T21:52:19.6550753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6550819Z outputs = self.mobilebert( 2025-08-14T21:52:19.6551084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6551153Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6551415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6551485Z layer_outputs = layer_module( 2025-08-14T21:52:19.6551738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6551834Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6552098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6552215Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6552484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6552564Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6552567Z 2025-08-14T21:52:19.6552667Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6552849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6552910Z return mod(**inputs) 2025-08-14T21:52:19.6553190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6553259Z outputs = self.mobilebert( 2025-08-14T21:52:19.6553537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6553629Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6553892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6553968Z layer_outputs = layer_module( 2025-08-14T21:52:19.6554232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6554321Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6554591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6554712Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6555007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6555126Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6555379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6555488Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6555491Z 2025-08-14T21:52:19.6555585Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6555764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6555834Z return mod(**inputs) 2025-08-14T21:52:19.6556091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6556177Z outputs = self.mobilebert( 2025-08-14T21:52:19.6556447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6556515Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6556776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6556842Z layer_outputs = layer_module( 2025-08-14T21:52:19.6557100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6557185Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6557437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6557543Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6557800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6557877Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6557880Z 2025-08-14T21:52:19.6557980Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6558156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6558226Z return mod(**inputs) 2025-08-14T21:52:19.6558481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6558545Z outputs = self.mobilebert( 2025-08-14T21:52:19.6558804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6558869Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6559130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6559195Z layer_outputs = layer_module( 2025-08-14T21:52:19.6559453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6559545Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6559795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6559897Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6560157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6560260Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6560264Z 2025-08-14T21:52:19.6560366Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6560553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6560632Z return mod(**inputs) 2025-08-14T21:52:19.6560907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6560975Z outputs = self.mobilebert( 2025-08-14T21:52:19.6561240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6561326Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6561585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6561659Z layer_outputs = layer_module( 2025-08-14T21:52:19.6561920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6562024Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6562319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6562440Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6562717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6562798Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6562801Z 2025-08-14T21:52:19.6562898Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6563096Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6563159Z return mod(**inputs) 2025-08-14T21:52:19.6563445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6563516Z outputs = self.mobilebert( 2025-08-14T21:52:19.6563792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6563870Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6564144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6564215Z layer_outputs = layer_module( 2025-08-14T21:52:19.6564497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6564588Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6564870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6564993Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6565269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6565469Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6565755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6565852Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6565856Z 2025-08-14T21:52:19.6565957Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6566149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6566223Z return mod(**inputs) 2025-08-14T21:52:19.6566502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6566573Z outputs = self.mobilebert( 2025-08-14T21:52:19.6566858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6566958Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6567228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6567315Z layer_outputs = layer_module( 2025-08-14T21:52:19.6567573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6567670Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6567932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6568039Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6568642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6568745Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6568749Z 2025-08-14T21:52:19.6568855Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6569041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6569114Z return mod(**inputs) 2025-08-14T21:52:19.6569379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6569448Z outputs = self.mobilebert( 2025-08-14T21:52:19.6569715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6569785Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6570047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6570124Z layer_outputs = layer_module( 2025-08-14T21:52:19.6570389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6570484Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6570749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6570854Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6571125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6571229Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6571232Z 2025-08-14T21:52:19.6571335Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6571524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6571588Z return mod(**inputs) 2025-08-14T21:52:19.6571861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6571927Z outputs = self.mobilebert( 2025-08-14T21:52:19.6572189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6572267Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6572534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6572610Z layer_outputs = layer_module( 2025-08-14T21:52:19.6572875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6572974Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6573261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6573385Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6573648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6573746Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6573749Z 2025-08-14T21:52:19.6573849Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6574041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6574105Z return mod(**inputs) 2025-08-14T21:52:19.6574371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6574462Z outputs = self.mobilebert( 2025-08-14T21:52:19.6574744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6574820Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6575082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6575153Z layer_outputs = layer_module( 2025-08-14T21:52:19.6575420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6575505Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6575771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6575886Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6576148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6576269Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6576530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6576617Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6576626Z 2025-08-14T21:52:19.6576721Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6576905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6576974Z return mod(**inputs) 2025-08-14T21:52:19.6577241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6577306Z outputs = self.mobilebert( 2025-08-14T21:52:19.6577576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6577647Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6577915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6577983Z layer_outputs = layer_module( 2025-08-14T21:52:19.6578241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6578360Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6578619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6578698Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6578709Z 2025-08-14T21:52:19.6578804Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6579015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6579086Z return mod(**inputs) 2025-08-14T21:52:19.6579352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6579418Z outputs = self.mobilebert( 2025-08-14T21:52:19.6579708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6579776Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6580040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6580108Z layer_outputs = layer_module( 2025-08-14T21:52:19.6580381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6580500Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6580779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6580882Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6580893Z 2025-08-14T21:52:19.6580988Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6581174Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6581244Z return mod(**inputs) 2025-08-14T21:52:19.6581510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6581574Z outputs = self.mobilebert( 2025-08-14T21:52:19.6581844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6581913Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6582183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6582250Z layer_outputs = layer_module( 2025-08-14T21:52:19.6582510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6582667Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6582929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.6583023Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.6583026Z 2025-08-14T21:52:19.6583120Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6583303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6583375Z return mod(**inputs) 2025-08-14T21:52:19.6583650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6583716Z outputs = self.mobilebert( 2025-08-14T21:52:19.6583992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6584062Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6584335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6584403Z layer_outputs = layer_module( 2025-08-14T21:52:19.6584670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6584829Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6585115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.6585238Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.6585506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6585609Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6585612Z 2025-08-14T21:52:19.6585716Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6585957Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6586019Z return mod(**inputs) 2025-08-14T21:52:19.6586349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6586418Z outputs = self.mobilebert( 2025-08-14T21:52:19.6586720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6586791Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6587104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6587182Z layer_outputs = layer_module( 2025-08-14T21:52:19.6587452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6587610Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6587879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6587999Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6588275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.6588356Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6588360Z 2025-08-14T21:52:19.6588465Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6588659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6588721Z return mod(**inputs) 2025-08-14T21:52:19.6588999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6589066Z outputs = self.mobilebert( 2025-08-14T21:52:19.6589337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6589416Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6589683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6589759Z layer_outputs = layer_module( 2025-08-14T21:52:19.6590029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6590179Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6590451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6590567Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6590841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.6590956Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6591249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6591344Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6591348Z 2025-08-14T21:52:19.6591446Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6591636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6591725Z return mod(**inputs) 2025-08-14T21:52:19.6591999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6592073Z outputs = self.mobilebert( 2025-08-14T21:52:19.6592338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6592408Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6592694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6592780Z layer_outputs = layer_module( 2025-08-14T21:52:19.6593060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6593214Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6593488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6593600Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6593868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6593948Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6593959Z 2025-08-14T21:52:19.6594056Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6594248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6594318Z return mod(**inputs) 2025-08-14T21:52:19.6594588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6594656Z outputs = self.mobilebert( 2025-08-14T21:52:19.6594929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6595000Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6595276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6595344Z layer_outputs = layer_module( 2025-08-14T21:52:19.6595614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6595774Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6596046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6596149Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6596425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.6596507Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.6596782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6596868Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6596871Z 2025-08-14T21:52:19.6596967Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6597184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6597248Z return mod(**inputs) 2025-08-14T21:52:19.6597522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6597589Z outputs = self.mobilebert( 2025-08-14T21:52:19.6597874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6597952Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6598224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6598290Z layer_outputs = layer_module( 2025-08-14T21:52:19.6598578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6598664Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6598947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6599017Z self_outputs = self.self( 2025-08-14T21:52:19.6599283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.6599362Z self.query(query_tensor) 2025-08-14T21:52:19.6599365Z 2025-08-14T21:52:19.6599460Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6599657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6599722Z return mod(**inputs) 2025-08-14T21:52:19.6599998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6600074Z outputs = self.mobilebert( 2025-08-14T21:52:19.6600353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6600424Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6600704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6600776Z layer_outputs = layer_module( 2025-08-14T21:52:19.6601059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6601140Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6601403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6601480Z self_outputs = self.self( 2025-08-14T21:52:19.6601746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.6601821Z self.key(key_tensor) 2025-08-14T21:52:19.6601825Z 2025-08-14T21:52:19.6601920Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6602110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6602182Z return mod(**inputs) 2025-08-14T21:52:19.6602451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6602516Z outputs = self.mobilebert( 2025-08-14T21:52:19.6602793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6602865Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6603146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6603233Z layer_outputs = layer_module( 2025-08-14T21:52:19.6603511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6603601Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6603878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6603972Z self_outputs = self.self( 2025-08-14T21:52:19.6604255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.6604326Z self.value(value_tensor) 2025-08-14T21:52:19.6604329Z 2025-08-14T21:52:19.6604417Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6604497Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6604617Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6604822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6604901Z return mod(**inputs) 2025-08-14T21:52:19.6605186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6605254Z outputs = self.mobilebert( 2025-08-14T21:52:19.6605598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6605686Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6605965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6606038Z layer_outputs = layer_module( 2025-08-14T21:52:19.6606331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6606421Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6606723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6606852Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6607133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.6607224Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6607227Z 2025-08-14T21:52:19.6607326Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6607524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6607587Z return mod(**inputs) 2025-08-14T21:52:19.6607871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6607948Z outputs = self.mobilebert( 2025-08-14T21:52:19.6608207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6608278Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6608548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6608619Z layer_outputs = layer_module( 2025-08-14T21:52:19.6608885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6609034Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6609295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.6609405Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.6609720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6609804Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6609807Z 2025-08-14T21:52:19.6609903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6610104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6610174Z return mod(**inputs) 2025-08-14T21:52:19.6610437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6610503Z outputs = self.mobilebert( 2025-08-14T21:52:19.6610766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6610856Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6611143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6611210Z layer_outputs = layer_module( 2025-08-14T21:52:19.6611471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6611557Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6611818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6611938Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6612202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.6612320Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6612599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6612690Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6612694Z 2025-08-14T21:52:19.6612800Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6612994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6613059Z return mod(**inputs) 2025-08-14T21:52:19.6613344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6613413Z outputs = self.mobilebert( 2025-08-14T21:52:19.6613689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6613766Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6614041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6614192Z layer_outputs = layer_module( 2025-08-14T21:52:19.6614491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6614719Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6615047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6615182Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6615468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6615597Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6615601Z 2025-08-14T21:52:19.6615708Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6615996Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6616091Z return mod(**inputs) 2025-08-14T21:52:19.6616376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6616495Z outputs = self.mobilebert( 2025-08-14T21:52:19.6616798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6616877Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6617222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6617311Z layer_outputs = layer_module( 2025-08-14T21:52:19.6617635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6617748Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6618045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6618250Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6618547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6618699Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6618703Z 2025-08-14T21:52:19.6618823Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6619028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6619134Z return mod(**inputs) 2025-08-14T21:52:19.6619440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6619565Z outputs = self.mobilebert( 2025-08-14T21:52:19.6619848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6619937Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6620254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6620335Z layer_outputs = layer_module( 2025-08-14T21:52:19.6620672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6620780Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6621058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6621233Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6621527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6621618Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6621665Z 2025-08-14T21:52:19.6622572Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6622793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6622909Z return mod(**inputs) 2025-08-14T21:52:19.6623214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6623305Z outputs = self.mobilebert( 2025-08-14T21:52:19.6623634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6623744Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6624088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6624190Z layer_outputs = layer_module( 2025-08-14T21:52:19.6624488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6624634Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6624950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6625135Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6625436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6625581Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6625924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6626044Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6626048Z 2025-08-14T21:52:19.6626241Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6626462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6626551Z return mod(**inputs) 2025-08-14T21:52:19.6626904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6626998Z outputs = self.mobilebert( 2025-08-14T21:52:19.6627339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6627448Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6627745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6627868Z layer_outputs = layer_module( 2025-08-14T21:52:19.6628166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6628280Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6628620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6628766Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6629090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6629199Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6629204Z 2025-08-14T21:52:19.6629334Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6629587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6629717Z return mod(**inputs) 2025-08-14T21:52:19.6630069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6630161Z outputs = self.mobilebert( 2025-08-14T21:52:19.6630457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6630579Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6630870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6631031Z layer_outputs = layer_module( 2025-08-14T21:52:19.6631327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6631458Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6631825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6631973Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6632349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6632513Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6632517Z 2025-08-14T21:52:19.6632641Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6632893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6632994Z return mod(**inputs) 2025-08-14T21:52:19.6633334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6633490Z outputs = self.mobilebert( 2025-08-14T21:52:19.6633832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6633958Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6634284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6634382Z layer_outputs = layer_module( 2025-08-14T21:52:19.6634695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6634831Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6635170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6635323Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6635626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6635764Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6635768Z 2025-08-14T21:52:19.6635879Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6636163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6636283Z return mod(**inputs) 2025-08-14T21:52:19.6636597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6636718Z outputs = self.mobilebert( 2025-08-14T21:52:19.6637016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6637143Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6637454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6637552Z layer_outputs = layer_module( 2025-08-14T21:52:19.6638023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6638149Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6638446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6638630Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6638941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6639117Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6639418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6639575Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6639581Z 2025-08-14T21:52:19.6639722Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6639968Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6640125Z return mod(**inputs) 2025-08-14T21:52:19.6640423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6640517Z outputs = self.mobilebert( 2025-08-14T21:52:19.6640848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6640951Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6641378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6641496Z layer_outputs = layer_module( 2025-08-14T21:52:19.6641798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6641944Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6642251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6642416Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6642734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6642845Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6642849Z 2025-08-14T21:52:19.6643013Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6643243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6643348Z return mod(**inputs) 2025-08-14T21:52:19.6643701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6643809Z outputs = self.mobilebert( 2025-08-14T21:52:19.6644157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6644254Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6644576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6644689Z layer_outputs = layer_module( 2025-08-14T21:52:19.6645029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6645188Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6645572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6645717Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6646070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6646228Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6646232Z 2025-08-14T21:52:19.6646433Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6646671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6646767Z return mod(**inputs) 2025-08-14T21:52:19.6647132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6647259Z outputs = self.mobilebert( 2025-08-14T21:52:19.6647633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6647754Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6648069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6648222Z layer_outputs = layer_module( 2025-08-14T21:52:19.6648534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6648654Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6649006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6649187Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6649555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6649667Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6649671Z 2025-08-14T21:52:19.6649804Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6650051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6650168Z return mod(**inputs) 2025-08-14T21:52:19.6650534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6650631Z outputs = self.mobilebert( 2025-08-14T21:52:19.6650974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6651111Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6651419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6676780Z layer_outputs = layer_module( 2025-08-14T21:52:19.6677241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6677371Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6677666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6677800Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6678090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6678211Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6678500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6678604Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6678610Z 2025-08-14T21:52:19.6678723Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6678942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6679018Z return mod(**inputs) 2025-08-14T21:52:19.6679303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6679381Z outputs = self.mobilebert( 2025-08-14T21:52:19.6679638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6679719Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6679974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6680115Z layer_outputs = layer_module( 2025-08-14T21:52:19.6680382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6680499Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6680799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6680880Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6680885Z 2025-08-14T21:52:19.6680985Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6681182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6681247Z return mod(**inputs) 2025-08-14T21:52:19.6681548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6681644Z outputs = self.mobilebert( 2025-08-14T21:52:19.6681911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6681992Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6682255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6682324Z layer_outputs = layer_module( 2025-08-14T21:52:19.6682601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6682719Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6683007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6683123Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6683127Z 2025-08-14T21:52:19.6683233Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6683447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6683514Z return mod(**inputs) 2025-08-14T21:52:19.6683801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6683873Z outputs = self.mobilebert( 2025-08-14T21:52:19.6684148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6684231Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6684509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6684583Z layer_outputs = layer_module( 2025-08-14T21:52:19.6684870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6685031Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6685313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.6685512Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.6685519Z 2025-08-14T21:52:19.6685631Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6685866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6685937Z return mod(**inputs) 2025-08-14T21:52:19.6686262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6686371Z outputs = self.mobilebert( 2025-08-14T21:52:19.6686655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6686737Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6687024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6687109Z layer_outputs = layer_module( 2025-08-14T21:52:19.6687378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6687540Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6687799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.6687937Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.6688210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6688297Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6688307Z 2025-08-14T21:52:19.6688403Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6688588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6688656Z return mod(**inputs) 2025-08-14T21:52:19.6688921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6688990Z outputs = self.mobilebert( 2025-08-14T21:52:19.6689265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6689338Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6689619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6689687Z layer_outputs = layer_module( 2025-08-14T21:52:19.6689955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6690114Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6690384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6690502Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6690785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.6690862Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6690867Z 2025-08-14T21:52:19.6690967Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6691151Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6691214Z return mod(**inputs) 2025-08-14T21:52:19.6691486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6691553Z outputs = self.mobilebert( 2025-08-14T21:52:19.6691821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6691888Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6692147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6692221Z layer_outputs = layer_module( 2025-08-14T21:52:19.6692481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6692658Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6692927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6693061Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6693341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.6693454Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6693718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6693809Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6693829Z 2025-08-14T21:52:19.6693926Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6694132Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6694197Z return mod(**inputs) 2025-08-14T21:52:19.6694468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6694541Z outputs = self.mobilebert( 2025-08-14T21:52:19.6694794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6694868Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6695121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6695187Z layer_outputs = layer_module( 2025-08-14T21:52:19.6695446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6695600Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6695858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6695972Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6696237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6696326Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6696330Z 2025-08-14T21:52:19.6696428Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6696619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6696692Z return mod(**inputs) 2025-08-14T21:52:19.6696965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6697043Z outputs = self.mobilebert( 2025-08-14T21:52:19.6697309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6697379Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6697656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6697724Z layer_outputs = layer_module( 2025-08-14T21:52:19.6697997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6698152Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6698414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6698541Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6698799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.6698884Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.6699179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6699263Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6699267Z 2025-08-14T21:52:19.6699373Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6699557Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6699619Z return mod(**inputs) 2025-08-14T21:52:19.6699902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6699973Z outputs = self.mobilebert( 2025-08-14T21:52:19.6700257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6700326Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6700586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6700661Z layer_outputs = layer_module( 2025-08-14T21:52:19.6700916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6700997Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6701264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6701332Z self_outputs = self.self( 2025-08-14T21:52:19.6701599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.6701666Z self.query(query_tensor) 2025-08-14T21:52:19.6701669Z 2025-08-14T21:52:19.6701765Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6701958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6702021Z return mod(**inputs) 2025-08-14T21:52:19.6702282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6702354Z outputs = self.mobilebert( 2025-08-14T21:52:19.6702611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6702687Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6702947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6703014Z layer_outputs = layer_module( 2025-08-14T21:52:19.6703281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6703363Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6703627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6703692Z self_outputs = self.self( 2025-08-14T21:52:19.6703948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.6704020Z self.key(key_tensor) 2025-08-14T21:52:19.6704023Z 2025-08-14T21:52:19.6704119Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6704323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6704394Z return mod(**inputs) 2025-08-14T21:52:19.6704673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6704752Z outputs = self.mobilebert( 2025-08-14T21:52:19.6705039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6705110Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6705390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6705460Z layer_outputs = layer_module( 2025-08-14T21:52:19.6705766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6705852Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6706134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6706212Z self_outputs = self.self( 2025-08-14T21:52:19.6706482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.6706550Z self.value(value_tensor) 2025-08-14T21:52:19.6706560Z 2025-08-14T21:52:19.6706641Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6706715Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6706819Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6707018Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6707079Z return mod(**inputs) 2025-08-14T21:52:19.6707353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6707419Z outputs = self.mobilebert( 2025-08-14T21:52:19.6707685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6707759Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6708021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6708093Z layer_outputs = layer_module( 2025-08-14T21:52:19.6708354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6708433Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6708705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6708821Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6709093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.6709172Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6709175Z 2025-08-14T21:52:19.6709271Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6709467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6709530Z return mod(**inputs) 2025-08-14T21:52:19.6709793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6709865Z outputs = self.mobilebert( 2025-08-14T21:52:19.6710127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6710203Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6710478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6710544Z layer_outputs = layer_module( 2025-08-14T21:52:19.6710810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6710979Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6711248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.6711352Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.6711616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6711721Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6711726Z 2025-08-14T21:52:19.6711825Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6712042Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6712107Z return mod(**inputs) 2025-08-14T21:52:19.6712377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6712452Z outputs = self.mobilebert( 2025-08-14T21:52:19.6712716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6712787Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6713068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6713138Z layer_outputs = layer_module( 2025-08-14T21:52:19.6713419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6713515Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6713771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6713894Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6714156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.6714284Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6714548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6714636Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6714641Z 2025-08-14T21:52:19.6714745Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6714936Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6715001Z return mod(**inputs) 2025-08-14T21:52:19.6715277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6715347Z outputs = self.mobilebert( 2025-08-14T21:52:19.6715615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6715685Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6715948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6716023Z layer_outputs = layer_module( 2025-08-14T21:52:19.6716290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6716409Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6716678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6716785Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6717073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6717153Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6717157Z 2025-08-14T21:52:19.6717253Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6717447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6717512Z return mod(**inputs) 2025-08-14T21:52:19.6717798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6717868Z outputs = self.mobilebert( 2025-08-14T21:52:19.6718151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6718232Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6718501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6718575Z layer_outputs = layer_module( 2025-08-14T21:52:19.6718842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6718931Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6719206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6719311Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6719581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6719697Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6719701Z 2025-08-14T21:52:19.6719798Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6719997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6720060Z return mod(**inputs) 2025-08-14T21:52:19.6720333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6720408Z outputs = self.mobilebert( 2025-08-14T21:52:19.6720679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6720757Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6721027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6721094Z layer_outputs = layer_module( 2025-08-14T21:52:19.6721370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6721462Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6721728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6721858Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6722123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6722211Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6722243Z 2025-08-14T21:52:19.6722342Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6722533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6722603Z return mod(**inputs) 2025-08-14T21:52:19.6722870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6722964Z outputs = self.mobilebert( 2025-08-14T21:52:19.6723231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6723298Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6723567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6723634Z layer_outputs = layer_module( 2025-08-14T21:52:19.6723914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6724030Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6724299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6724423Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6724696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6724814Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6725103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6725192Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6725196Z 2025-08-14T21:52:19.6725304Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6725583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6725662Z return mod(**inputs) 2025-08-14T21:52:19.6725976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6726054Z outputs = self.mobilebert( 2025-08-14T21:52:19.6726359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6726439Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6726741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6726834Z layer_outputs = layer_module( 2025-08-14T21:52:19.6727128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6727227Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6727525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6727641Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6727945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6728034Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6728038Z 2025-08-14T21:52:19.6728145Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6728365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6728431Z return mod(**inputs) 2025-08-14T21:52:19.6728717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6728812Z outputs = self.mobilebert( 2025-08-14T21:52:19.6729087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6729165Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6729443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6729531Z layer_outputs = layer_module( 2025-08-14T21:52:19.6729819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6729910Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6730198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6730321Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6730615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6730738Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6730742Z 2025-08-14T21:52:19.6730849Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6731064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6731133Z return mod(**inputs) 2025-08-14T21:52:19.6731429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6731518Z outputs = self.mobilebert( 2025-08-14T21:52:19.6731794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6731867Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6732157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6732230Z layer_outputs = layer_module( 2025-08-14T21:52:19.6732529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6732627Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6732919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6733055Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6733346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6733443Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6733448Z 2025-08-14T21:52:19.6733553Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6733774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6733850Z return mod(**inputs) 2025-08-14T21:52:19.6734147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6734221Z outputs = self.mobilebert( 2025-08-14T21:52:19.6734528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6734599Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6734880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6734950Z layer_outputs = layer_module( 2025-08-14T21:52:19.6735226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6735342Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6735620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6735749Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6736039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6736156Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6736442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6736531Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6736535Z 2025-08-14T21:52:19.6736659Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6736875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6736953Z return mod(**inputs) 2025-08-14T21:52:19.6737232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6737301Z outputs = self.mobilebert( 2025-08-14T21:52:19.6737570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6737774Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6738097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6738174Z layer_outputs = layer_module( 2025-08-14T21:52:19.6738455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6738548Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6738832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6738942Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6739232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6739317Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6739321Z 2025-08-14T21:52:19.6739434Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6739630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6739692Z return mod(**inputs) 2025-08-14T21:52:19.6739957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6740035Z outputs = self.mobilebert( 2025-08-14T21:52:19.6740297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6740372Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6740633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6740702Z layer_outputs = layer_module( 2025-08-14T21:52:19.6740972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6741058Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6741325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6741428Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6741747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6741858Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6741862Z 2025-08-14T21:52:19.6741959Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6742171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6742241Z return mod(**inputs) 2025-08-14T21:52:19.6742508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6742583Z outputs = self.mobilebert( 2025-08-14T21:52:19.6742851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6742946Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6743265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6743333Z layer_outputs = layer_module( 2025-08-14T21:52:19.6743604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6743693Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6743953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6744075Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6744333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6744415Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6744427Z 2025-08-14T21:52:19.6744523Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6744709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6744779Z return mod(**inputs) 2025-08-14T21:52:19.6745046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6745114Z outputs = self.mobilebert( 2025-08-14T21:52:19.6745383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6745450Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6745718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6745785Z layer_outputs = layer_module( 2025-08-14T21:52:19.6746046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6746142Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6746402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6746516Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6746785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6746897Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6747163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6747247Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6747251Z 2025-08-14T21:52:19.6747349Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6747563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6747625Z return mod(**inputs) 2025-08-14T21:52:19.6747892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6747977Z outputs = self.mobilebert( 2025-08-14T21:52:19.6748234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6748307Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6748563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6748628Z layer_outputs = layer_module( 2025-08-14T21:52:19.6748908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6749023Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6749309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6749389Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6749394Z 2025-08-14T21:52:19.6749488Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6749681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6749742Z return mod(**inputs) 2025-08-14T21:52:19.6750016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6750089Z outputs = self.mobilebert( 2025-08-14T21:52:19.6750363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6750438Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6750696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6750768Z layer_outputs = layer_module( 2025-08-14T21:52:19.6751033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6751146Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6751425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6751524Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6751528Z 2025-08-14T21:52:19.6751628Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6751810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6751873Z return mod(**inputs) 2025-08-14T21:52:19.6752145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6752211Z outputs = self.mobilebert( 2025-08-14T21:52:19.6752473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6752548Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6752812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6752885Z layer_outputs = layer_module( 2025-08-14T21:52:19.6753150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6753297Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6753583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.6753671Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.6753674Z 2025-08-14T21:52:19.6753778Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6753978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6754041Z return mod(**inputs) 2025-08-14T21:52:19.6754313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6754378Z outputs = self.mobilebert( 2025-08-14T21:52:19.6754635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6754726Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6755008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6755084Z layer_outputs = layer_module( 2025-08-14T21:52:19.6755342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6755491Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6755763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.6755874Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.6756134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6756220Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6756225Z 2025-08-14T21:52:19.6756318Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6756506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6756568Z return mod(**inputs) 2025-08-14T21:52:19.6756824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6756896Z outputs = self.mobilebert( 2025-08-14T21:52:19.6757146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6757221Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6757475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6757540Z layer_outputs = layer_module( 2025-08-14T21:52:19.6757953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6758109Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6758378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6758492Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6758746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.6758831Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6758835Z 2025-08-14T21:52:19.6758930Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6759119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6759185Z return mod(**inputs) 2025-08-14T21:52:19.6759466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6759543Z outputs = self.mobilebert( 2025-08-14T21:52:19.6759797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6759919Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6760186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6760251Z layer_outputs = layer_module( 2025-08-14T21:52:19.6760514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6760656Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6760926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6761060Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6761314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.6761432Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6761684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6761768Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6761771Z 2025-08-14T21:52:19.6761870Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6762051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6762114Z return mod(**inputs) 2025-08-14T21:52:19.6762380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6762446Z outputs = self.mobilebert( 2025-08-14T21:52:19.6762711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6762782Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6763041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6763113Z layer_outputs = layer_module( 2025-08-14T21:52:19.6763373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6763530Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6763791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6763896Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6764161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6764237Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6764243Z 2025-08-14T21:52:19.6764343Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6764527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6764590Z return mod(**inputs) 2025-08-14T21:52:19.6764857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6764924Z outputs = self.mobilebert( 2025-08-14T21:52:19.6765184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6765277Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6765587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6765676Z layer_outputs = layer_module( 2025-08-14T21:52:19.6766005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6766176Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6766484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6766596Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6766918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.6767011Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.6767325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6767430Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6767436Z 2025-08-14T21:52:19.6767548Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6767732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6767803Z return mod(**inputs) 2025-08-14T21:52:19.6768066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6768140Z outputs = self.mobilebert( 2025-08-14T21:52:19.6768400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6768470Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6768738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6768805Z layer_outputs = layer_module( 2025-08-14T21:52:19.6769071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6769151Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6769409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6769481Z self_outputs = self.self( 2025-08-14T21:52:19.6769737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.6769807Z self.query(query_tensor) 2025-08-14T21:52:19.6769811Z 2025-08-14T21:52:19.6769916Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6770103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6770172Z return mod(**inputs) 2025-08-14T21:52:19.6770434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6770502Z outputs = self.mobilebert( 2025-08-14T21:52:19.6770766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6770834Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6771096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6771168Z layer_outputs = layer_module( 2025-08-14T21:52:19.6771431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6771538Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6771795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6771862Z self_outputs = self.self( 2025-08-14T21:52:19.6772147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.6772209Z self.key(key_tensor) 2025-08-14T21:52:19.6772213Z 2025-08-14T21:52:19.6772315Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6772499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6772560Z return mod(**inputs) 2025-08-14T21:52:19.6772847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6772915Z outputs = self.mobilebert( 2025-08-14T21:52:19.6773191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6773269Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6773529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6773604Z layer_outputs = layer_module( 2025-08-14T21:52:19.6773869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6773948Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6774215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6774282Z self_outputs = self.self( 2025-08-14T21:52:19.6774553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.6774619Z self.value(value_tensor) 2025-08-14T21:52:19.6774622Z 2025-08-14T21:52:19.6774698Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6774777Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6774876Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6775062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6775131Z return mod(**inputs) 2025-08-14T21:52:19.6775392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6775464Z outputs = self.mobilebert( 2025-08-14T21:52:19.6775725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6775793Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6776061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6776127Z layer_outputs = layer_module( 2025-08-14T21:52:19.6776386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6776471Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6776730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6776852Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6777112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.6777191Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6777214Z 2025-08-14T21:52:19.6777321Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6777503Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6777572Z return mod(**inputs) 2025-08-14T21:52:19.6777835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6777917Z outputs = self.mobilebert( 2025-08-14T21:52:19.6778187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6778254Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6778518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6778608Z layer_outputs = layer_module( 2025-08-14T21:52:19.6778892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6779047Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6779303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.6779403Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.6779665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6779740Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6779744Z 2025-08-14T21:52:19.6779844Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6780024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6780087Z return mod(**inputs) 2025-08-14T21:52:19.6780349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6780414Z outputs = self.mobilebert( 2025-08-14T21:52:19.6780671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6780739Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6780990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6781061Z layer_outputs = layer_module( 2025-08-14T21:52:19.6781309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6781384Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6781642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6781755Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6782011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.6782124Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6782381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6782473Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6782476Z 2025-08-14T21:52:19.6782570Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6782768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6782828Z return mod(**inputs) 2025-08-14T21:52:19.6783082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6783169Z outputs = self.mobilebert( 2025-08-14T21:52:19.6783423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6783489Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6783762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6783826Z layer_outputs = layer_module( 2025-08-14T21:52:19.6784082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6784169Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6784438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6784549Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6784818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6784904Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6784908Z 2025-08-14T21:52:19.6785002Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6785185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6785252Z return mod(**inputs) 2025-08-14T21:52:19.6785510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6785574Z outputs = self.mobilebert( 2025-08-14T21:52:19.6785836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6785904Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6786164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6786229Z layer_outputs = layer_module( 2025-08-14T21:52:19.6786481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6786575Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6786829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6786936Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6787188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6787291Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6787295Z 2025-08-14T21:52:19.6787397Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6787578Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6787639Z return mod(**inputs) 2025-08-14T21:52:19.6787901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6787967Z outputs = self.mobilebert( 2025-08-14T21:52:19.6788229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6788294Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6788548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6788626Z layer_outputs = layer_module( 2025-08-14T21:52:19.6788899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6788990Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6789243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6789384Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6789645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6789722Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6789725Z 2025-08-14T21:52:19.6789826Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6790012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6790090Z return mod(**inputs) 2025-08-14T21:52:19.6790370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6790438Z outputs = self.mobilebert( 2025-08-14T21:52:19.6790702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6790779Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6791046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6791120Z layer_outputs = layer_module( 2025-08-14T21:52:19.6791385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6791470Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6791744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6791865Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6792126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6792247Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6792512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6792605Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6792609Z 2025-08-14T21:52:19.6792706Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6792892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6792964Z return mod(**inputs) 2025-08-14T21:52:19.6793229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6793304Z outputs = self.mobilebert( 2025-08-14T21:52:19.6793566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6793634Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6793905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6793970Z layer_outputs = layer_module( 2025-08-14T21:52:19.6794243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6794332Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6794595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6794723Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6794993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6795069Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6795072Z 2025-08-14T21:52:19.6795189Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6795369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6795438Z return mod(**inputs) 2025-08-14T21:52:19.6795692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6795755Z outputs = self.mobilebert( 2025-08-14T21:52:19.6796027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6796094Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6796367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6796433Z layer_outputs = layer_module( 2025-08-14T21:52:19.6796685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6796776Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6797027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6797126Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6797382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6797482Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6797487Z 2025-08-14T21:52:19.6797588Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6797768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6797826Z return mod(**inputs) 2025-08-14T21:52:19.6798088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6798153Z outputs = self.mobilebert( 2025-08-14T21:52:19.6798411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6798480Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6798736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6798812Z layer_outputs = layer_module( 2025-08-14T21:52:19.6799070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6799157Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6799420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6799536Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6799797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6799875Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6799878Z 2025-08-14T21:52:19.6799974Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6800168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6800232Z return mod(**inputs) 2025-08-14T21:52:19.6800546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6800612Z outputs = self.mobilebert( 2025-08-14T21:52:19.6800871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6800987Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6801252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6801326Z layer_outputs = layer_module( 2025-08-14T21:52:19.6801582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6801677Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6801950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6802085Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6802360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6802474Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6802754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6802841Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6802845Z 2025-08-14T21:52:19.6802942Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6803140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6803202Z return mod(**inputs) 2025-08-14T21:52:19.6803476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6803554Z outputs = self.mobilebert( 2025-08-14T21:52:19.6803828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6803903Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6804176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6804243Z layer_outputs = layer_module( 2025-08-14T21:52:19.6804523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6804609Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6804890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6804995Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6805269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6805356Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6805360Z 2025-08-14T21:52:19.6805541Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6805780Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6805851Z return mod(**inputs) 2025-08-14T21:52:19.6806164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6806248Z outputs = self.mobilebert( 2025-08-14T21:52:19.6806561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6806664Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6806964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6807034Z layer_outputs = layer_module( 2025-08-14T21:52:19.6807315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6807426Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6807703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6807817Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6808104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6808230Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6808236Z 2025-08-14T21:52:19.6808348Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6808535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6808603Z return mod(**inputs) 2025-08-14T21:52:19.6808866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6808932Z outputs = self.mobilebert( 2025-08-14T21:52:19.6809201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6809267Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6809536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6809603Z layer_outputs = layer_module( 2025-08-14T21:52:19.6809871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6809964Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6810228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6810352Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6810618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6810696Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6810699Z 2025-08-14T21:52:19.6810799Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6810984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6811046Z return mod(**inputs) 2025-08-14T21:52:19.6811327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6811392Z outputs = self.mobilebert( 2025-08-14T21:52:19.6811670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6811740Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6812014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6812091Z layer_outputs = layer_module( 2025-08-14T21:52:19.6812373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6812470Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6812753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6812912Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6813216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6813341Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6813662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6813763Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6813767Z 2025-08-14T21:52:19.6813875Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6814093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6814157Z return mod(**inputs) 2025-08-14T21:52:19.6814453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6814551Z outputs = self.mobilebert( 2025-08-14T21:52:19.6814838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6814916Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6815186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6815255Z layer_outputs = layer_module( 2025-08-14T21:52:19.6815530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6815645Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6815921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6816013Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6816016Z 2025-08-14T21:52:19.6816116Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6816318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6816393Z return mod(**inputs) 2025-08-14T21:52:19.6816666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6816741Z outputs = self.mobilebert( 2025-08-14T21:52:19.6817008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6817084Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6817355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6817425Z layer_outputs = layer_module( 2025-08-14T21:52:19.6817701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6817812Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6818083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6818199Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6818202Z 2025-08-14T21:52:19.6818308Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6818497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6818558Z return mod(**inputs) 2025-08-14T21:52:19.6818823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6818914Z outputs = self.mobilebert( 2025-08-14T21:52:19.6819174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6819248Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6819510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6819594Z layer_outputs = layer_module( 2025-08-14T21:52:19.6819858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6820006Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6820263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.6820373Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.6820379Z 2025-08-14T21:52:19.6820476Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6820691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6820757Z return mod(**inputs) 2025-08-14T21:52:19.6821028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6821108Z outputs = self.mobilebert( 2025-08-14T21:52:19.6821376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6821452Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6821721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6821788Z layer_outputs = layer_module( 2025-08-14T21:52:19.6822068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6822220Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6822500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.6822618Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.6822896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6822987Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6822990Z 2025-08-14T21:52:19.6823085Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6823268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6823338Z return mod(**inputs) 2025-08-14T21:52:19.6823607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6823677Z outputs = self.mobilebert( 2025-08-14T21:52:19.6823938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6824005Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6824273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6826731Z layer_outputs = layer_module( 2025-08-14T21:52:19.6827006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6827165Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6827426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6827575Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6827845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.6827928Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6827933Z 2025-08-14T21:52:19.6828042Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6828239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6828332Z return mod(**inputs) 2025-08-14T21:52:19.6828611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6828678Z outputs = self.mobilebert( 2025-08-14T21:52:19.6828961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6829058Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6829326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6829404Z layer_outputs = layer_module( 2025-08-14T21:52:19.6829675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6829827Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6830109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6830229Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6830500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.6830627Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6830899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6830995Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6830999Z 2025-08-14T21:52:19.6831102Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6831296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6831372Z return mod(**inputs) 2025-08-14T21:52:19.6831649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6831726Z outputs = self.mobilebert( 2025-08-14T21:52:19.6831997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6832073Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6832350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6832420Z layer_outputs = layer_module( 2025-08-14T21:52:19.6832690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6832856Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6833184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6833303Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6833576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6833676Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6833680Z 2025-08-14T21:52:19.6833790Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6833985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6834056Z return mod(**inputs) 2025-08-14T21:52:19.6834337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6834405Z outputs = self.mobilebert( 2025-08-14T21:52:19.6834692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6834763Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6835060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6835140Z layer_outputs = layer_module( 2025-08-14T21:52:19.6835446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6835608Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6835880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6835985Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6836260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.6836346Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.6836622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6836711Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6836715Z 2025-08-14T21:52:19.6836815Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6837010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6837072Z return mod(**inputs) 2025-08-14T21:52:19.6837344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6837412Z outputs = self.mobilebert( 2025-08-14T21:52:19.6837852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6837941Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6838216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6838293Z layer_outputs = layer_module( 2025-08-14T21:52:19.6838595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6838686Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6838984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6839060Z self_outputs = self.self( 2025-08-14T21:52:19.6839351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.6839488Z self.query(query_tensor) 2025-08-14T21:52:19.6839492Z 2025-08-14T21:52:19.6839602Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6839831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6839903Z return mod(**inputs) 2025-08-14T21:52:19.6841255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6841338Z outputs = self.mobilebert( 2025-08-14T21:52:19.6841627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6841702Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6841999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6842073Z layer_outputs = layer_module( 2025-08-14T21:52:19.6842373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6842463Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6842786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6842874Z self_outputs = self.self( 2025-08-14T21:52:19.6843199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.6843273Z self.key(key_tensor) 2025-08-14T21:52:19.6843284Z 2025-08-14T21:52:19.6843402Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6843621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6843697Z return mod(**inputs) 2025-08-14T21:52:19.6843990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6844065Z outputs = self.mobilebert( 2025-08-14T21:52:19.6844364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6844441Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6844736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6844808Z layer_outputs = layer_module( 2025-08-14T21:52:19.6845097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6845191Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6845561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6845648Z self_outputs = self.self( 2025-08-14T21:52:19.6845947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.6846022Z self.value(value_tensor) 2025-08-14T21:52:19.6846028Z 2025-08-14T21:52:19.6846126Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6846210Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6846318Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6846536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6846606Z return mod(**inputs) 2025-08-14T21:52:19.6846899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6846984Z outputs = self.mobilebert( 2025-08-14T21:52:19.6847273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6847381Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6847678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6847772Z layer_outputs = layer_module( 2025-08-14T21:52:19.6848080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6848166Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6848470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6848599Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6848898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.6848994Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6848998Z 2025-08-14T21:52:19.6849102Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6849333Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6849405Z return mod(**inputs) 2025-08-14T21:52:19.6849711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6849792Z outputs = self.mobilebert( 2025-08-14T21:52:19.6850083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6850159Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6850456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6850532Z layer_outputs = layer_module( 2025-08-14T21:52:19.6850828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6850998Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6851290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.6851413Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.6851702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6851794Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6851798Z 2025-08-14T21:52:19.6851904Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6852106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6852180Z return mod(**inputs) 2025-08-14T21:52:19.6852450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6852519Z outputs = self.mobilebert( 2025-08-14T21:52:19.6852798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6852868Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6853145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6853213Z layer_outputs = layer_module( 2025-08-14T21:52:19.6853486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6853574Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6853866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6853988Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6854259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.6854403Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6854690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6854777Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6854781Z 2025-08-14T21:52:19.6854880Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6855076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6855140Z return mod(**inputs) 2025-08-14T21:52:19.6855419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6855487Z outputs = self.mobilebert( 2025-08-14T21:52:19.6855765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6855861Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6856127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6856202Z layer_outputs = layer_module( 2025-08-14T21:52:19.6856470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6856564Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6856844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6856956Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6857231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6857324Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6857328Z 2025-08-14T21:52:19.6857429Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6857633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6857698Z return mod(**inputs) 2025-08-14T21:52:19.6857972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6858053Z outputs = self.mobilebert( 2025-08-14T21:52:19.6858328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6858405Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6858672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6858742Z layer_outputs = layer_module( 2025-08-14T21:52:19.6859014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6859104Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6859370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6859484Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6859749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6859919Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6859923Z 2025-08-14T21:52:19.6860021Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6860214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6860304Z return mod(**inputs) 2025-08-14T21:52:19.6860574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6860648Z outputs = self.mobilebert( 2025-08-14T21:52:19.6860913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6860983Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6861255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6861326Z layer_outputs = layer_module( 2025-08-14T21:52:19.6861591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6861687Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6861984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6862139Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6862411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6862494Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6862498Z 2025-08-14T21:52:19.6862610Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6862811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6862889Z return mod(**inputs) 2025-08-14T21:52:19.6863173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6863245Z outputs = self.mobilebert( 2025-08-14T21:52:19.6863539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6863616Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6863899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6863979Z layer_outputs = layer_module( 2025-08-14T21:52:19.6864260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6864362Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6864648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6864776Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6865078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6865199Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6865477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6865568Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6865572Z 2025-08-14T21:52:19.6865674Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6865882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6865965Z return mod(**inputs) 2025-08-14T21:52:19.6866251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6866320Z outputs = self.mobilebert( 2025-08-14T21:52:19.6866590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6866689Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6866968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6867037Z layer_outputs = layer_module( 2025-08-14T21:52:19.6867322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6867415Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6867694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6867803Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6868097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6868192Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6868195Z 2025-08-14T21:52:19.6868312Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6868514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6868579Z return mod(**inputs) 2025-08-14T21:52:19.6868862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6868940Z outputs = self.mobilebert( 2025-08-14T21:52:19.6869227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6869311Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6869589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6869662Z layer_outputs = layer_module( 2025-08-14T21:52:19.6869954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6870052Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6870352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6870469Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6870760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6870889Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6870892Z 2025-08-14T21:52:19.6871001Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6871219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6871293Z return mod(**inputs) 2025-08-14T21:52:19.6871591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6871678Z outputs = self.mobilebert( 2025-08-14T21:52:19.6871972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6872051Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6872355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6872449Z layer_outputs = layer_module( 2025-08-14T21:52:19.6872745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6872841Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6873139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6873289Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6873565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6873656Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6873660Z 2025-08-14T21:52:19.6873761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6873960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6874035Z return mod(**inputs) 2025-08-14T21:52:19.6874322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6874391Z outputs = self.mobilebert( 2025-08-14T21:52:19.6874694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6874784Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6875068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6875138Z layer_outputs = layer_module( 2025-08-14T21:52:19.6875414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6875512Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6875795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6875922Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6876199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6876321Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6876607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6876696Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6876700Z 2025-08-14T21:52:19.6876806Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6877002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6877067Z return mod(**inputs) 2025-08-14T21:52:19.6877357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6877428Z outputs = self.mobilebert( 2025-08-14T21:52:19.6877708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6877791Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6878073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6878152Z layer_outputs = layer_module( 2025-08-14T21:52:19.6878433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6878526Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6878818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6878945Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6879229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6879330Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6879334Z 2025-08-14T21:52:19.6879436Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6879638Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6879704Z return mod(**inputs) 2025-08-14T21:52:19.6879982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6880059Z outputs = self.mobilebert( 2025-08-14T21:52:19.6880328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6880409Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6880685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6880770Z layer_outputs = layer_module( 2025-08-14T21:52:19.6881072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6881164Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6881445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6881553Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6881826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6881946Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6881950Z 2025-08-14T21:52:19.6882057Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6882272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6882349Z return mod(**inputs) 2025-08-14T21:52:19.6882647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6882730Z outputs = self.mobilebert( 2025-08-14T21:52:19.6883024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6883100Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6883396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6883472Z layer_outputs = layer_module( 2025-08-14T21:52:19.6883771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6883866Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6884155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6884294Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6884584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6884671Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6884683Z 2025-08-14T21:52:19.6884789Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6885006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6885103Z return mod(**inputs) 2025-08-14T21:52:19.6885469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6885553Z outputs = self.mobilebert( 2025-08-14T21:52:19.6885862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6885971Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6886274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6886349Z layer_outputs = layer_module( 2025-08-14T21:52:19.6886636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6886742Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6887037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6887163Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6887481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6887630Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6887933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6888027Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6888032Z 2025-08-14T21:52:19.6888140Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6888360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6888428Z return mod(**inputs) 2025-08-14T21:52:19.6888730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6888802Z outputs = self.mobilebert( 2025-08-14T21:52:19.6889093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6889177Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6889469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6889544Z layer_outputs = layer_module( 2025-08-14T21:52:19.6889839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6889963Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6890261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6890349Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6890353Z 2025-08-14T21:52:19.6890460Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6890674Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6890745Z return mod(**inputs) 2025-08-14T21:52:19.6891046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6891119Z outputs = self.mobilebert( 2025-08-14T21:52:19.6891407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6891487Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6891775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6891869Z layer_outputs = layer_module( 2025-08-14T21:52:19.6892167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6892289Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6892603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6892717Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6892721Z 2025-08-14T21:52:19.6892827Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6893041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6893110Z return mod(**inputs) 2025-08-14T21:52:19.6893410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6893484Z outputs = self.mobilebert( 2025-08-14T21:52:19.6893772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6893874Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6894183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6894258Z layer_outputs = layer_module( 2025-08-14T21:52:19.6894550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6894715Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6895007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.6895107Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.6895111Z 2025-08-14T21:52:19.6895215Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6895426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6895496Z return mod(**inputs) 2025-08-14T21:52:19.6895798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6895872Z outputs = self.mobilebert( 2025-08-14T21:52:19.6896160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6896244Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6896540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6896619Z layer_outputs = layer_module( 2025-08-14T21:52:19.6896883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6897034Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6897312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.6897427Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.6897691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6897785Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6897789Z 2025-08-14T21:52:19.6897886Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6898080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6898163Z return mod(**inputs) 2025-08-14T21:52:19.6898442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6898520Z outputs = self.mobilebert( 2025-08-14T21:52:19.6898806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6898884Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6899169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6899243Z layer_outputs = layer_module( 2025-08-14T21:52:19.6899537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6899703Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6899974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6900107Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6900410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.6900526Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6900530Z 2025-08-14T21:52:19.6900637Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6900841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6900916Z return mod(**inputs) 2025-08-14T21:52:19.6901208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6901290Z outputs = self.mobilebert( 2025-08-14T21:52:19.6901570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6901639Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6901911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6901982Z layer_outputs = layer_module( 2025-08-14T21:52:19.6902250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6902406Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6902676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6902802Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6903076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.6903196Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6903479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6903571Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6903575Z 2025-08-14T21:52:19.6903683Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6903876Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6903941Z return mod(**inputs) 2025-08-14T21:52:19.6904223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6904292Z outputs = self.mobilebert( 2025-08-14T21:52:19.6904601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6904670Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6904937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6905031Z layer_outputs = layer_module( 2025-08-14T21:52:19.6905310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6905467Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6905749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6905856Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6906138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6906217Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6906221Z 2025-08-14T21:52:19.6906337Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6906539Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6906617Z return mod(**inputs) 2025-08-14T21:52:19.6906894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6906963Z outputs = self.mobilebert( 2025-08-14T21:52:19.6907226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6907302Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6907566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6907635Z layer_outputs = layer_module( 2025-08-14T21:52:19.6907914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6908069Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6908345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6908450Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6908714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.6908804Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.6909069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6909164Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6909167Z 2025-08-14T21:52:19.6909264Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6909455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6909528Z return mod(**inputs) 2025-08-14T21:52:19.6909795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6909871Z outputs = self.mobilebert( 2025-08-14T21:52:19.6910137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6910206Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6910474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6910571Z layer_outputs = layer_module( 2025-08-14T21:52:19.6910838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6910929Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6911211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6911288Z self_outputs = self.self( 2025-08-14T21:52:19.6911553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.6911622Z self.query(query_tensor) 2025-08-14T21:52:19.6911626Z 2025-08-14T21:52:19.6911728Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6911916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6911998Z return mod(**inputs) 2025-08-14T21:52:19.6912257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6912323Z outputs = self.mobilebert( 2025-08-14T21:52:19.6912605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6912694Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6912959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6913034Z layer_outputs = layer_module( 2025-08-14T21:52:19.6913296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6913383Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6913656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6913725Z self_outputs = self.self( 2025-08-14T21:52:19.6914009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.6914076Z self.key(key_tensor) 2025-08-14T21:52:19.6914079Z 2025-08-14T21:52:19.6914187Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6914381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6914445Z return mod(**inputs) 2025-08-14T21:52:19.6914739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6914806Z outputs = self.mobilebert( 2025-08-14T21:52:19.6915071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6915149Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6915412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6915488Z layer_outputs = layer_module( 2025-08-14T21:52:19.6915754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6915835Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6916106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6916173Z self_outputs = self.self( 2025-08-14T21:52:19.6916447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.6916541Z self.value(value_tensor) 2025-08-14T21:52:19.6916544Z 2025-08-14T21:52:19.6916621Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6916705Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6916802Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6916988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6917075Z return mod(**inputs) 2025-08-14T21:52:19.6917339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6917406Z outputs = self.mobilebert( 2025-08-14T21:52:19.6917672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6917740Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6918004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6918071Z layer_outputs = layer_module( 2025-08-14T21:52:19.6918330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6918432Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6918710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6918833Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6919090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.6919169Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6919172Z 2025-08-14T21:52:19.6919276Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6919460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6919522Z return mod(**inputs) 2025-08-14T21:52:19.6919794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6919861Z outputs = self.mobilebert( 2025-08-14T21:52:19.6920131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6920197Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6920455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6920528Z layer_outputs = layer_module( 2025-08-14T21:52:19.6920787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6920944Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6921205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.6921307Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.6921577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6921656Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6921659Z 2025-08-14T21:52:19.6921763Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6921955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6922018Z return mod(**inputs) 2025-08-14T21:52:19.6922297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6922385Z outputs = self.mobilebert( 2025-08-14T21:52:19.6922649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6922725Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6922999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6923090Z layer_outputs = layer_module( 2025-08-14T21:52:19.6923365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6923446Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6923724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6923844Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6924123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.6924246Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6924542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6924665Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6924669Z 2025-08-14T21:52:19.6924776Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6924988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6925063Z return mod(**inputs) 2025-08-14T21:52:19.6925357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6925505Z outputs = self.mobilebert( 2025-08-14T21:52:19.6925809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6925886Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6926195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6926274Z layer_outputs = layer_module( 2025-08-14T21:52:19.6926586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6926681Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6926952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6927071Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6927341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6927427Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6927440Z 2025-08-14T21:52:19.6927542Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6927737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6927813Z return mod(**inputs) 2025-08-14T21:52:19.6928091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6928161Z outputs = self.mobilebert( 2025-08-14T21:52:19.6928441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6928514Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6928794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6928890Z layer_outputs = layer_module( 2025-08-14T21:52:19.6929162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6929263Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6929558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6929667Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6929947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6930055Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6930059Z 2025-08-14T21:52:19.6930168Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6930362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6930429Z return mod(**inputs) 2025-08-14T21:52:19.6930713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6930802Z outputs = self.mobilebert( 2025-08-14T21:52:19.6931108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6931182Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6931458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6931536Z layer_outputs = layer_module( 2025-08-14T21:52:19.6931813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6931905Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6932190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6932316Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6932600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6932687Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6932691Z 2025-08-14T21:52:19.6932796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6933010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6933079Z return mod(**inputs) 2025-08-14T21:52:19.6933383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6933457Z outputs = self.mobilebert( 2025-08-14T21:52:19.6933750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6933833Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6934133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6934205Z layer_outputs = layer_module( 2025-08-14T21:52:19.6934485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6934576Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6934855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6934977Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6935289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6935417Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6935691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6935807Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6935810Z 2025-08-14T21:52:19.6935911Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6936102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6936176Z return mod(**inputs) 2025-08-14T21:52:19.6936451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6936529Z outputs = self.mobilebert( 2025-08-14T21:52:19.6936803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6936874Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6937172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6937244Z layer_outputs = layer_module( 2025-08-14T21:52:19.6937544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6937795Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6938080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6938197Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6938472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6938558Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6938562Z 2025-08-14T21:52:19.6938673Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6938869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6938951Z return mod(**inputs) 2025-08-14T21:52:19.6939231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6939302Z outputs = self.mobilebert( 2025-08-14T21:52:19.6939587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6939660Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6939936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6940017Z layer_outputs = layer_module( 2025-08-14T21:52:19.6940290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6940389Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6940666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6940776Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6941057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6941163Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6941167Z 2025-08-14T21:52:19.6941277Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6941472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6941593Z return mod(**inputs) 2025-08-14T21:52:19.6941879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6941950Z outputs = self.mobilebert( 2025-08-14T21:52:19.6942256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6942336Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6942609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6942684Z layer_outputs = layer_module( 2025-08-14T21:52:19.6942967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6943052Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6943324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6943441Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6943745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6943855Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6943858Z 2025-08-14T21:52:19.6943957Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6944163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6944224Z return mod(**inputs) 2025-08-14T21:52:19.6944482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6944558Z outputs = self.mobilebert( 2025-08-14T21:52:19.6944816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6944893Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6945152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6945220Z layer_outputs = layer_module( 2025-08-14T21:52:19.6945485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6945571Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6945837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6945951Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6946215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6946333Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6946591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6946677Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6946688Z 2025-08-14T21:52:19.6946784Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6946969Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6947036Z return mod(**inputs) 2025-08-14T21:52:19.6947297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6947362Z outputs = self.mobilebert( 2025-08-14T21:52:19.6947648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6947715Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6947982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6948070Z layer_outputs = layer_module( 2025-08-14T21:52:19.6948334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6948427Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6948691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6948794Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6949063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6949143Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6949147Z 2025-08-14T21:52:19.6949246Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6949446Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6949510Z return mod(**inputs) 2025-08-14T21:52:19.6949803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6949871Z outputs = self.mobilebert( 2025-08-14T21:52:19.6950143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6950210Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6950482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6950558Z layer_outputs = layer_module( 2025-08-14T21:52:19.6950829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6950920Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6951197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6951301Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6951578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6951683Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6951686Z 2025-08-14T21:52:19.6951784Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6951981Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6952046Z return mod(**inputs) 2025-08-14T21:52:19.6952335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6952401Z outputs = self.mobilebert( 2025-08-14T21:52:19.6952668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6952744Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6953007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6953079Z layer_outputs = layer_module( 2025-08-14T21:52:19.6953341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6953429Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6953719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6953835Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6954100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6954204Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6954207Z 2025-08-14T21:52:19.6954303Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6954493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6954556Z return mod(**inputs) 2025-08-14T21:52:19.6954818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6954894Z outputs = self.mobilebert( 2025-08-14T21:52:19.6955155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6955230Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6955515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6955621Z layer_outputs = layer_module( 2025-08-14T21:52:19.6955891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6955976Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6956234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6956356Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6956618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.6956737Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6956998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6957084Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6957088Z 2025-08-14T21:52:19.6957191Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6957372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6957440Z return mod(**inputs) 2025-08-14T21:52:19.6957702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6957766Z outputs = self.mobilebert( 2025-08-14T21:52:19.6958030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6958097Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6958354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6958427Z layer_outputs = layer_module( 2025-08-14T21:52:19.6958687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6958803Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6959057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6959135Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6959139Z 2025-08-14T21:52:19.6959242Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6959454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6959520Z return mod(**inputs) 2025-08-14T21:52:19.6959777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6959856Z outputs = self.mobilebert( 2025-08-14T21:52:19.6960121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6960187Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6960439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6960513Z layer_outputs = layer_module( 2025-08-14T21:52:19.6960770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.6960889Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.6961148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6961265Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6961270Z 2025-08-14T21:52:19.6961375Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6961574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6961645Z return mod(**inputs) 2025-08-14T21:52:19.6961914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6961981Z outputs = self.mobilebert( 2025-08-14T21:52:19.6962253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6962325Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6962596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6962676Z layer_outputs = layer_module( 2025-08-14T21:52:19.6962950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6963114Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6963388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.6963480Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.6963484Z 2025-08-14T21:52:19.6963591Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6963784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6963857Z return mod(**inputs) 2025-08-14T21:52:19.6964136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6964205Z outputs = self.mobilebert( 2025-08-14T21:52:19.6964483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6964554Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6964832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6964902Z layer_outputs = layer_module( 2025-08-14T21:52:19.6965171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6965336Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6965694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.6965820Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.6966108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6966225Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6966229Z 2025-08-14T21:52:19.6966345Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6966565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6966637Z return mod(**inputs) 2025-08-14T21:52:19.6966939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6967008Z outputs = self.mobilebert( 2025-08-14T21:52:19.6967282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6967354Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6967646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6967744Z layer_outputs = layer_module( 2025-08-14T21:52:19.6968011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6968161Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6968434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6968551Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6968836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.6968913Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6968917Z 2025-08-14T21:52:19.6969014Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6969208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6969269Z return mod(**inputs) 2025-08-14T21:52:19.6969540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6969605Z outputs = self.mobilebert( 2025-08-14T21:52:19.6969861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6969938Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6970204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6970268Z layer_outputs = layer_module( 2025-08-14T21:52:19.6970529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.6970676Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.6970941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.6971053Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.6971310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.6971430Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6971711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6971800Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6971804Z 2025-08-14T21:52:19.6971909Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6972106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6972175Z return mod(**inputs) 2025-08-14T21:52:19.6972430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6972502Z outputs = self.mobilebert( 2025-08-14T21:52:19.6972760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6972827Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6973090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6973156Z layer_outputs = layer_module( 2025-08-14T21:52:19.6973428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6973589Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6973866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6973976Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6974247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6974320Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6974324Z 2025-08-14T21:52:19.6974425Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6974601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6974666Z return mod(**inputs) 2025-08-14T21:52:19.6974919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6974983Z outputs = self.mobilebert( 2025-08-14T21:52:19.6975239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6975304Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6975561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6975635Z layer_outputs = layer_module( 2025-08-14T21:52:19.6975891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6976048Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6976311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.6976414Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.6976679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.6976758Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.6977022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6977107Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6977110Z 2025-08-14T21:52:19.6977203Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6977416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6977475Z return mod(**inputs) 2025-08-14T21:52:19.6977748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6977840Z outputs = self.mobilebert( 2025-08-14T21:52:19.6978102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6978176Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6978435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6978500Z layer_outputs = layer_module( 2025-08-14T21:52:19.6978766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6978848Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6979114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6979180Z self_outputs = self.self( 2025-08-14T21:52:19.6979457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.6979548Z self.query(query_tensor) 2025-08-14T21:52:19.6979552Z 2025-08-14T21:52:19.6979648Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6979843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6979903Z return mod(**inputs) 2025-08-14T21:52:19.6980169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6980242Z outputs = self.mobilebert( 2025-08-14T21:52:19.6980506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6980574Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6980844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6980913Z layer_outputs = layer_module( 2025-08-14T21:52:19.6981183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6981263Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6981524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6981597Z self_outputs = self.self( 2025-08-14T21:52:19.6981857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.6981920Z self.key(key_tensor) 2025-08-14T21:52:19.6981931Z 2025-08-14T21:52:19.6982025Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6982211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6982281Z return mod(**inputs) 2025-08-14T21:52:19.6982547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6982613Z outputs = self.mobilebert( 2025-08-14T21:52:19.6982884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6982951Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6983222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6983307Z layer_outputs = layer_module( 2025-08-14T21:52:19.6983566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6983653Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6983928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.6983993Z self_outputs = self.self( 2025-08-14T21:52:19.6984258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.6984324Z self.value(value_tensor) 2025-08-14T21:52:19.6984328Z 2025-08-14T21:52:19.6984412Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6984485Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.6984580Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6984771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6984833Z return mod(**inputs) 2025-08-14T21:52:19.6985110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6985186Z outputs = self.mobilebert( 2025-08-14T21:52:19.6985461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6985538Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6985795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6985861Z layer_outputs = layer_module( 2025-08-14T21:52:19.6986124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6986202Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6986465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6986579Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6986846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.6986934Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6986937Z 2025-08-14T21:52:19.6987032Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6987220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6987290Z return mod(**inputs) 2025-08-14T21:52:19.6987558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6987634Z outputs = self.mobilebert( 2025-08-14T21:52:19.6987907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6987976Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6988241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6988311Z layer_outputs = layer_module( 2025-08-14T21:52:19.6988573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.6988720Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.6988978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.6989087Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.6989363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.6989447Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.6989452Z 2025-08-14T21:52:19.6989570Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6989772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6989841Z return mod(**inputs) 2025-08-14T21:52:19.6990105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6990169Z outputs = self.mobilebert( 2025-08-14T21:52:19.6990432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6990499Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6990763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6990829Z layer_outputs = layer_module( 2025-08-14T21:52:19.6991107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.6991195Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.6991468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.6991585Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.6991854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.6991970Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.6992240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.6992329Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.6992332Z 2025-08-14T21:52:19.6992428Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6992622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6992685Z return mod(**inputs) 2025-08-14T21:52:19.6992957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6993023Z outputs = self.mobilebert( 2025-08-14T21:52:19.6993283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6993360Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6993622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6993699Z layer_outputs = layer_module( 2025-08-14T21:52:19.6993962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6994053Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6994325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6994430Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6994693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.6994778Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.6994782Z 2025-08-14T21:52:19.6994877Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6995083Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6995146Z return mod(**inputs) 2025-08-14T21:52:19.6995411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6995504Z outputs = self.mobilebert( 2025-08-14T21:52:19.6995763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6995839Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6996100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6996165Z layer_outputs = layer_module( 2025-08-14T21:52:19.6996428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6996515Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6996770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.6996895Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.6997170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.6997284Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.6997288Z 2025-08-14T21:52:19.6997383Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.6997567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.6997639Z return mod(**inputs) 2025-08-14T21:52:19.6997903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.6997980Z outputs = self.mobilebert( 2025-08-14T21:52:19.6998243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.6998314Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.6998583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.6998654Z layer_outputs = layer_module( 2025-08-14T21:52:19.6998915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.6999012Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.6999274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.6999402Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.6999666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.6999748Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.6999752Z 2025-08-14T21:52:19.6999860Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7000050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7000121Z return mod(**inputs) 2025-08-14T21:52:19.7000385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7000453Z outputs = self.mobilebert( 2025-08-14T21:52:19.7000723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7000793Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7001082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7001156Z layer_outputs = layer_module( 2025-08-14T21:52:19.7001417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7001528Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7001790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.7001905Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.7002174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.7002286Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.7002557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7002645Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7002649Z 2025-08-14T21:52:19.7002764Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7002965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7003048Z return mod(**inputs) 2025-08-14T21:52:19.7003326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7003393Z outputs = self.mobilebert( 2025-08-14T21:52:19.7003655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7003729Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7003995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7004064Z layer_outputs = layer_module( 2025-08-14T21:52:19.7004335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7004423Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7004691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.7004796Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.7005066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.7005156Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.7005159Z 2025-08-14T21:52:19.7005258Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7005525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7005598Z return mod(**inputs) 2025-08-14T21:52:19.7005883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7005962Z outputs = self.mobilebert( 2025-08-14T21:52:19.7006237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7006309Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7006593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7006664Z layer_outputs = layer_module( 2025-08-14T21:52:19.7006945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7007060Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7007336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.7007455Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.7007753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.7007870Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.7007874Z 2025-08-14T21:52:19.7007976Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7008178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7008248Z return mod(**inputs) 2025-08-14T21:52:19.7008509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7008577Z outputs = self.mobilebert( 2025-08-14T21:52:19.7008843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7008911Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7009193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7009279Z layer_outputs = layer_module( 2025-08-14T21:52:19.7009542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7009636Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7009897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.7010018Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.7010280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.7010359Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.7010362Z 2025-08-14T21:52:19.7010469Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7010654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7010717Z return mod(**inputs) 2025-08-14T21:52:19.7010989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7011053Z outputs = self.mobilebert( 2025-08-14T21:52:19.7011318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7011385Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7011647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7011720Z layer_outputs = layer_module( 2025-08-14T21:52:19.7011981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7012076Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7012344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.7012461Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.7012735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.7012849Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.7013115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7013227Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7013231Z 2025-08-14T21:52:19.7013330Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7013544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7013611Z return mod(**inputs) 2025-08-14T21:52:19.7013885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7013962Z outputs = self.mobilebert( 2025-08-14T21:52:19.7014236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7014314Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7014575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7014647Z layer_outputs = layer_module( 2025-08-14T21:52:19.7014946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7015036Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7015312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.7015425Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.7015682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.7015768Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.7015771Z 2025-08-14T21:52:19.7015865Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7016049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7016119Z return mod(**inputs) 2025-08-14T21:52:19.7016382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7016454Z outputs = self.mobilebert( 2025-08-14T21:52:19.7016713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7016781Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7017047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7017113Z layer_outputs = layer_module( 2025-08-14T21:52:19.7017371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7017466Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7017732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.7017843Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.7018120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.7018223Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.7018227Z 2025-08-14T21:52:19.7018329Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7018512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7018577Z return mod(**inputs) 2025-08-14T21:52:19.7018834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7018917Z outputs = self.mobilebert( 2025-08-14T21:52:19.7019183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7019249Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7019512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7019596Z layer_outputs = layer_module( 2025-08-14T21:52:19.7019852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7019945Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7020202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.7020316Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.7020582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.7020661Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.7020664Z 2025-08-14T21:52:19.7020780Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7020990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7021054Z return mod(**inputs) 2025-08-14T21:52:19.7021326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7021391Z outputs = self.mobilebert( 2025-08-14T21:52:19.7021656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7021724Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7021981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7022053Z layer_outputs = layer_module( 2025-08-14T21:52:19.7022315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7022404Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7022680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.7022797Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.7023069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.7023184Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.7023448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7023544Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7023547Z 2025-08-14T21:52:19.7023645Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7023844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7023911Z return mod(**inputs) 2025-08-14T21:52:19.7024188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7024262Z outputs = self.mobilebert( 2025-08-14T21:52:19.7024520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7024589Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7024858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7024942Z layer_outputs = layer_module( 2025-08-14T21:52:19.7025211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.7025326Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.7025626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.7025716Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.7025720Z 2025-08-14T21:52:19.7025818Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7026019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7026083Z return mod(**inputs) 2025-08-14T21:52:19.7026358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7026436Z outputs = self.mobilebert( 2025-08-14T21:52:19.7026709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7026799Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7027111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7027180Z layer_outputs = layer_module( 2025-08-14T21:52:19.7027453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.7027566Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.7027831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.7027945Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.7027948Z 2025-08-14T21:52:19.7028045Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7028240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7028304Z return mod(**inputs) 2025-08-14T21:52:19.7028575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7028648Z outputs = self.mobilebert( 2025-08-14T21:52:19.7028915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7028984Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7029256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7029325Z layer_outputs = layer_module( 2025-08-14T21:52:19.7029598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.7029749Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.7030018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.7030116Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.7030120Z 2025-08-14T21:52:19.7030217Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7030412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7030475Z return mod(**inputs) 2025-08-14T21:52:19.7030746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7030843Z outputs = self.mobilebert( 2025-08-14T21:52:19.7031112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7031182Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7031459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7031546Z layer_outputs = layer_module( 2025-08-14T21:52:19.7031823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.7031975Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.7032241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.7032366Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.7032636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7032730Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7032734Z 2025-08-14T21:52:19.7032847Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7033053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7033125Z return mod(**inputs) 2025-08-14T21:52:19.7033398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7033472Z outputs = self.mobilebert( 2025-08-14T21:52:19.7033738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7033807Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7034084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7034153Z layer_outputs = layer_module( 2025-08-14T21:52:19.7034422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.7034580Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.7034846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.7034967Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.7035235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.7035315Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.7035320Z 2025-08-14T21:52:19.7035426Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7035617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7035685Z return mod(**inputs) 2025-08-14T21:52:19.7035965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7036040Z outputs = self.mobilebert( 2025-08-14T21:52:19.7036341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7036416Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7036706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7036788Z layer_outputs = layer_module( 2025-08-14T21:52:19.7037077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.7037277Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.7037549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.7037806Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.7038099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.7038218Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.7038497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7038588Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7038594Z 2025-08-14T21:52:19.7038694Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7038900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7038967Z return mod(**inputs) 2025-08-14T21:52:19.7039307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7039408Z outputs = self.mobilebert( 2025-08-14T21:52:19.7039701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7039783Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7040069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7040143Z layer_outputs = layer_module( 2025-08-14T21:52:19.7040438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.7040605Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.7040918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.7041028Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.7041303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.7041395Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.7041399Z 2025-08-14T21:52:19.7041499Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7041705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7041771Z return mod(**inputs) 2025-08-14T21:52:19.7042044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7042125Z outputs = self.mobilebert( 2025-08-14T21:52:19.7042400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7042474Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7042758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7042833Z layer_outputs = layer_module( 2025-08-14T21:52:19.7043129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.7043290Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.7043579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.7043730Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.7044024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.7044145Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.7044435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7044527Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7044531Z 2025-08-14T21:52:19.7044642Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7044865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7044940Z return mod(**inputs) 2025-08-14T21:52:19.7045234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7045309Z outputs = self.mobilebert( 2025-08-14T21:52:19.7045667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7045779Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7046092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7046176Z layer_outputs = layer_module( 2025-08-14T21:52:19.7046468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.7046568Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.7046833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.7046906Z self_outputs = self.self( 2025-08-14T21:52:19.7047183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.7047252Z self.query(query_tensor) 2025-08-14T21:52:19.7047256Z 2025-08-14T21:52:19.7047364Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7047560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7047626Z return mod(**inputs) 2025-08-14T21:52:19.7047914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7047981Z outputs = self.mobilebert( 2025-08-14T21:52:19.7048254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7048333Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7048610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7048686Z layer_outputs = layer_module( 2025-08-14T21:52:19.7048964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.7049045Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.7049327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.7049396Z self_outputs = self.self( 2025-08-14T21:52:19.7049670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.7049744Z self.key(key_tensor) 2025-08-14T21:52:19.7049747Z 2025-08-14T21:52:19.7049844Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7050060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7050123Z return mod(**inputs) 2025-08-14T21:52:19.7050390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7050480Z outputs = self.mobilebert( 2025-08-14T21:52:19.7050754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7050829Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7051103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7051170Z layer_outputs = layer_module( 2025-08-14T21:52:19.7051447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.7051529Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.7051798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.7051870Z self_outputs = self.self( 2025-08-14T21:52:19.7052191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.7052283Z self.value(value_tensor) 2025-08-14T21:52:19.7052288Z 2025-08-14T21:52:19.7052378Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.7052452Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.7052553Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7052738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7052799Z return mod(**inputs) 2025-08-14T21:52:19.7053068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7053135Z outputs = self.mobilebert( 2025-08-14T21:52:19.7053399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7053466Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7053728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7053803Z layer_outputs = layer_module( 2025-08-14T21:52:19.7054062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.7054148Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.7054409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.7054524Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.7054795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.7054878Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.7054883Z 2025-08-14T21:52:19.7054981Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7055179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7055242Z return mod(**inputs) 2025-08-14T21:52:19.7055519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7055588Z outputs = self.mobilebert( 2025-08-14T21:52:19.7055857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7055958Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7056234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7056315Z layer_outputs = layer_module( 2025-08-14T21:52:19.7056593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.7056773Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.7057057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.7057167Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.7057441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.7057531Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.7057536Z 2025-08-14T21:52:19.7057632Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7057825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7057889Z return mod(**inputs) 2025-08-14T21:52:19.7058169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7058265Z outputs = self.mobilebert( 2025-08-14T21:52:19.7058539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7058615Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7058874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7058942Z layer_outputs = layer_module( 2025-08-14T21:52:19.7059210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.7059287Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.7059542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.7059664Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.7059919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.7060042Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.7060297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7060382Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7060385Z 2025-08-14T21:52:19.7060490Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7060669Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7060738Z return mod(**inputs) 2025-08-14T21:52:19.7060997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7061066Z outputs = self.mobilebert( 2025-08-14T21:52:19.7061334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7061402Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7061671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7061739Z layer_outputs = layer_module( 2025-08-14T21:52:19.7061998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7062125Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7062392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.7062500Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.7062794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.7062873Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.7062876Z 2025-08-14T21:52:19.7062977Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7063162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7063226Z return mod(**inputs) 2025-08-14T21:52:19.7063496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7063566Z outputs = self.mobilebert( 2025-08-14T21:52:19.7063838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7063929Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7064213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7064294Z layer_outputs = layer_module( 2025-08-14T21:52:19.7064555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7064644Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7064919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.7065026Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.7065295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.7065404Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.7065408Z 2025-08-14T21:52:19.7065508Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7065710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7065776Z return mod(**inputs) 2025-08-14T21:52:19.7066053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7066124Z outputs = self.mobilebert( 2025-08-14T21:52:19.7066392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7066476Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7066743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7066812Z layer_outputs = layer_module( 2025-08-14T21:52:19.7067087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7067180Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7067466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.7067585Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.7067846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.7067938Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.7067964Z 2025-08-14T21:52:19.7068065Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7068265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7068330Z return mod(**inputs) 2025-08-14T21:52:19.7068599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7068708Z outputs = self.mobilebert( 2025-08-14T21:52:19.7068984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7069056Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7069341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7069422Z layer_outputs = layer_module( 2025-08-14T21:52:19.7069703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7069794Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7070080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.7070211Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.7070492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.7070620Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.7070886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7070975Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7070978Z 2025-08-14T21:52:19.7071085Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7071277Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7071348Z return mod(**inputs) 2025-08-14T21:52:19.7071621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7071691Z outputs = self.mobilebert( 2025-08-14T21:52:19.7071968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7072037Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7072307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7072384Z layer_outputs = layer_module( 2025-08-14T21:52:19.7072653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7072751Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7073019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.7073126Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.7073405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.7073485Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.7073489Z 2025-08-14T21:52:19.7073596Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7073788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7073852Z return mod(**inputs) 2025-08-14T21:52:19.7074129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7074213Z outputs = self.mobilebert( 2025-08-14T21:52:19.7074490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7074569Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7074853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7074928Z layer_outputs = layer_module( 2025-08-14T21:52:19.7075195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7075282Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7075554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.7075659Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.7075929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.7076034Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.7076056Z 2025-08-14T21:52:19.7076156Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7076369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7076432Z return mod(**inputs) 2025-08-14T21:52:19.7076702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7076779Z outputs = self.mobilebert( 2025-08-14T21:52:19.7077046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7077124Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7077391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7077458Z layer_outputs = layer_module( 2025-08-14T21:52:19.7077732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7077822Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7078096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.7078215Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.7078483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.7078570Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.7078575Z 2025-08-14T21:52:19.7078672Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7078863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7078933Z return mod(**inputs) 2025-08-14T21:52:19.7079229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7079316Z outputs = self.mobilebert( 2025-08-14T21:52:19.7079617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7079693Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7079999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7080073Z layer_outputs = layer_module( 2025-08-14T21:52:19.7080388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7081245Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7081553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.7081714Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.7082016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.7082147Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.7082469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7082566Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7082570Z 2025-08-14T21:52:19.7082688Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7082906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7082979Z return mod(**inputs) 2025-08-14T21:52:19.7083310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7083388Z outputs = self.mobilebert( 2025-08-14T21:52:19.7083722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7083799Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7084111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7084193Z layer_outputs = layer_module( 2025-08-14T21:52:19.7084502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7084601Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7084919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.7085037Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.7085353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.7085588Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.7085596Z 2025-08-14T21:52:19.7085713Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7085947Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7086019Z return mod(**inputs) 2025-08-14T21:52:19.7086331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7086411Z outputs = self.mobilebert( 2025-08-14T21:52:19.7086719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7086810Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7087122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7087200Z layer_outputs = layer_module( 2025-08-14T21:52:19.7087518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7087615Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7087928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.7088073Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.7088442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.7088569Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.7088575Z 2025-08-14T21:52:19.7088704Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7088938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7089009Z return mod(**inputs) 2025-08-14T21:52:19.7089317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7089399Z outputs = self.mobilebert( 2025-08-14T21:52:19.7089710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7089790Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7090109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7090185Z layer_outputs = layer_module( 2025-08-14T21:52:19.7090514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7090632Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7090945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.7091086Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.7091396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.7091493Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.7091498Z 2025-08-14T21:52:19.7091608Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7091830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7091909Z return mod(**inputs) 2025-08-14T21:52:19.7092212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7092298Z outputs = self.mobilebert( 2025-08-14T21:52:19.7092606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7092683Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7093060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7093137Z layer_outputs = layer_module( 2025-08-14T21:52:19.7093446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7093555Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7093854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.7093995Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.7094293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.7094421Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.7094732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7094839Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7094843Z 2025-08-14T21:52:19.7094956Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7095188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7095256Z return mod(**inputs) 2025-08-14T21:52:19.7095561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7095651Z outputs = self.mobilebert( 2025-08-14T21:52:19.7095943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7096025Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7096313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7096394Z layer_outputs = layer_module( 2025-08-14T21:52:19.7096693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.7096817Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.7097116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.7097219Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.7097224Z 2025-08-14T21:52:19.7097356Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7097566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7097634Z return mod(**inputs) 2025-08-14T21:52:19.7097933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7098005Z outputs = self.mobilebert( 2025-08-14T21:52:19.7098303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7098389Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7098678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7098760Z layer_outputs = layer_module( 2025-08-14T21:52:19.7099050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.7099172Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.7099462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.7099575Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.7099579Z 2025-08-14T21:52:19.7099691Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7099893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7099962Z return mod(**inputs) 2025-08-14T21:52:19.7100260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7100335Z outputs = self.mobilebert( 2025-08-14T21:52:19.7100623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7100705Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7100990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7101068Z layer_outputs = layer_module( 2025-08-14T21:52:19.7101366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.7101529Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.7101844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.7101942Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.7101948Z 2025-08-14T21:52:19.7102079Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7102289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7102356Z return mod(**inputs) 2025-08-14T21:52:19.7102667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7102737Z outputs = self.mobilebert( 2025-08-14T21:52:19.7103014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7103093Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7103370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7103450Z layer_outputs = layer_module( 2025-08-14T21:52:19.7103756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.7103940Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.7104243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.7104369Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.7104666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7104760Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7104766Z 2025-08-14T21:52:19.7104872Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7105084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7105152Z return mod(**inputs) 2025-08-14T21:52:19.7105454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7105525Z outputs = self.mobilebert( 2025-08-14T21:52:19.7105800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7105879Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7106152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7106222Z layer_outputs = layer_module( 2025-08-14T21:52:19.7106499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.7106653Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.7106935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.7107058Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.7107331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.7107422Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.7107426Z 2025-08-14T21:52:19.7107526Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7107727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7107792Z return mod(**inputs) 2025-08-14T21:52:19.7108091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7108170Z outputs = self.mobilebert( 2025-08-14T21:52:19.7108444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7108531Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7108816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7108887Z layer_outputs = layer_module( 2025-08-14T21:52:19.7109169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.7109323Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.7109598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.7109726Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.7110017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.7110146Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.7110434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7110526Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7110530Z 2025-08-14T21:52:19.7110638Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7110832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7110905Z return mod(**inputs) 2025-08-14T21:52:19.7111181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7111250Z outputs = self.mobilebert( 2025-08-14T21:52:19.7111530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7111602Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7111877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7111954Z layer_outputs = layer_module( 2025-08-14T21:52:19.7112223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.7112389Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.7112665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.7112775Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.7113059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.7113142Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.7113145Z 2025-08-14T21:52:19.7113255Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7113454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7113519Z return mod(**inputs) 2025-08-14T21:52:19.7113804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7113872Z outputs = self.mobilebert( 2025-08-14T21:52:19.7114143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7114241Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7114515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7114595Z layer_outputs = layer_module( 2025-08-14T21:52:19.7114890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.7115046Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.7115325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.7115432Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.7115708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.7115795Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.7116067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7116181Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7116187Z 2025-08-14T21:52:19.7116313Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7116518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7116583Z return mod(**inputs) 2025-08-14T21:52:19.7116859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7116936Z outputs = self.mobilebert( 2025-08-14T21:52:19.7117211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7117284Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7117568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7117638Z layer_outputs = layer_module( 2025-08-14T21:52:19.7117924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.7118007Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.7118286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.7118363Z self_outputs = self.self( 2025-08-14T21:52:19.7118628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.7118695Z self.query(query_tensor) 2025-08-14T21:52:19.7118708Z 2025-08-14T21:52:19.7118804Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7118994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7119064Z return mod(**inputs) 2025-08-14T21:52:19.7119336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7119404Z outputs = self.mobilebert( 2025-08-14T21:52:19.7119680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7119747Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7120020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7120088Z layer_outputs = layer_module( 2025-08-14T21:52:19.7120355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.7120463Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.7120730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.7120815Z self_outputs = self.self( 2025-08-14T21:52:19.7121099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.7121163Z self.key(key_tensor) 2025-08-14T21:52:19.7121166Z 2025-08-14T21:52:19.7121271Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7121461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7121524Z return mod(**inputs) 2025-08-14T21:52:19.7121807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7121877Z outputs = self.mobilebert( 2025-08-14T21:52:19.7122158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7122244Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7122529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7122606Z layer_outputs = layer_module( 2025-08-14T21:52:19.7122872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.7122951Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.7123224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.7123293Z self_outputs = self.self( 2025-08-14T21:52:19.7123567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.7123636Z self.value(value_tensor) 2025-08-14T21:52:19.7123639Z 2025-08-14T21:52:19.7123720Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.7123805Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.7123903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7124095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7124168Z return mod(**inputs) 2025-08-14T21:52:19.7124442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7124517Z outputs = self.mobilebert( 2025-08-14T21:52:19.7124788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7124859Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7125136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7125206Z layer_outputs = layer_module( 2025-08-14T21:52:19.7125567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.7125656Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.7125950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.7126092Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.7126395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.7126486Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.7126523Z 2025-08-14T21:52:19.7126636Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7126851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7126932Z return mod(**inputs) 2025-08-14T21:52:19.7127259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7127337Z outputs = self.mobilebert( 2025-08-14T21:52:19.7127643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7127715Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7127994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7128064Z layer_outputs = layer_module( 2025-08-14T21:52:19.7128338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.7128512Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.7128824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.7128959Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.7129260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.7129346Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.7129349Z 2025-08-14T21:52:19.7129463Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7129682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7129753Z return mod(**inputs) 2025-08-14T21:52:19.7130056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7130130Z outputs = self.mobilebert( 2025-08-14T21:52:19.7130427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7130506Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7130796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7130887Z layer_outputs = layer_module( 2025-08-14T21:52:19.7131162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.7131244Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.7131527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.7131649Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.7131931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.7132058Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.7132345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7132450Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7132453Z 2025-08-14T21:52:19.7132559Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7132786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7132854Z return mod(**inputs) 2025-08-14T21:52:19.7133169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7133249Z outputs = self.mobilebert( 2025-08-14T21:52:19.7133541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7133642Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7133939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7134012Z layer_outputs = layer_module( 2025-08-14T21:52:19.7134310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7134408Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7134705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.7134824Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.7135100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.7135208Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.7135214Z 2025-08-14T21:52:19.7135322Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7135546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7135623Z return mod(**inputs) 2025-08-14T21:52:19.7135914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7135993Z outputs = self.mobilebert( 2025-08-14T21:52:19.7136279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7136357Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7136651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7136726Z layer_outputs = layer_module( 2025-08-14T21:52:19.7137016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7137123Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7137408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.7137527Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.7137962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.7138083Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.7138090Z 2025-08-14T21:52:19.7138252Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7138476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7138550Z return mod(**inputs) 2025-08-14T21:52:19.7138844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7138917Z outputs = self.mobilebert( 2025-08-14T21:52:19.7139213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7139289Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7139576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7139662Z layer_outputs = layer_module( 2025-08-14T21:52:19.7139999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7140103Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7140394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.7140555Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.7140854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.7140940Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.7140944Z 2025-08-14T21:52:19.7141059Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7141264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7141334Z return mod(**inputs) 2025-08-14T21:52:19.7141635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7141708Z outputs = self.mobilebert( 2025-08-14T21:52:19.7142032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7142120Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7142442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7142524Z layer_outputs = layer_module( 2025-08-14T21:52:19.7142814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7142909Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7143205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.7143334Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.7143630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.7143756Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.7144044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7144146Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7144149Z 2025-08-14T21:52:19.7144253Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7144467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7144530Z return mod(**inputs) 2025-08-14T21:52:19.7144802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7144878Z outputs = self.mobilebert( 2025-08-14T21:52:19.7145152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7145221Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7145488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7145555Z layer_outputs = layer_module( 2025-08-14T21:52:19.7145821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7145908Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7146164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.7146294Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.7146556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.7146641Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.7146659Z 2025-08-14T21:52:19.7146755Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7146941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7147009Z return mod(**inputs) 2025-08-14T21:52:19.7147273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7147341Z outputs = self.mobilebert( 2025-08-14T21:52:19.7147614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7147687Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7147962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7148030Z layer_outputs = layer_module( 2025-08-14T21:52:19.7148311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7148424Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7148688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.7148798Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.7149063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.7149169Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.7149174Z 2025-08-14T21:52:19.7149276Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7149463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7149527Z return mod(**inputs) 2025-08-14T21:52:19.7149802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7149869Z outputs = self.mobilebert( 2025-08-14T21:52:19.7150138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7150208Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7150470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7150544Z layer_outputs = layer_module( 2025-08-14T21:52:19.7150808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7150903Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7151173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.7151296Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.7151573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.7151654Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.7151658Z 2025-08-14T21:52:19.7151757Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7151953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7152018Z return mod(**inputs) 2025-08-14T21:52:19.7152353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7152432Z outputs = self.mobilebert( 2025-08-14T21:52:19.7152732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7152865Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7153158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7153240Z layer_outputs = layer_module( 2025-08-14T21:52:19.7153535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7153624Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7153908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.7154030Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.7154322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.7154453Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.7154753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7154851Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7154854Z 2025-08-14T21:52:19.7154956Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7155149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7155218Z return mod(**inputs) 2025-08-14T21:52:19.7155493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7155569Z outputs = self.mobilebert( 2025-08-14T21:52:19.7155843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7155914Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7156205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7156272Z layer_outputs = layer_module( 2025-08-14T21:52:19.7156538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7156632Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7156897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.7157010Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.7157276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.7157358Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.7157362Z 2025-08-14T21:52:19.7157468Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7157657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7157726Z return mod(**inputs) 2025-08-14T21:52:19.7157996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7158062Z outputs = self.mobilebert( 2025-08-14T21:52:19.7158336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7158424Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7158682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7158756Z layer_outputs = layer_module( 2025-08-14T21:52:19.7159012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7159121Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7159382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.7159485Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.7159753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.7159855Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.7159860Z 2025-08-14T21:52:19.7159962Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7160148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7160224Z return mod(**inputs) 2025-08-14T21:52:19.7160515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7160581Z outputs = self.mobilebert( 2025-08-14T21:52:19.7160848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7160916Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7161174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7161249Z layer_outputs = layer_module( 2025-08-14T21:52:19.7161509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7161594Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7161863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.7161985Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.7162258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.7162339Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.7162342Z 2025-08-14T21:52:19.7162436Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7162632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7162695Z return mod(**inputs) 2025-08-14T21:52:19.7162972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7163039Z outputs = self.mobilebert( 2025-08-14T21:52:19.7163306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7163383Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7163651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7163719Z layer_outputs = layer_module( 2025-08-14T21:52:19.7163992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7164079Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7164351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.7164485Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.7164755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.7164908Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.7165174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7165265Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7165269Z 2025-08-14T21:52:19.7165366Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7165644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7165726Z return mod(**inputs) 2025-08-14T21:52:19.7166030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7166108Z outputs = self.mobilebert( 2025-08-14T21:52:19.7166436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7166518Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7166848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7166926Z layer_outputs = layer_module( 2025-08-14T21:52:19.7167205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.7167329Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.7167594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.7167685Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.7167688Z 2025-08-14T21:52:19.7167784Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7167976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7168050Z return mod(**inputs) 2025-08-14T21:52:19.7168317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7168386Z outputs = self.mobilebert( 2025-08-14T21:52:19.7168660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7168730Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7169004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7169074Z layer_outputs = layer_module( 2025-08-14T21:52:19.7169339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.7169459Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.7169722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.7169838Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.7169842Z 2025-08-14T21:52:19.7169940Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7170127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7170198Z return mod(**inputs) 2025-08-14T21:52:19.7170466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7170554Z outputs = self.mobilebert( 2025-08-14T21:52:19.7170827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7170897Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7171171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7171256Z layer_outputs = layer_module( 2025-08-14T21:52:19.7171522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.7171682Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.7171948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.7172045Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.7172050Z 2025-08-14T21:52:19.7172148Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7172337Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7172406Z return mod(**inputs) 2025-08-14T21:52:19.7172688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7172773Z outputs = self.mobilebert( 2025-08-14T21:52:19.7173050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7173120Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7173394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7173462Z layer_outputs = layer_module( 2025-08-14T21:52:19.7173734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.7173893Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.7174160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.7174288Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.7174554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7174643Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7174646Z 2025-08-14T21:52:19.7174752Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7174942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7175013Z return mod(**inputs) 2025-08-14T21:52:19.7175281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7175347Z outputs = self.mobilebert( 2025-08-14T21:52:19.7175621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7175693Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7175959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7176034Z layer_outputs = layer_module( 2025-08-14T21:52:19.7176299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.7176455Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.7176740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.7176857Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.7177135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.7177232Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.7177237Z 2025-08-14T21:52:19.7177345Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7177537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7177601Z return mod(**inputs) 2025-08-14T21:52:19.7177880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7177947Z outputs = self.mobilebert( 2025-08-14T21:52:19.7178216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7178295Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7178579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7178659Z layer_outputs = layer_module( 2025-08-14T21:52:19.7178938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.7179090Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.7179375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.7179488Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.7179753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.7179867Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.7180128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7180222Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7180225Z 2025-08-14T21:52:19.7180323Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7180513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7180574Z return mod(**inputs) 2025-08-14T21:52:19.7180835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7180905Z outputs = self.mobilebert( 2025-08-14T21:52:19.7181163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7181233Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7181505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7181574Z layer_outputs = layer_module( 2025-08-14T21:52:19.7181840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.7181991Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.7182250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.7182361Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.7182618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.7182722Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.7182726Z 2025-08-14T21:52:19.7182821Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7183006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7183092Z return mod(**inputs) 2025-08-14T21:52:19.7183360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7183425Z outputs = self.mobilebert( 2025-08-14T21:52:19.7183693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7183761Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7184031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7184100Z layer_outputs = layer_module( 2025-08-14T21:52:19.7184369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.7184546Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.7184841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 444, in forward 2025-08-14T21:52:19.7184953Z shared_attention_input = self.attention(hidden_states) 2025-08-14T21:52:19.7185214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 410, in forward 2025-08-14T21:52:19.7185296Z layer_input = self.LayerNorm(layer_input) 2025-08-14T21:52:19.7185564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7185651Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7185654Z 2025-08-14T21:52:19.7185756Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7185946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7186009Z return mod(**inputs) 2025-08-14T21:52:19.7186289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7186355Z outputs = self.mobilebert( 2025-08-14T21:52:19.7186622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7186697Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7186964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7187039Z layer_outputs = layer_module( 2025-08-14T21:52:19.7187308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.7187388Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.7187667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.7187739Z self_outputs = self.self( 2025-08-14T21:52:19.7188017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 245, in forward 2025-08-14T21:52:19.7188095Z self.query(query_tensor) 2025-08-14T21:52:19.7188099Z 2025-08-14T21:52:19.7188196Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7188399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7188461Z return mod(**inputs) 2025-08-14T21:52:19.7188760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7188834Z outputs = self.mobilebert( 2025-08-14T21:52:19.7189104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7189199Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7189521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7189587Z layer_outputs = layer_module( 2025-08-14T21:52:19.7189849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.7189927Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.7190192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.7190267Z self_outputs = self.self( 2025-08-14T21:52:19.7190525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 250, in forward 2025-08-14T21:52:19.7190611Z self.key(key_tensor) 2025-08-14T21:52:19.7190616Z 2025-08-14T21:52:19.7190718Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7190923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7190996Z return mod(**inputs) 2025-08-14T21:52:19.7191280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7191353Z outputs = self.mobilebert( 2025-08-14T21:52:19.7191614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7191683Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7191962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7192030Z layer_outputs = layer_module( 2025-08-14T21:52:19.7192299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.7192391Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.7192670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 334, in forward 2025-08-14T21:52:19.7192745Z self_outputs = self.self( 2025-08-14T21:52:19.7193026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 255, in forward 2025-08-14T21:52:19.7193097Z self.value(value_tensor) 2025-08-14T21:52:19.7193100Z 2025-08-14T21:52:19.7193192Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.7193270Z cudagraph partition due to non gpu ops 2025-08-14T21:52:19.7193368Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7193570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7193636Z return mod(**inputs) 2025-08-14T21:52:19.7193929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7193996Z outputs = self.mobilebert( 2025-08-14T21:52:19.7194267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7194354Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7194612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7194705Z layer_outputs = layer_module( 2025-08-14T21:52:19.7194965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.7195041Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.7195307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.7195440Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.7195713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 292, in forward 2025-08-14T21:52:19.7195801Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.7195805Z 2025-08-14T21:52:19.7195903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7196101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7196168Z return mod(**inputs) 2025-08-14T21:52:19.7196443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7196520Z outputs = self.mobilebert( 2025-08-14T21:52:19.7196808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7196903Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7197180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7197250Z layer_outputs = layer_module( 2025-08-14T21:52:19.7197538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 496, in forward 2025-08-14T21:52:19.7197690Z query_tensor, key_tensor, value_tensor, layer_input = self.bottleneck(hidden_states) 2025-08-14T21:52:19.7197960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 440, in forward 2025-08-14T21:52:19.7198073Z bottlenecked_hidden_states = self.input(hidden_states) 2025-08-14T21:52:19.7198339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 409, in forward 2025-08-14T21:52:19.7198428Z layer_input = self.dense(hidden_states) 2025-08-14T21:52:19.7198432Z 2025-08-14T21:52:19.7198529Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7198718Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7198788Z return mod(**inputs) 2025-08-14T21:52:19.7199056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7199130Z outputs = self.mobilebert( 2025-08-14T21:52:19.7199401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7199472Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7199753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7199824Z layer_outputs = layer_module( 2025-08-14T21:52:19.7200098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 500, in forward 2025-08-14T21:52:19.7200186Z self_attention_outputs = self.attention( 2025-08-14T21:52:19.7200459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 344, in forward 2025-08-14T21:52:19.7200584Z attention_output = self.output(self_outputs[0], layer_input) 2025-08-14T21:52:19.7200855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 295, in forward 2025-08-14T21:52:19.7200997Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.7201282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7201401Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7201404Z 2025-08-14T21:52:19.7201514Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7201711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7201777Z return mod(**inputs) 2025-08-14T21:52:19.7202063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7202130Z outputs = self.mobilebert( 2025-08-14T21:52:19.7202412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7202490Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7202776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7202876Z layer_outputs = layer_module( 2025-08-14T21:52:19.7203187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7203286Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7203577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.7203691Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.7203986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.7204075Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.7204079Z 2025-08-14T21:52:19.7204184Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7204409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7204474Z return mod(**inputs) 2025-08-14T21:52:19.7204758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7204829Z outputs = self.mobilebert( 2025-08-14T21:52:19.7205100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7205177Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7205526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7205610Z layer_outputs = layer_module( 2025-08-14T21:52:19.7205921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7206022Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7206333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.7206456Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.7206755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.7206893Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.7206897Z 2025-08-14T21:52:19.7206999Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7207200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7207291Z return mod(**inputs) 2025-08-14T21:52:19.7207572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7207654Z outputs = self.mobilebert( 2025-08-14T21:52:19.7207929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7208021Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7208309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7208380Z layer_outputs = layer_module( 2025-08-14T21:52:19.7208660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7208752Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7209027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.7209160Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.7209454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.7209548Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.7209574Z 2025-08-14T21:52:19.7209678Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7209873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7209947Z return mod(**inputs) 2025-08-14T21:52:19.7210227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7210296Z outputs = self.mobilebert( 2025-08-14T21:52:19.7210578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7210650Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7210931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7211003Z layer_outputs = layer_module( 2025-08-14T21:52:19.7211280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7211379Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7211653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.7211784Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.7212058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.7212177Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.7212458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7212548Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7212551Z 2025-08-14T21:52:19.7212660Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7212851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7212917Z return mod(**inputs) 2025-08-14T21:52:19.7213206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7213274Z outputs = self.mobilebert( 2025-08-14T21:52:19.7213547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7213643Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7213918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7213996Z layer_outputs = layer_module( 2025-08-14T21:52:19.7214286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7214377Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7214656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.7214764Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.7215050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.7215131Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.7215135Z 2025-08-14T21:52:19.7215232Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7215445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7215513Z return mod(**inputs) 2025-08-14T21:52:19.7215799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7215876Z outputs = self.mobilebert( 2025-08-14T21:52:19.7216143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7216221Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7216486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7216555Z layer_outputs = layer_module( 2025-08-14T21:52:19.7216831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7216918Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7217195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.7217302Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.7217570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.7217681Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.7217685Z 2025-08-14T21:52:19.7217783Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7217974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7218046Z return mod(**inputs) 2025-08-14T21:52:19.7218318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7218394Z outputs = self.mobilebert( 2025-08-14T21:52:19.7218661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7218734Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7219015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7219084Z layer_outputs = layer_module( 2025-08-14T21:52:19.7219358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7219447Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7219732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.7219859Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.7220123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.7220220Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.7220232Z 2025-08-14T21:52:19.7220332Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7220521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7220592Z return mod(**inputs) 2025-08-14T21:52:19.7220864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7220931Z outputs = self.mobilebert( 2025-08-14T21:52:19.7221206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7221276Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7221568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7221639Z layer_outputs = layer_module( 2025-08-14T21:52:19.7221919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7222017Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7222285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.7222402Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.7222675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.7222795Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.7223077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7223169Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7223172Z 2025-08-14T21:52:19.7223275Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7223477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7223543Z return mod(**inputs) 2025-08-14T21:52:19.7223831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7223902Z outputs = self.mobilebert( 2025-08-14T21:52:19.7224179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7224264Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7224541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7224615Z layer_outputs = layer_module( 2025-08-14T21:52:19.7224903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7224995Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7225272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.7225379Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.7225651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.7225759Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.7225763Z 2025-08-14T21:52:19.7225861Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7226058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7226123Z return mod(**inputs) 2025-08-14T21:52:19.7226410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7226486Z outputs = self.mobilebert( 2025-08-14T21:52:19.7226749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7226819Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7227095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7227166Z layer_outputs = layer_module( 2025-08-14T21:52:19.7227438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7227523Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7227800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 469, in forward 2025-08-14T21:52:19.7227934Z intermediate_output = self.intermediate(hidden_states) 2025-08-14T21:52:19.7228202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.7228312Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.7228316Z 2025-08-14T21:52:19.7228413Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7228601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7228674Z return mod(**inputs) 2025-08-14T21:52:19.7228942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7229011Z outputs = self.mobilebert( 2025-08-14T21:52:19.7229287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7229360Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7229633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7229700Z layer_outputs = layer_module( 2025-08-14T21:52:19.7229964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7230059Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7230322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.7230447Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.7230713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 457, in forward 2025-08-14T21:52:19.7230796Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.7230801Z 2025-08-14T21:52:19.7230905Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7231093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7231163Z return mod(**inputs) 2025-08-14T21:52:19.7231429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7231496Z outputs = self.mobilebert( 2025-08-14T21:52:19.7231767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7231873Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7232140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7232237Z layer_outputs = layer_module( 2025-08-14T21:52:19.7232504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 515, in forward 2025-08-14T21:52:19.7232597Z attention_output = ffn_module(attention_output) 2025-08-14T21:52:19.7232863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 470, in forward 2025-08-14T21:52:19.7232979Z layer_outputs = self.output(intermediate_output, hidden_states) 2025-08-14T21:52:19.7233249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 458, in forward 2025-08-14T21:52:19.7233367Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.7233662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7233755Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7233759Z 2025-08-14T21:52:19.7233873Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7234072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7234139Z return mod(**inputs) 2025-08-14T21:52:19.7234417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7234493Z outputs = self.mobilebert( 2025-08-14T21:52:19.7234778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7234858Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7235127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7235199Z layer_outputs = layer_module( 2025-08-14T21:52:19.7235480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.7235596Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.7235869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 359, in forward 2025-08-14T21:52:19.7235951Z hidden_states = self.dense(hidden_states) 2025-08-14T21:52:19.7235954Z 2025-08-14T21:52:19.7236051Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7236251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7236315Z return mod(**inputs) 2025-08-14T21:52:19.7236582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7236658Z outputs = self.mobilebert( 2025-08-14T21:52:19.7236927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7237002Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7237268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7237335Z layer_outputs = layer_module( 2025-08-14T21:52:19.7237722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 518, in forward 2025-08-14T21:52:19.7237848Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:52:19.7238171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 360, in forward 2025-08-14T21:52:19.7238278Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:52:19.7238283Z 2025-08-14T21:52:19.7238408Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7238611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7238675Z return mod(**inputs) 2025-08-14T21:52:19.7238944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7239018Z outputs = self.mobilebert( 2025-08-14T21:52:19.7239286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7239364Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7239634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7239702Z layer_outputs = layer_module( 2025-08-14T21:52:19.7240000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.7240182Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.7240454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 392, in forward 2025-08-14T21:52:19.7240543Z layer_output = self.dense(intermediate_states) 2025-08-14T21:52:19.7240547Z 2025-08-14T21:52:19.7240643Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7240839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7240904Z return mod(**inputs) 2025-08-14T21:52:19.7241172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7241247Z outputs = self.mobilebert( 2025-08-14T21:52:19.7241513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7241593Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7241855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7241924Z layer_outputs = layer_module( 2025-08-14T21:52:19.7242196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.7242347Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.7242628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 397, in forward 2025-08-14T21:52:19.7242746Z layer_output = self.LayerNorm(layer_output + residual_tensor_1) 2025-08-14T21:52:19.7243017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7243119Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7243123Z 2025-08-14T21:52:19.7243224Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7243424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7243488Z return mod(**inputs) 2025-08-14T21:52:19.7243763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7243837Z outputs = self.mobilebert( 2025-08-14T21:52:19.7244132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7244203Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7244485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7244572Z layer_outputs = layer_module( 2025-08-14T21:52:19.7244854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.7245007Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.7245278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.7245450Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.7245761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 372, in forward 2025-08-14T21:52:19.7245862Z layer_outputs = self.dense(hidden_states) 2025-08-14T21:52:19.7245866Z 2025-08-14T21:52:19.7245974Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7246205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7246301Z return mod(**inputs) 2025-08-14T21:52:19.7246602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1242, in forward 2025-08-14T21:52:19.7246677Z outputs = self.mobilebert( 2025-08-14T21:52:19.7246980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 794, in forward 2025-08-14T21:52:19.7247051Z encoder_outputs = self.encoder( 2025-08-14T21:52:19.7247333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 557, in forward 2025-08-14T21:52:19.7247405Z layer_outputs = layer_module( 2025-08-14T21:52:19.7247674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 519, in forward 2025-08-14T21:52:19.7247836Z layer_output = self.output(intermediate_output, attention_output, hidden_states) 2025-08-14T21:52:19.7248110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 398, in forward 2025-08-14T21:52:19.7248236Z layer_output = self.bottleneck(layer_output, residual_tensor_2) 2025-08-14T21:52:19.7248512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 374, in forward 2025-08-14T21:52:19.7248630Z layer_outputs = self.LayerNorm(layer_outputs + residual_tensor) 2025-08-14T21:52:19.7248913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 138, in forward 2025-08-14T21:52:19.7249005Z return input_tensor * self.weight + self.bias 2025-08-14T21:52:19.7249009Z 2025-08-14T21:52:19.7249114Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7249309Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7249375Z return mod(**inputs) 2025-08-14T21:52:19.7249659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1256, in forward 2025-08-14T21:52:19.7249744Z logits = self.qa_outputs(sequence_output) 2025-08-14T21:52:19.7249747Z 2025-08-14T21:52:19.7249846Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7250046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7250110Z return mod(**inputs) 2025-08-14T21:52:19.7250438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1274, in forward 2025-08-14T21:52:19.7250543Z start_loss = loss_fct(start_logits, start_positions) 2025-08-14T21:52:19.7250546Z 2025-08-14T21:52:19.7250648Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:19.7250867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:19.7250933Z return mod(**inputs) 2025-08-14T21:52:19.7251215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/mobilebert/modeling_mobilebert.py", line 1275, in forward 2025-08-14T21:52:19.7251307Z end_loss = loss_fct(end_logits, end_positions) 2025-08-14T21:52:19.7251310Z 2025-08-14T21:52:31.5124991Z Compilation time (from dynamo_timed): 36.717849517 2025-08-14T21:52:31.5125572Z pass 2025-08-14T21:52:31.5125884Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:52:31.5127049Z TIMING: _recursive_pre_grad_passes:0.02245 _recursive_joint_graph_passes:1.31854 _recursive_post_grad_passes:0.22438 async_compile.wait:0.34012 code_gen:8.78846 inductor_compile:13.21676 backend_compile:25.38711 gc:0.00099 entire_frame_compile:36.71785 total_wall_time:36.71785 2025-08-14T21:52:31.5128109Z STATS: call_* op count: 1453 | FakeTensorMode.__torch_dispatch__:56761 | FakeTensor.__torch_dispatch__:16441 | ProxyTorchDispatchMode.__torch_dispatch__:21655 2025-08-14T21:52:31.5128600Z Dynamo produced 1 graphs covering 1453 ops with 0 graph breaks (0 unique) 2025-08-14T21:52:37.6491377Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:52:37.6493372Z from pkg_resources import resource_filename 2025-08-14T21:52:38.2199572Z 2025-08-14T21:52:40.0806302Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:52:40.0809885Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:52:40.0816645Z cpu eval OPTForCausalLM 2025-08-14T21:52:41.7172046Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:52:42.5297226Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:52:43.3274325Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:52:50.6244083Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6244461Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6244683Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6244883Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6245090Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6245294Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6245654Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6245858Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6246066Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6246274Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6246478Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6246696Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6246959Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6247404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6247784Z return mod(**inputs) 2025-08-14T21:52:50.6248193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6248599Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6249028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6249795Z outputs = self.model.decoder( 2025-08-14T21:52:50.6250170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6250545Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6250989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6251479Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6251873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6252238Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6252617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6253018Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6253420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:52:50.6253830Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:52:50.6254008Z 2025-08-14T21:52:50.6254128Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6254569Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6254910Z return mod(**inputs) 2025-08-14T21:52:50.6255378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6255745Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6256125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6256504Z outputs = self.model.decoder( 2025-08-14T21:52:50.6256864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6257248Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6257618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6257997Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6258349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6258749Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6259122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6259526Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6259938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:52:50.6260324Z key_states = self.k_proj(hidden_states) 2025-08-14T21:52:50.6260460Z 2025-08-14T21:52:50.6260574Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6260925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6261240Z return mod(**inputs) 2025-08-14T21:52:50.6261554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6261898Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6262259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6262625Z outputs = self.model.decoder( 2025-08-14T21:52:50.6262954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6263292Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6263653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6264047Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6264384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6264737Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6265097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6265507Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6265889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:52:50.6266274Z value_states = self.v_proj(hidden_states) 2025-08-14T21:52:50.6266412Z 2025-08-14T21:52:50.6266500Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6266704Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6266910Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6267110Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6267335Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6267692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6268027Z return mod(**inputs) 2025-08-14T21:52:50.6268359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6268729Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6269090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6269458Z outputs = self.model.decoder( 2025-08-14T21:52:50.6269782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6270121Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6270476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6270836Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6271167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6271515Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6271885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6272264Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6272646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6273034Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6273465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:52:50.6273932Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:52:50.6274123Z 2025-08-14T21:52:50.6274225Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6274570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6274883Z return mod(**inputs) 2025-08-14T21:52:50.6275194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6275535Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6275904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6276248Z outputs = self.model.decoder( 2025-08-14T21:52:50.6276571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6276901Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6277250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6277620Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6277943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6278282Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6278647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6279023Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6279394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6279800Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6280223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:52:50.6280672Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:52:50.6280832Z 2025-08-14T21:52:50.6280938Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6281299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6281628Z return mod(**inputs) 2025-08-14T21:52:50.6282011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6282358Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6282732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6283106Z outputs = self.model.decoder( 2025-08-14T21:52:50.6283448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6283795Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6284169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6284544Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6284885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6285269Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6285768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6286195Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6286603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:52:50.6287017Z attn_output = self.out_proj(attn_output) 2025-08-14T21:52:50.6287147Z 2025-08-14T21:52:50.6287257Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6287606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6287913Z return mod(**inputs) 2025-08-14T21:52:50.6288239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6288579Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6288938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6289310Z outputs = self.model.decoder( 2025-08-14T21:52:50.6289649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6289997Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6290360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6290726Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6291093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6291438Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6291809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:52:50.6292205Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:52:50.6292336Z 2025-08-14T21:52:50.6292447Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6292789Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6293119Z return mod(**inputs) 2025-08-14T21:52:50.6293447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6293792Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6294170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6294560Z outputs = self.model.decoder( 2025-08-14T21:52:50.6294903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6295279Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6295673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6296083Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6296452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6296825Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6297215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:52:50.6297618Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:52:50.6297770Z 2025-08-14T21:52:50.6297872Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6298238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6298562Z return mod(**inputs) 2025-08-14T21:52:50.6298889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6299238Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6299615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6299991Z outputs = self.model.decoder( 2025-08-14T21:52:50.6300324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6300698Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6301066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6301444Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6301788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6302148Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6302522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:52:50.6302907Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:52:50.6303042Z 2025-08-14T21:52:50.6303143Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6303498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6303823Z return mod(**inputs) 2025-08-14T21:52:50.6304130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6304512Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6304930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6305334Z outputs = self.model.decoder( 2025-08-14T21:52:50.6305694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6306087Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6306448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6306801Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6307138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6307496Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6307866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6308256Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6308647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:52:50.6309079Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:52:50.6309252Z 2025-08-14T21:52:50.6309363Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6309725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6310045Z return mod(**inputs) 2025-08-14T21:52:50.6310364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6310696Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6311065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6311437Z outputs = self.model.decoder( 2025-08-14T21:52:50.6311777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6312125Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6312494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6312868Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6313205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6313571Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6313934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6314319Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6314701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:52:50.6315081Z key_states = self.k_proj(hidden_states) 2025-08-14T21:52:50.6315214Z 2025-08-14T21:52:50.6315323Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6315680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6315997Z return mod(**inputs) 2025-08-14T21:52:50.6316323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6316674Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6317033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6317406Z outputs = self.model.decoder( 2025-08-14T21:52:50.6317747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6318099Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6318487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6318855Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6319196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6319576Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6319957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6320366Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6320763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:52:50.6321141Z value_states = self.v_proj(hidden_states) 2025-08-14T21:52:50.6321288Z 2025-08-14T21:52:50.6321369Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6321585Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6321788Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6321994Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6322232Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6322617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6322937Z return mod(**inputs) 2025-08-14T21:52:50.6323289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6323655Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6324019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6324381Z outputs = self.model.decoder( 2025-08-14T21:52:50.6324722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6325068Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6325564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6325998Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6326386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6326762Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6327132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6327523Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6327909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6328288Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6328724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:52:50.6329188Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:52:50.6329362Z 2025-08-14T21:52:50.6329467Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6329800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6330109Z return mod(**inputs) 2025-08-14T21:52:50.6330422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6330760Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6331130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6331485Z outputs = self.model.decoder( 2025-08-14T21:52:50.6331808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6332158Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6332506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6332857Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6333179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6333546Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6333901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6334281Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6334646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6335023Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6335443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:52:50.6335890Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:52:50.6336049Z 2025-08-14T21:52:50.6336189Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6336541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6336890Z return mod(**inputs) 2025-08-14T21:52:50.6337192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6337527Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6338104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6338461Z outputs = self.model.decoder( 2025-08-14T21:52:50.6338783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6339118Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6339469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6339817Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6340154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6340498Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6340857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6341227Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6341597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:52:50.6341974Z attn_output = self.out_proj(attn_output) 2025-08-14T21:52:50.6342104Z 2025-08-14T21:52:50.6342205Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6342534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6342840Z return mod(**inputs) 2025-08-14T21:52:50.6343147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6343472Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6343824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6344176Z outputs = self.model.decoder( 2025-08-14T21:52:50.6344497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6344817Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6345166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6345580Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6345899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6346242Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6346637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:52:50.6347005Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:52:50.6347139Z 2025-08-14T21:52:50.6347240Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6347587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6347905Z return mod(**inputs) 2025-08-14T21:52:50.6348220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6348570Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6348940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6349339Z outputs = self.model.decoder( 2025-08-14T21:52:50.6349701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6350035Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6350412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6350780Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6351109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6351458Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6351821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:52:50.6352202Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:52:50.6352359Z 2025-08-14T21:52:50.6352458Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6352805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6353120Z return mod(**inputs) 2025-08-14T21:52:50.6353431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6353777Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6354141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6354499Z outputs = self.model.decoder( 2025-08-14T21:52:50.6354833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6355178Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6355546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6355901Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6356238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6356591Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6356952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:52:50.6357325Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:52:50.6357461Z 2025-08-14T21:52:50.6357560Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6357904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6358213Z return mod(**inputs) 2025-08-14T21:52:50.6358527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6358889Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6359252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6359609Z outputs = self.model.decoder( 2025-08-14T21:52:50.6359960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6360300Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6360654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6361018Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6361355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6361708Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6362071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6362447Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6362835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:52:50.6363220Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:52:50.6363400Z 2025-08-14T21:52:50.6363501Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6363837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6364140Z return mod(**inputs) 2025-08-14T21:52:50.6364450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6364791Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6365154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6365584Z outputs = self.model.decoder( 2025-08-14T21:52:50.6366209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6366587Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6366974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6367332Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6367680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6368024Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6368379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6368752Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6369127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:52:50.6369489Z key_states = self.k_proj(hidden_states) 2025-08-14T21:52:50.6369616Z 2025-08-14T21:52:50.6369722Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6370055Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6370366Z return mod(**inputs) 2025-08-14T21:52:50.6370672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6370998Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6371351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6371710Z outputs = self.model.decoder( 2025-08-14T21:52:50.6372036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6372391Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6372744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6373099Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6373444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6373784Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6374139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6374513Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6374880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:52:50.6375267Z value_states = self.v_proj(hidden_states) 2025-08-14T21:52:50.6375399Z 2025-08-14T21:52:50.6375486Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6375686Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6375888Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6376088Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6376329Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6377555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6377913Z return mod(**inputs) 2025-08-14T21:52:50.6378225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6378552Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6378903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6379259Z outputs = self.model.decoder( 2025-08-14T21:52:50.6379586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6379916Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6380269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6380620Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6380943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6381279Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6381634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6382010Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6382376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6382751Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6383168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:52:50.6383621Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:52:50.6383792Z 2025-08-14T21:52:50.6383894Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6384238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6384547Z return mod(**inputs) 2025-08-14T21:52:50.6384849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6385183Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6385537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6385893Z outputs = self.model.decoder( 2025-08-14T21:52:50.6386244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6386574Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6386927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6387318Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6387655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6388000Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6388363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6388742Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6389123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6389499Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6389913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:52:50.6390362Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:52:50.6390524Z 2025-08-14T21:52:50.6390625Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6390980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6391280Z return mod(**inputs) 2025-08-14T21:52:50.6391584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6391917Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6392267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6392616Z outputs = self.model.decoder( 2025-08-14T21:52:50.6392942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6393275Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6393622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6393981Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6394313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6394665Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6395001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6395367Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6395726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:52:50.6396079Z attn_output = self.out_proj(attn_output) 2025-08-14T21:52:50.6396202Z 2025-08-14T21:52:50.6396295Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6396637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6396944Z return mod(**inputs) 2025-08-14T21:52:50.6397244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6397574Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6397920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6398283Z outputs = self.model.decoder( 2025-08-14T21:52:50.6398591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6398909Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6399274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6399609Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6399929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6400282Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6400627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:52:50.6400976Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:52:50.6401109Z 2025-08-14T21:52:50.6401202Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6401536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6401827Z return mod(**inputs) 2025-08-14T21:52:50.6402129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6402457Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6402817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6403159Z outputs = self.model.decoder( 2025-08-14T21:52:50.6403492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6403827Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6404182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6404550Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6404894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6405250Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6405684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:52:50.6406069Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:52:50.6406222Z 2025-08-14T21:52:50.6406354Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6406719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6407047Z return mod(**inputs) 2025-08-14T21:52:50.6407368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6407721Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6408067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6408423Z outputs = self.model.decoder( 2025-08-14T21:52:50.6408758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6409083Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6409418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6409764Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6410087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6410413Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6410759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:52:50.6411112Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:52:50.6411235Z 2025-08-14T21:52:50.6411337Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6411659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6412014Z return mod(**inputs) 2025-08-14T21:52:50.6412314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6412643Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6412982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6413348Z outputs = self.model.decoder( 2025-08-14T21:52:50.6413666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6413983Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6414328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6414683Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6415009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6415342Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6415693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-08-14T21:52:50.6416127Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-08-14T21:52:50.6416304Z 2025-08-14T21:52:50.6416400Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6416764Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6417083Z return mod(**inputs) 2025-08-14T21:52:50.6417397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6417733Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6418091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6418458Z outputs = self.model.decoder( 2025-08-14T21:52:50.6418782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6419125Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6419488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6419854Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6420187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6420539Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6420902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6421290Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6421664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:52:50.6422065Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:52:50.6422222Z 2025-08-14T21:52:50.6422330Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6422671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6422985Z return mod(**inputs) 2025-08-14T21:52:50.6423299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6423646Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6423987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6424346Z outputs = self.model.decoder( 2025-08-14T21:52:50.6424680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6425040Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6425397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6425765Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6426106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6426470Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6426829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6427209Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6427585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:52:50.6427939Z key_states = self.k_proj(hidden_states) 2025-08-14T21:52:50.6428072Z 2025-08-14T21:52:50.6428169Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6428511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6428812Z return mod(**inputs) 2025-08-14T21:52:50.6429140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6429477Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6429843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6430192Z outputs = self.model.decoder( 2025-08-14T21:52:50.6430514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6430850Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6431196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6431560Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6431892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6432250Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6432626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6433013Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6433418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:52:50.6433807Z value_states = self.v_proj(hidden_states) 2025-08-14T21:52:50.6433940Z 2025-08-14T21:52:50.6434021Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6434230Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6434433Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6434626Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6434855Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6435202Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6435522Z return mod(**inputs) 2025-08-14T21:52:50.6435834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6436169Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6436526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6436876Z outputs = self.model.decoder( 2025-08-14T21:52:50.6437198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6437532Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6438019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6438454Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6438778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6439120Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6439478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6439904Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6440302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6440681Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6441091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:52:50.6441547Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:52:50.6441719Z 2025-08-14T21:52:50.6441825Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6442159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6442466Z return mod(**inputs) 2025-08-14T21:52:50.6442800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6443169Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6443520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6443880Z outputs = self.model.decoder( 2025-08-14T21:52:50.6444210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6444549Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6444914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6445283Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6445679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6446035Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6446429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6446846Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6447241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6447609Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6448026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:52:50.6448461Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:52:50.6448614Z 2025-08-14T21:52:50.6448713Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6449058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6449371Z return mod(**inputs) 2025-08-14T21:52:50.6449681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6450008Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6450359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6450714Z outputs = self.model.decoder( 2025-08-14T21:52:50.6451038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6451362Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6451702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6452073Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6452390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6452729Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6453105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6453483Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6453851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:52:50.6454222Z attn_output = self.out_proj(attn_output) 2025-08-14T21:52:50.6454354Z 2025-08-14T21:52:50.6454463Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6454803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6455121Z return mod(**inputs) 2025-08-14T21:52:50.6455440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6455799Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6456167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6456520Z outputs = self.model.decoder( 2025-08-14T21:52:50.6456847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6457173Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6457525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6457879Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6458202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6458536Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6458897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:52:50.6459250Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:52:50.6459378Z 2025-08-14T21:52:50.6459481Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6459803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6460098Z return mod(**inputs) 2025-08-14T21:52:50.6460395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6460708Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6461049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6461393Z outputs = self.model.decoder( 2025-08-14T21:52:50.6461706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6462024Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6462373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6462734Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6463069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6463408Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6463763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:52:50.6464138Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:52:50.6464282Z 2025-08-14T21:52:50.6464403Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6464740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6465051Z return mod(**inputs) 2025-08-14T21:52:50.6465360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6465723Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6466084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6466455Z outputs = self.model.decoder( 2025-08-14T21:52:50.6466769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6467101Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6467454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6467812Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6468134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6468477Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6468851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:52:50.6469226Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:52:50.6469364Z 2025-08-14T21:52:50.6469464Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6469810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6470133Z return mod(**inputs) 2025-08-14T21:52:50.6470439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6470779Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6471138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6471497Z outputs = self.model.decoder( 2025-08-14T21:52:50.6471829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6472171Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6472534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6472894Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6473230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6473583Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6473939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6474327Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6474711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:52:50.6475115Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:52:50.6475276Z 2025-08-14T21:52:50.6475377Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6475728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6476043Z return mod(**inputs) 2025-08-14T21:52:50.6476360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6476697Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6477059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6477422Z outputs = self.model.decoder( 2025-08-14T21:52:50.6477765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6478098Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6478455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6478829Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6499064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6499469Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6499866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6500260Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6500652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:52:50.6501027Z key_states = self.k_proj(hidden_states) 2025-08-14T21:52:50.6501163Z 2025-08-14T21:52:50.6501276Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6501622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6502037Z return mod(**inputs) 2025-08-14T21:52:50.6502439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6502781Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6503143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6503506Z outputs = self.model.decoder( 2025-08-14T21:52:50.6503841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6504170Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6504538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6504916Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6505260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6505635Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6506003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6506390Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6506766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:52:50.6507137Z value_states = self.v_proj(hidden_states) 2025-08-14T21:52:50.6507268Z 2025-08-14T21:52:50.6507356Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6507558Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6507748Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6507941Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6508162Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6508501Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6508820Z return mod(**inputs) 2025-08-14T21:52:50.6509136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6509472Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6509827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6510185Z outputs = self.model.decoder( 2025-08-14T21:52:50.6510504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6510837Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6511228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6511577Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6511911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6512290Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6512655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6513035Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6513421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6513809Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6514234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:52:50.6514707Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:52:50.6514887Z 2025-08-14T21:52:50.6514987Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6515346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6515654Z return mod(**inputs) 2025-08-14T21:52:50.6515984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6516322Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6516702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6517079Z outputs = self.model.decoder( 2025-08-14T21:52:50.6517415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6517782Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6518153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6518533Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6518868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6519219Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6519599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6519993Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6520368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6520739Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6521158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:52:50.6521587Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:52:50.6521740Z 2025-08-14T21:52:50.6521845Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6522182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6522496Z return mod(**inputs) 2025-08-14T21:52:50.6522804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6523137Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6523483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6523839Z outputs = self.model.decoder( 2025-08-14T21:52:50.6524172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6524537Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6524900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6525277Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6525720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6526096Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6526487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6526932Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6527351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:52:50.6527767Z attn_output = self.out_proj(attn_output) 2025-08-14T21:52:50.6527912Z 2025-08-14T21:52:50.6528020Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6528387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6528698Z return mod(**inputs) 2025-08-14T21:52:50.6529039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6529377Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6529750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6530102Z outputs = self.model.decoder( 2025-08-14T21:52:50.6530426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6530756Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6531098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6531451Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6531777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6532111Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6532458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:52:50.6532820Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:52:50.6532949Z 2025-08-14T21:52:50.6533054Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6533386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6533695Z return mod(**inputs) 2025-08-14T21:52:50.6534003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6534337Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6534691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6535055Z outputs = self.model.decoder( 2025-08-14T21:52:50.6535386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6535722Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6536084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6536447Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6536785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6537125Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6537489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:52:50.6538055Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:52:50.6538206Z 2025-08-14T21:52:50.6538316Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6538663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6538995Z return mod(**inputs) 2025-08-14T21:52:50.6539402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6539749Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6540127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6540493Z outputs = self.model.decoder( 2025-08-14T21:52:50.6540826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6541161Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6541526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6541891Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6542250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6542609Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6543057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:52:50.6543437Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:52:50.6543570Z 2025-08-14T21:52:50.6543670Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6544023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6544345Z return mod(**inputs) 2025-08-14T21:52:50.6544665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6545020Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6545438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6545804Z outputs = self.model.decoder( 2025-08-14T21:52:50.6546130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6546473Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6546833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6547198Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6547529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6547873Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6548237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-08-14T21:52:50.6548651Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-08-14T21:52:50.6548837Z 2025-08-14T21:52:50.6548938Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6549289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6549607Z return mod(**inputs) 2025-08-14T21:52:50.6549916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6550261Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6550622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6550977Z outputs = self.model.decoder( 2025-08-14T21:52:50.6551303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6551676Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6552035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6552391Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6552734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6553107Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6553473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6553856Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6554248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:52:50.6554637Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:52:50.6554792Z 2025-08-14T21:52:50.6554889Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6555225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6555534Z return mod(**inputs) 2025-08-14T21:52:50.6555879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6556225Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6556578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6556926Z outputs = self.model.decoder( 2025-08-14T21:52:50.6557235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6557566Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6557912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6558262Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6558579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6558921Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6559284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6559674Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6560047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:52:50.6560416Z key_states = self.k_proj(hidden_states) 2025-08-14T21:52:50.6560544Z 2025-08-14T21:52:50.6560650Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6560987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6561304Z return mod(**inputs) 2025-08-14T21:52:50.6561617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6561953Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6562306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6562668Z outputs = self.model.decoder( 2025-08-14T21:52:50.6562994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6563323Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6563679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6564036Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6564368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6564733Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6565095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6565570Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6566033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:52:50.6566457Z value_states = self.v_proj(hidden_states) 2025-08-14T21:52:50.6566615Z 2025-08-14T21:52:50.6566714Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6566930Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6567132Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6567342Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6567576Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6567931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6568273Z return mod(**inputs) 2025-08-14T21:52:50.6568592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6568938Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6569312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6569700Z outputs = self.model.decoder( 2025-08-14T21:52:50.6570040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6570375Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6570740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6571109Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6571447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6571793Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6572173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6572558Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6572935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6573308Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6573733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:52:50.6574199Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:52:50.6574377Z 2025-08-14T21:52:50.6574476Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6574831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6575150Z return mod(**inputs) 2025-08-14T21:52:50.6575469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6575812Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6576181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6576540Z outputs = self.model.decoder( 2025-08-14T21:52:50.6576865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6577197Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6577547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6577904Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6578247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6578587Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6578946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6579349Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6579718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6580092Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6580507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:52:50.6580942Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:52:50.6581093Z 2025-08-14T21:52:50.6581190Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6581534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6581843Z return mod(**inputs) 2025-08-14T21:52:50.6582169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6582511Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6582880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6583242Z outputs = self.model.decoder( 2025-08-14T21:52:50.6583561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6583898Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6584254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6584604Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6584937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6585284Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6585642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6586027Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6586415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:52:50.6586798Z attn_output = self.out_proj(attn_output) 2025-08-14T21:52:50.6586927Z 2025-08-14T21:52:50.6587032Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6587365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6587677Z return mod(**inputs) 2025-08-14T21:52:50.6587986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6588315Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6588675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6589038Z outputs = self.model.decoder( 2025-08-14T21:52:50.6589367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6589693Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6590042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6590401Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6590728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6591071Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6591452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:52:50.6591813Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:52:50.6591940Z 2025-08-14T21:52:50.6592042Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6592400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6592707Z return mod(**inputs) 2025-08-14T21:52:50.6593003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6593333Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6593684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6594036Z outputs = self.model.decoder( 2025-08-14T21:52:50.6594349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6594683Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6595031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6595394Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6595739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6596082Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6596444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:52:50.6596822Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:52:50.6596976Z 2025-08-14T21:52:50.6597077Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6597442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6597751Z return mod(**inputs) 2025-08-14T21:52:50.6598052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6598383Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6598737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6599096Z outputs = self.model.decoder( 2025-08-14T21:52:50.6599425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6599763Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6600129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6600473Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6600797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6601135Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6601481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:52:50.6601836Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:52:50.6601973Z 2025-08-14T21:52:50.6602070Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6602412Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6602711Z return mod(**inputs) 2025-08-14T21:52:50.6603020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6603354Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6603701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6604063Z outputs = self.model.decoder( 2025-08-14T21:52:50.6604413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6604754Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6605106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6605589Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6605979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6606378Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6606767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6607190Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6607606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:52:50.6608011Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:52:50.6608175Z 2025-08-14T21:52:50.6608273Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6608654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6608962Z return mod(**inputs) 2025-08-14T21:52:50.6609280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6609620Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6609987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6610348Z outputs = self.model.decoder( 2025-08-14T21:52:50.6610685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6611030Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6611399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6611760Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6612105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6612467Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6612842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6613231Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6613629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:52:50.6614019Z key_states = self.k_proj(hidden_states) 2025-08-14T21:52:50.6614152Z 2025-08-14T21:52:50.6614256Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6614615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6614939Z return mod(**inputs) 2025-08-14T21:52:50.6615262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6615603Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6615972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6616341Z outputs = self.model.decoder( 2025-08-14T21:52:50.6616672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6617024Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6617389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6617760Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6618108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6618451Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6618812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6619208Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6619589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:52:50.6619963Z value_states = self.v_proj(hidden_states) 2025-08-14T21:52:50.6620097Z 2025-08-14T21:52:50.6620182Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6620382Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6620464Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6620537Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6620639Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6620843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6620907Z return mod(**inputs) 2025-08-14T21:52:50.6621140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6621216Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6621461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6621543Z outputs = self.model.decoder( 2025-08-14T21:52:50.6621751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6621822Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6622060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6622132Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6622347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6622423Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6622655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6622761Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6622989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6623091Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6623369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:52:50.6623499Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:52:50.6623504Z 2025-08-14T21:52:50.6623609Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6623801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6623864Z return mod(**inputs) 2025-08-14T21:52:50.6624082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6624158Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6624395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6624467Z outputs = self.model.decoder( 2025-08-14T21:52:50.6624674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6624755Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6624985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6625082Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6625296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6625375Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6625625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6625723Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6625954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6626061Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6626339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:52:50.6626453Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:52:50.6626458Z 2025-08-14T21:52:50.6626560Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6626756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6626850Z return mod(**inputs) 2025-08-14T21:52:50.6627058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6627156Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6627390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6627462Z outputs = self.model.decoder( 2025-08-14T21:52:50.6627677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6627747Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6627996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6628068Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6628286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6628362Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6628597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6628699Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6628931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:52:50.6629013Z attn_output = self.out_proj(attn_output) 2025-08-14T21:52:50.6629026Z 2025-08-14T21:52:50.6629126Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6629321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6629395Z return mod(**inputs) 2025-08-14T21:52:50.6629604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6629675Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6629912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6629985Z outputs = self.model.decoder( 2025-08-14T21:52:50.6630201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6630272Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6630511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6630587Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6630795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6630918Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6631150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:52:50.6631227Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:52:50.6631247Z 2025-08-14T21:52:50.6631351Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6631545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6631611Z return mod(**inputs) 2025-08-14T21:52:50.6631828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6631899Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6632129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6632209Z outputs = self.model.decoder( 2025-08-14T21:52:50.6632417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6632495Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6632744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6632818Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6633052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6633129Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6633374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:52:50.6633470Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:52:50.6633474Z 2025-08-14T21:52:50.6633583Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6633782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6633849Z return mod(**inputs) 2025-08-14T21:52:50.6634056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6634139Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6634367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6634446Z outputs = self.model.decoder( 2025-08-14T21:52:50.6634649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6634720Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6634952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6635024Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6635238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6635329Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6635561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:52:50.6635652Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:52:50.6635657Z 2025-08-14T21:52:50.6635759Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6635959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6636034Z return mod(**inputs) 2025-08-14T21:52:50.6636238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6636319Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6636547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6636642Z outputs = self.model.decoder( 2025-08-14T21:52:50.6636857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6636928Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6637170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6637248Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6637452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6637533Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6637934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-08-14T21:52:50.6638067Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-08-14T21:52:50.6638073Z 2025-08-14T21:52:50.6638178Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6638367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6638490Z return mod(**inputs) 2025-08-14T21:52:50.6638701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6638800Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6639034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6639104Z outputs = self.model.decoder( 2025-08-14T21:52:50.6639304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6639381Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6639605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6639686Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6639895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6639974Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6640220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6640318Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6640548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:52:50.6640666Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:52:50.6640670Z 2025-08-14T21:52:50.6640769Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6640972Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6641037Z return mod(**inputs) 2025-08-14T21:52:50.6641245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6641326Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6641558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6641637Z outputs = self.model.decoder( 2025-08-14T21:52:50.6641844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6641917Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6642172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6642241Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6642450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6642565Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6642799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6642904Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6643172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:52:50.6643248Z key_states = self.k_proj(hidden_states) 2025-08-14T21:52:50.6643252Z 2025-08-14T21:52:50.6643357Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6643550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6643615Z return mod(**inputs) 2025-08-14T21:52:50.6643828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6643901Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6644144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6644219Z outputs = self.model.decoder( 2025-08-14T21:52:50.6644453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6644557Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6644792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6644871Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6645084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6645162Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6645453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6645562Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6645800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:52:50.6645897Z value_states = self.v_proj(hidden_states) 2025-08-14T21:52:50.6645902Z 2025-08-14T21:52:50.6645983Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6646072Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6646149Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6646227Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6646336Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6646537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6646603Z return mod(**inputs) 2025-08-14T21:52:50.6646891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6646965Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6647207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6647281Z outputs = self.model.decoder( 2025-08-14T21:52:50.6647492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6647574Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6647814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6647887Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6648112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6648187Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6648461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6648557Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6648797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6648926Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6649217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:52:50.6649359Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:52:50.6649363Z 2025-08-14T21:52:50.6649465Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6649661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6649737Z return mod(**inputs) 2025-08-14T21:52:50.6649953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6650025Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6650303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6650378Z outputs = self.model.decoder( 2025-08-14T21:52:50.6650618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6650692Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6650934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6651014Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6651235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6651312Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6651562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6651657Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6651901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6651999Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6652285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:52:50.6652405Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:52:50.6652409Z 2025-08-14T21:52:50.6652511Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6652719Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6652786Z return mod(**inputs) 2025-08-14T21:52:50.6653000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6653085Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6653322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6653397Z outputs = self.model.decoder( 2025-08-14T21:52:50.6653622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6653696Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6653974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6654047Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6654266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6654378Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6654617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6654722Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6654958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:52:50.6655066Z attn_output = self.out_proj(attn_output) 2025-08-14T21:52:50.6655069Z 2025-08-14T21:52:50.6655180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6655378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6655443Z return mod(**inputs) 2025-08-14T21:52:50.6655663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6655737Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6655985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6656061Z outputs = self.model.decoder( 2025-08-14T21:52:50.6656296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6656381Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6656650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6656723Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6656952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6657030Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6657278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:52:50.6657360Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:52:50.6657364Z 2025-08-14T21:52:50.6657465Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6657673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6657740Z return mod(**inputs) 2025-08-14T21:52:50.6657967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6658041Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6658278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6658358Z outputs = self.model.decoder( 2025-08-14T21:52:50.6658573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6658646Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6658893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6658973Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6659197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6659276Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6659514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:52:50.6659617Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:52:50.6659621Z 2025-08-14T21:52:50.6659719Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6659919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6659982Z return mod(**inputs) 2025-08-14T21:52:50.6660190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6660290Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6660523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6660595Z outputs = self.model.decoder( 2025-08-14T21:52:50.6660830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6660902Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6661136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6661205Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6661414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6661496Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6661743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:52:50.6661822Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:52:50.6661826Z 2025-08-14T21:52:50.6661931Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6662139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6662212Z return mod(**inputs) 2025-08-14T21:52:50.6662436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6662509Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6662752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6662821Z outputs = self.model.decoder( 2025-08-14T21:52:50.6663029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6663110Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6663342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6663419Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6663631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6663708Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6663948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6664041Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6664281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:52:50.6664396Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:52:50.6664400Z 2025-08-14T21:52:50.6664508Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6664734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6664804Z return mod(**inputs) 2025-08-14T21:52:50.6665039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6665124Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6665366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6665446Z outputs = self.model.decoder( 2025-08-14T21:52:50.6665659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6665734Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6665981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6666075Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6666303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6666389Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6666624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6666747Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6666980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:52:50.6667058Z key_states = self.k_proj(hidden_states) 2025-08-14T21:52:50.6667061Z 2025-08-14T21:52:50.6667169Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6667360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6667432Z return mod(**inputs) 2025-08-14T21:52:50.6667642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6667713Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6667986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6668061Z outputs = self.model.decoder( 2025-08-14T21:52:50.6668287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6668368Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6668617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6668698Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6668911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6668991Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6669240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6669336Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6669591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:52:50.6669678Z value_states = self.v_proj(hidden_states) 2025-08-14T21:52:50.6669682Z 2025-08-14T21:52:50.6669762Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6669847Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6669923Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6669999Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6670106Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6670296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6670365Z return mod(**inputs) 2025-08-14T21:52:50.6670582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6670655Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6670930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6671004Z outputs = self.model.decoder( 2025-08-14T21:52:50.6671207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6671286Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6671512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6671593Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6671802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6671906Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6672134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6672226Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6672469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6672567Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6672838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:52:50.6672971Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:52:50.6672974Z 2025-08-14T21:52:50.6673069Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6673254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6673325Z return mod(**inputs) 2025-08-14T21:52:50.6673526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6673620Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6673851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6673937Z outputs = self.model.decoder( 2025-08-14T21:52:50.6674153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6674223Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6674448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6674524Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6674732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6674816Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6675056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6675146Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6675380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6675471Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6675739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:52:50.6675849Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:52:50.6675852Z 2025-08-14T21:52:50.6675947Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6676144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6676205Z return mod(**inputs) 2025-08-14T21:52:50.6676405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6676483Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6676707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6676780Z outputs = self.model.decoder( 2025-08-14T21:52:50.6676981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6677049Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6677279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6677347Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6677574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6677655Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6677879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6677996Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6678217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:52:50.6678294Z attn_output = self.out_proj(attn_output) 2025-08-14T21:52:50.6678297Z 2025-08-14T21:52:50.6678398Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6678584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6678653Z return mod(**inputs) 2025-08-14T21:52:50.6678855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6678927Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6679154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6679253Z outputs = self.model.decoder( 2025-08-14T21:52:50.6679475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6679555Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6679777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6679852Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6680057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6680131Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6680361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:52:50.6680436Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:52:50.6680439Z 2025-08-14T21:52:50.6680534Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6680728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6680793Z return mod(**inputs) 2025-08-14T21:52:50.6680998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6681067Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6681288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6681366Z outputs = self.model.decoder( 2025-08-14T21:52:50.6681565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6681645Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6681867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6681935Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6682147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6682223Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6682444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:52:50.6682543Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:52:50.6682546Z 2025-08-14T21:52:50.6682643Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6682843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6682931Z return mod(**inputs) 2025-08-14T21:52:50.6683139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6683220Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6683452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6683551Z outputs = self.model.decoder( 2025-08-14T21:52:50.6683766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6683836Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6684074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6684145Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6684361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6684447Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6684733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:52:50.6684816Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:52:50.6684836Z 2025-08-14T21:52:50.6684938Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6685149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6685222Z return mod(**inputs) 2025-08-14T21:52:50.6685513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6685591Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6685837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6685909Z outputs = self.model.decoder( 2025-08-14T21:52:50.6686135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6686207Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6686446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6686528Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6686746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6686823Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6687085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-08-14T21:52:50.6687227Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-08-14T21:52:50.6687232Z 2025-08-14T21:52:50.6687339Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6687535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6687599Z return mod(**inputs) 2025-08-14T21:52:50.6687812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6687883Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6688124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6688196Z outputs = self.model.decoder( 2025-08-14T21:52:50.6688406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6688486Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6688724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6688792Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6689031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6689103Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6689339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6689452Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6689685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:52:50.6689802Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:52:50.6689805Z 2025-08-14T21:52:50.6689903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6690107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6690170Z return mod(**inputs) 2025-08-14T21:52:50.6690378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6690456Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6690687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6690774Z outputs = self.model.decoder( 2025-08-14T21:52:50.6691010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6691082Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6691322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6691391Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6691603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6691686Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6691917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6692011Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6692253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:52:50.6692331Z key_states = self.k_proj(hidden_states) 2025-08-14T21:52:50.6692334Z 2025-08-14T21:52:50.6692441Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6692632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6692696Z return mod(**inputs) 2025-08-14T21:52:50.6692913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6692981Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6693217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6693288Z outputs = self.model.decoder( 2025-08-14T21:52:50.6693495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6693575Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6693808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6693876Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6694098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6694175Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6694413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6694507Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6694754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:52:50.6694846Z value_states = self.v_proj(hidden_states) 2025-08-14T21:52:50.6694850Z 2025-08-14T21:52:50.6694926Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6695024Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6695106Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6695182Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6695288Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6695479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6695542Z return mod(**inputs) 2025-08-14T21:52:50.6695758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6695828Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6696059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6696137Z outputs = self.model.decoder( 2025-08-14T21:52:50.6696369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6696450Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6696697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6696769Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6696986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6697061Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6697289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6697389Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6697616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6697715Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6697996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:52:50.6698126Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:52:50.6698130Z 2025-08-14T21:52:50.6698235Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6698424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6698494Z return mod(**inputs) 2025-08-14T21:52:50.6698698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6698768Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6699003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6699073Z outputs = self.model.decoder( 2025-08-14T21:52:50.6699279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6699360Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6699589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6699667Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6699875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6699950Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6700188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6700305Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6700543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6700636Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6700946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:52:50.6701060Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:52:50.6701064Z 2025-08-14T21:52:50.6701164Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6701358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6701431Z return mod(**inputs) 2025-08-14T21:52:50.6701642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6701724Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6701955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6702027Z outputs = self.model.decoder( 2025-08-14T21:52:50.6702261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6702355Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6702592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6702662Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6702869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6702959Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6703182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6703273Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6703504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:52:50.6703580Z attn_output = self.out_proj(attn_output) 2025-08-14T21:52:50.6703585Z 2025-08-14T21:52:50.6703685Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6703871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6703932Z return mod(**inputs) 2025-08-14T21:52:50.6704137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6704208Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6704435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6704516Z outputs = self.model.decoder( 2025-08-14T21:52:50.6704720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6704797Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6705030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6705099Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6705317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6705390Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6705627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:52:50.6705704Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:52:50.6705707Z 2025-08-14T21:52:50.6705804Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6706029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6706093Z return mod(**inputs) 2025-08-14T21:52:50.6706300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6706397Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6706692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6706769Z outputs = self.model.decoder( 2025-08-14T21:52:50.6706967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6707035Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6707263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6707330Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6707536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6707616Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6707856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:52:50.6707958Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:52:50.6707978Z 2025-08-14T21:52:50.6708073Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6708260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6708331Z return mod(**inputs) 2025-08-14T21:52:50.6708533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6708611Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6708836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6708906Z outputs = self.model.decoder( 2025-08-14T21:52:50.6709116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6709187Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6709416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6709492Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6709696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6709777Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6710001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:52:50.6710076Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:52:50.6710081Z 2025-08-14T21:52:50.6710181Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6710369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6710432Z return mod(**inputs) 2025-08-14T21:52:50.6710646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6710718Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6710952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6711019Z outputs = self.model.decoder( 2025-08-14T21:52:50.6711218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6711295Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6711521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6711621Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6711830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6711905Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6712157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6712248Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6712472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:52:50.6712582Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:52:50.6712586Z 2025-08-14T21:52:50.6712681Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6712874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6712937Z return mod(**inputs) 2025-08-14T21:52:50.6713140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6713219Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6713460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6713548Z outputs = self.model.decoder( 2025-08-14T21:52:50.6713757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6713827Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6714055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6714122Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6714326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6714412Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6714638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6714740Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6714969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:52:50.6715045Z key_states = self.k_proj(hidden_states) 2025-08-14T21:52:50.6715049Z 2025-08-14T21:52:50.6715155Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6715343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6715405Z return mod(**inputs) 2025-08-14T21:52:50.6715626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6715697Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6715923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6715990Z outputs = self.model.decoder( 2025-08-14T21:52:50.6716189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6716267Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6716488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6716561Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6716765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6716836Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6717063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6717174Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6717393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:52:50.6717480Z value_states = self.v_proj(hidden_states) 2025-08-14T21:52:50.6717515Z 2025-08-14T21:52:50.6717590Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6717671Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6717742Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6717812Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6717915Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6718102Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6718162Z return mod(**inputs) 2025-08-14T21:52:50.6718371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6718441Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6718670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6718758Z outputs = self.model.decoder( 2025-08-14T21:52:50.6718962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6719054Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6719281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6719349Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6719564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6719635Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6719863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6719953Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6720178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6720278Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6720551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:52:50.6720682Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:52:50.6720685Z 2025-08-14T21:52:50.6720781Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6720966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6721035Z return mod(**inputs) 2025-08-14T21:52:50.6721236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6721307Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6721540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6721611Z outputs = self.model.decoder( 2025-08-14T21:52:50.6721824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6721894Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6722120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6722198Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6722407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6722478Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6722747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6722836Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6723066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6723174Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6723446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:52:50.6723556Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:52:50.6723559Z 2025-08-14T21:52:50.6723656Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6723852Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6723914Z return mod(**inputs) 2025-08-14T21:52:50.6724119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6724197Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6724441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6724516Z outputs = self.model.decoder( 2025-08-14T21:52:50.6724748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6724821Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6725061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6725130Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6725340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6725498Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6725757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6725869Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6726130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:52:50.6726220Z attn_output = self.out_proj(attn_output) 2025-08-14T21:52:50.6726224Z 2025-08-14T21:52:50.6726341Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6726562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6726631Z return mod(**inputs) 2025-08-14T21:52:50.6726867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6726937Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6727183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6727256Z outputs = self.model.decoder( 2025-08-14T21:52:50.6727468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6727550Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6727782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6727854Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6728073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6728149Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6728411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:52:50.6728494Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:52:50.6728527Z 2025-08-14T21:52:50.6728637Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6728867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6728939Z return mod(**inputs) 2025-08-14T21:52:50.6729171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6729270Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6729520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6729603Z outputs = self.model.decoder( 2025-08-14T21:52:50.6729829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6729904Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6730171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6730249Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6730481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6730579Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6730846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:52:50.6730954Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:52:50.6730957Z 2025-08-14T21:52:50.6731063Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6731280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6731356Z return mod(**inputs) 2025-08-14T21:52:50.6731581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6731667Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6731926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6732002Z outputs = self.model.decoder( 2025-08-14T21:52:50.6732235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6732313Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6732569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6732642Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6732870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6732959Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6733215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:52:50.6733300Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:52:50.6733305Z 2025-08-14T21:52:50.6733416Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6733630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6733702Z return mod(**inputs) 2025-08-14T21:52:50.6733906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6733975Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6734207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6734277Z outputs = self.model.decoder( 2025-08-14T21:52:50.6734484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6734587Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6734819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6734897Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6735108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6735200Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6735441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 291, in forward 2025-08-14T21:52:50.6735571Z hidden_states = (residual + hidden_states).view(hidden_states_shape) 2025-08-14T21:52:50.6735574Z 2025-08-14T21:52:50.6735679Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6735870Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6735933Z return mod(**inputs) 2025-08-14T21:52:50.6736159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6736230Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6736472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6736553Z outputs = self.model.decoder( 2025-08-14T21:52:50.6736769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6736851Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6737075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6737143Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6737355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6737429Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6737784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6737895Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6738127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 159, in forward 2025-08-14T21:52:50.6738245Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:52:50.6738250Z 2025-08-14T21:52:50.6738349Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6738538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6738612Z return mod(**inputs) 2025-08-14T21:52:50.6738827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6738912Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6739145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6739219Z outputs = self.model.decoder( 2025-08-14T21:52:50.6739442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6739520Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6739757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6739839Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6740055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6740142Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6740384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6740536Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6740777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 162, in forward 2025-08-14T21:52:50.6740853Z key_states = self.k_proj(hidden_states) 2025-08-14T21:52:50.6740857Z 2025-08-14T21:52:50.6740963Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6741200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6741264Z return mod(**inputs) 2025-08-14T21:52:50.6741482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6741554Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6741785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6741865Z outputs = self.model.decoder( 2025-08-14T21:52:50.6742071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6742150Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6742412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6742485Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6742745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6742824Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6743054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6743156Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6743383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 163, in forward 2025-08-14T21:52:50.6743476Z value_states = self.v_proj(hidden_states) 2025-08-14T21:52:50.6743479Z 2025-08-14T21:52:50.6743555Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6743632Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6743715Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6743790Z cudagraph partition due to non gpu ops 2025-08-14T21:52:50.6743890Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6744091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6744153Z return mod(**inputs) 2025-08-14T21:52:50.6744368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6744438Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6744674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6744752Z outputs = self.model.decoder( 2025-08-14T21:52:50.6744954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6745021Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6745252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6745320Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6745530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6745604Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6745832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6745931Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6746160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6746284Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6746564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:52:50.6746695Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:52:50.6746721Z 2025-08-14T21:52:50.6746829Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6747019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6747081Z return mod(**inputs) 2025-08-14T21:52:50.6747300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6747373Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6747625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6747695Z outputs = self.model.decoder( 2025-08-14T21:52:50.6747901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6747977Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6748572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6748671Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6748883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6748958Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6749192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6749288Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6749517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 184, in forward 2025-08-14T21:52:50.6749622Z attn_output, attn_weights = attention_interface( 2025-08-14T21:52:50.6749905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:52:50.6750019Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:52:50.6750023Z 2025-08-14T21:52:50.6750126Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6750324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6750398Z return mod(**inputs) 2025-08-14T21:52:50.6750614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6750697Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6750937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6751014Z outputs = self.model.decoder( 2025-08-14T21:52:50.6751233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6751308Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6751550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6751630Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6751859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6751940Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6752173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 259, in forward 2025-08-14T21:52:50.6752272Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:52:50.6752507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 196, in forward 2025-08-14T21:52:50.6752615Z attn_output = self.out_proj(attn_output) 2025-08-14T21:52:50.6752619Z 2025-08-14T21:52:50.6752721Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6752927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6753009Z return mod(**inputs) 2025-08-14T21:52:50.6753223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6753294Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6753521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6753599Z outputs = self.model.decoder( 2025-08-14T21:52:50.6753805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6753886Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6754116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6754187Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6754425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6754521Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6754749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 285, in forward 2025-08-14T21:52:50.6754833Z hidden_states = self.fc1(hidden_states) 2025-08-14T21:52:50.6754837Z 2025-08-14T21:52:50.6754934Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6755134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6755199Z return mod(**inputs) 2025-08-14T21:52:50.6755411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6755492Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6755729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6755802Z outputs = self.model.decoder( 2025-08-14T21:52:50.6756021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6756095Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6756334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6756405Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6756620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6756707Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6756948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 286, in forward 2025-08-14T21:52:50.6757046Z hidden_states = self.activation_fn(hidden_states) 2025-08-14T21:52:50.6757051Z 2025-08-14T21:52:50.6757151Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6757344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6757415Z return mod(**inputs) 2025-08-14T21:52:50.6757621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6757693Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6757929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 826, in forward 2025-08-14T21:52:50.6757999Z outputs = self.model.decoder( 2025-08-14T21:52:50.6758238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6758310Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6758545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 653, in forward 2025-08-14T21:52:50.6758642Z layer_outputs = decoder_layer( 2025-08-14T21:52:50.6758863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:52:50.6758940Z return super().__call__(*args, **kwargs) 2025-08-14T21:52:50.6759185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 288, in forward 2025-08-14T21:52:50.6759263Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:52:50.6759267Z 2025-08-14T21:52:50.6759370Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6759568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6759635Z return mod(**inputs) 2025-08-14T21:52:50.6759861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6759949Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6760210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 841, in forward 2025-08-14T21:52:50.6760304Z logits = self.lm_head(outputs[0]).contiguous() 2025-08-14T21:52:50.6760308Z 2025-08-14T21:52:50.6760406Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:52:50.6760609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:52:50.6760672Z return mod(**inputs) 2025-08-14T21:52:50.6760883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/generic.py", line 961, in wrapper 2025-08-14T21:52:50.6760966Z output = func(self, *args, **kwargs) 2025-08-14T21:52:50.6761209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/opt/modeling_opt.py", line 847, in forward 2025-08-14T21:52:50.6761295Z loss = self.loss_function( 2025-08-14T21:52:50.6761550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-08-14T21:52:50.6761749Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-08-14T21:52:50.6762002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-08-14T21:52:50.6762194Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-08-14T21:52:50.6762198Z 2025-08-14T21:53:01.3228645Z Compilation time (from dynamo_timed): 15.717350689 2025-08-14T21:53:01.3793147Z pass 2025-08-14T21:53:01.3793753Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:53:01.3798837Z TIMING: _recursive_pre_grad_passes:0.0078 _recursive_joint_graph_passes:0.63977 _recursive_post_grad_passes:0.09605 async_compile.wait:0.80896 code_gen:9.30226 inductor_compile:10.48806 backend_compile:13.5768 gc:0.00128 entire_frame_compile:15.71735 total_wall_time:15.71735 2025-08-14T21:53:01.3800786Z STATS: call_* op count: 415 | FakeTensorMode.__torch_dispatch__:12797 | FakeTensor.__torch_dispatch__:4472 | ProxyTorchDispatchMode.__torch_dispatch__:4707 2025-08-14T21:53:01.3801934Z Dynamo produced 1 graphs covering 415 ops with 0 graph breaks (0 unique) 2025-08-14T21:53:06.6881111Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:53:06.6882473Z from pkg_resources import resource_filename 2025-08-14T21:53:07.2600193Z 2025-08-14T21:53:08.6273320Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:53:08.6278059Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:53:08.6282861Z cpu eval PLBartForCausalLM 2025-08-14T21:53:09.3123295Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:53:09.5881082Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:53:09.8643207Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:53:14.7084240Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7084854Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7085318Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7085860Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7086128Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7086425Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7086691Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7087147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7087885Z return mod(**inputs) 2025-08-14T21:53:14.7088491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7088989Z outputs = self.model.decoder( 2025-08-14T21:53:14.7089438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7089884Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7090265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7090653Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7091092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7091568Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7092016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:14.7092539Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:14.7092768Z 2025-08-14T21:53:14.7092883Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7093271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7093613Z return mod(**inputs) 2025-08-14T21:53:14.7094018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7094441Z outputs = self.model.decoder( 2025-08-14T21:53:14.7094831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7095232Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7095583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7095959Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7096393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7096853Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7097311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:14.7097763Z key_states = self.k_proj(current_states) 2025-08-14T21:53:14.7097898Z 2025-08-14T21:53:14.7098004Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7098444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7098770Z return mod(**inputs) 2025-08-14T21:53:14.7099129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7099526Z outputs = self.model.decoder( 2025-08-14T21:53:14.7099978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7100401Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7100750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7101106Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7101492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7101910Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7102325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:14.7102732Z value_states = self.v_proj(current_states) 2025-08-14T21:53:14.7102874Z 2025-08-14T21:53:14.7102979Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7103196Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7103425Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7103629Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7103856Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7104217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7104544Z return mod(**inputs) 2025-08-14T21:53:14.7104907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7105306Z outputs = self.model.decoder( 2025-08-14T21:53:14.7105691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7106082Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7106548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7106919Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7107332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7107762Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7108197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:14.7108606Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:14.7109050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:14.7109525Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:14.7109718Z 2025-08-14T21:53:14.7109821Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7110180Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7110502Z return mod(**inputs) 2025-08-14T21:53:14.7110862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7111257Z outputs = self.model.decoder( 2025-08-14T21:53:14.7111641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7112022Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7112371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7113635Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7114023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7114427Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7114855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:14.7115257Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:14.7115685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:14.7116124Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:14.7116293Z 2025-08-14T21:53:14.7116395Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7116751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7117068Z return mod(**inputs) 2025-08-14T21:53:14.7117436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7117859Z outputs = self.model.decoder( 2025-08-14T21:53:14.7118252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7118627Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7118968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7119325Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7119706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7120116Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7120525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:14.7120917Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:14.7121053Z 2025-08-14T21:53:14.7121157Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7121517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7121851Z return mod(**inputs) 2025-08-14T21:53:14.7122221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7122611Z outputs = self.model.decoder( 2025-08-14T21:53:14.7122999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7123399Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7123733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7124089Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7124474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:14.7124899Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:14.7125073Z 2025-08-14T21:53:14.7125184Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7125671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7126027Z return mod(**inputs) 2025-08-14T21:53:14.7126404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7126812Z outputs = self.model.decoder( 2025-08-14T21:53:14.7127211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7127670Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7128029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7128411Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7128850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:14.7129318Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:14.7129720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:53:14.7130079Z return self.act(input) 2025-08-14T21:53:14.7130195Z 2025-08-14T21:53:14.7130310Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7130700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7131041Z return mod(**inputs) 2025-08-14T21:53:14.7131428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7131847Z outputs = self.model.decoder( 2025-08-14T21:53:14.7132263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7132701Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7133071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7133443Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7133859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:53:14.7134281Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:53:14.7134426Z 2025-08-14T21:53:14.7134540Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7134912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7135258Z return mod(**inputs) 2025-08-14T21:53:14.7135645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7136062Z outputs = self.model.decoder( 2025-08-14T21:53:14.7136461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7136876Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7137236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7137604Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7138242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7138691Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7139126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:14.7139598Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:14.7139810Z 2025-08-14T21:53:14.7139912Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7140271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7140591Z return mod(**inputs) 2025-08-14T21:53:14.7140949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7141340Z outputs = self.model.decoder( 2025-08-14T21:53:14.7141719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7142166Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7142515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7142874Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7143273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7143739Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7144154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:14.7144551Z key_states = self.k_proj(current_states) 2025-08-14T21:53:14.7144687Z 2025-08-14T21:53:14.7144788Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7145142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7145464Z return mod(**inputs) 2025-08-14T21:53:14.7145874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7146256Z outputs = self.model.decoder( 2025-08-14T21:53:14.7146676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7147071Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7147444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7147803Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7148202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7148622Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7149037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:14.7149442Z value_states = self.v_proj(current_states) 2025-08-14T21:53:14.7149590Z 2025-08-14T21:53:14.7149674Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7149892Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7150105Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7150324Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7150563Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7150921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7151253Z return mod(**inputs) 2025-08-14T21:53:14.7151629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7152048Z outputs = self.model.decoder( 2025-08-14T21:53:14.7152423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7152814Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7153161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7153514Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7153911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7154320Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7154731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:14.7155133Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:14.7155578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:14.7156053Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:14.7156252Z 2025-08-14T21:53:14.7156359Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7156699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7157013Z return mod(**inputs) 2025-08-14T21:53:14.7157395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7157771Z outputs = self.model.decoder( 2025-08-14T21:53:14.7158146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7158530Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7158868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7159211Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7159594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7160001Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7160407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:14.7160810Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:14.7161272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:14.7161717Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:14.7161873Z 2025-08-14T21:53:14.7161972Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7162318Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7162630Z return mod(**inputs) 2025-08-14T21:53:14.7162994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7163361Z outputs = self.model.decoder( 2025-08-14T21:53:14.7163726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7164107Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7164440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7164788Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7165193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7165730Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7166177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:14.7166608Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:14.7166753Z 2025-08-14T21:53:14.7166869Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7167253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7167592Z return mod(**inputs) 2025-08-14T21:53:14.7167978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7168369Z outputs = self.model.decoder( 2025-08-14T21:53:14.7168733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7169115Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7169443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7169783Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7170177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:14.7170590Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:14.7170753Z 2025-08-14T21:53:14.7170860Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7171226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7171531Z return mod(**inputs) 2025-08-14T21:53:14.7171878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7172251Z outputs = self.model.decoder( 2025-08-14T21:53:14.7172603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7172971Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7173295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7173633Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7174011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:14.7174430Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:14.7174813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:53:14.7175131Z return self.act(input) 2025-08-14T21:53:14.7175242Z 2025-08-14T21:53:14.7175338Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7175676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7175984Z return mod(**inputs) 2025-08-14T21:53:14.7176319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7176691Z outputs = self.model.decoder( 2025-08-14T21:53:14.7177050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7177411Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7177740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7178078Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7178446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:53:14.7178812Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:53:14.7178946Z 2025-08-14T21:53:14.7179042Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7179376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7179670Z return mod(**inputs) 2025-08-14T21:53:14.7180016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7180383Z outputs = self.model.decoder( 2025-08-14T21:53:14.7180742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7181103Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7181428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7181765Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7182138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7182520Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7182907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:14.7183364Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:14.7183553Z 2025-08-14T21:53:14.7183651Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7184015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7184321Z return mod(**inputs) 2025-08-14T21:53:14.7184680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7185039Z outputs = self.model.decoder( 2025-08-14T21:53:14.7185395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7185795Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7186127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7186466Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7186844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7187259Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7187657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:14.7188039Z key_states = self.k_proj(current_states) 2025-08-14T21:53:14.7188174Z 2025-08-14T21:53:14.7188274Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7188621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7188933Z return mod(**inputs) 2025-08-14T21:53:14.7189295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7189671Z outputs = self.model.decoder( 2025-08-14T21:53:14.7190030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7190402Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7190735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7191079Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7191446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7191840Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7192239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:14.7192624Z value_states = self.v_proj(current_states) 2025-08-14T21:53:14.7192758Z 2025-08-14T21:53:14.7192834Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7193037Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7193237Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7193423Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7193654Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7193982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7194275Z return mod(**inputs) 2025-08-14T21:53:14.7194617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7194980Z outputs = self.model.decoder( 2025-08-14T21:53:14.7195337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7195688Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7196037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7196427Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7196797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7197208Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7197619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:14.7198012Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:14.7198428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:14.7198885Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:14.7199065Z 2025-08-14T21:53:14.7199162Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7199504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7199811Z return mod(**inputs) 2025-08-14T21:53:14.7200191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7200581Z outputs = self.model.decoder( 2025-08-14T21:53:14.7200972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7201350Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7201684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7202031Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7202404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7202808Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7203207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:14.7203608Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:14.7204027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:14.7204470Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:14.7204623Z 2025-08-14T21:53:14.7204730Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7205081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7205499Z return mod(**inputs) 2025-08-14T21:53:14.7205875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7206275Z outputs = self.model.decoder( 2025-08-14T21:53:14.7206650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7207048Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7207390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7207746Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7208131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7208538Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7208941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:14.7209323Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:14.7209554Z 2025-08-14T21:53:14.7209656Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7210010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7210330Z return mod(**inputs) 2025-08-14T21:53:14.7210689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7211098Z outputs = self.model.decoder( 2025-08-14T21:53:14.7211474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7211861Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7212189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7212542Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7212929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:14.7213347Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:14.7213520Z 2025-08-14T21:53:14.7213620Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7214004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7214319Z return mod(**inputs) 2025-08-14T21:53:14.7214680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7215063Z outputs = self.model.decoder( 2025-08-14T21:53:14.7215432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7215803Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7216139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7216487Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7216872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:14.7217283Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:14.7217657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:53:14.7217992Z return self.act(input) 2025-08-14T21:53:14.7218100Z 2025-08-14T21:53:14.7218207Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7218543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7218875Z return mod(**inputs) 2025-08-14T21:53:14.7219220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7219585Z outputs = self.model.decoder( 2025-08-14T21:53:14.7219948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7220319Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7220647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7220983Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7221351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:53:14.7221724Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:53:14.7221851Z 2025-08-14T21:53:14.7221948Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7222286Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7222592Z return mod(**inputs) 2025-08-14T21:53:14.7222938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7223323Z outputs = self.model.decoder( 2025-08-14T21:53:14.7223683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7224071Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7224391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7224730Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7225101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7225493Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7225873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:14.7226319Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:14.7226518Z 2025-08-14T21:53:14.7226617Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7226973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7227272Z return mod(**inputs) 2025-08-14T21:53:14.7227635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7228010Z outputs = self.model.decoder( 2025-08-14T21:53:14.7228367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7228735Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7229061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7229397Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7229764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7230153Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7230540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:14.7230918Z key_states = self.k_proj(current_states) 2025-08-14T21:53:14.7231046Z 2025-08-14T21:53:14.7231140Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7231476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7231782Z return mod(**inputs) 2025-08-14T21:53:14.7232121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7232494Z outputs = self.model.decoder( 2025-08-14T21:53:14.7232859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7233236Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7233560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7233908Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7234286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7234682Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7235074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:14.7235447Z value_states = self.v_proj(current_states) 2025-08-14T21:53:14.7235578Z 2025-08-14T21:53:14.7235664Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7235886Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7236088Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7236284Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7236502Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7236847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7237178Z return mod(**inputs) 2025-08-14T21:53:14.7237537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7238059Z outputs = self.model.decoder( 2025-08-14T21:53:14.7238420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7238783Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7239096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7239434Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7239802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7240187Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7240626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:14.7241089Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:14.7241504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:14.7241950Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:14.7242118Z 2025-08-14T21:53:14.7242213Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7242552Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7242860Z return mod(**inputs) 2025-08-14T21:53:14.7243237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7243615Z outputs = self.model.decoder( 2025-08-14T21:53:14.7243985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7244348Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7244659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7244988Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7245398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7245791Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7246163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:14.7246550Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:14.7246971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:14.7247406Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:14.7247572Z 2025-08-14T21:53:14.7247671Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7248016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7248336Z return mod(**inputs) 2025-08-14T21:53:14.7248669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7249035Z outputs = self.model.decoder( 2025-08-14T21:53:14.7249389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7249788Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7250103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7250434Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7250838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7251215Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7251591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:14.7251958Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:14.7252082Z 2025-08-14T21:53:14.7252187Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7252507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7252806Z return mod(**inputs) 2025-08-14T21:53:14.7253143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7253520Z outputs = self.model.decoder( 2025-08-14T21:53:14.7253896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7254261Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7254581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7254904Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7255268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:14.7255669Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:14.7255829Z 2025-08-14T21:53:14.7255934Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7256263Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7256568Z return mod(**inputs) 2025-08-14T21:53:14.7256913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7257293Z outputs = self.model.decoder( 2025-08-14T21:53:14.7257672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7258060Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7258403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7258736Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7259110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:14.7259524Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:14.7259893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:53:14.7260228Z return self.act(input) 2025-08-14T21:53:14.7260340Z 2025-08-14T21:53:14.7260443Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7260793Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7261099Z return mod(**inputs) 2025-08-14T21:53:14.7261452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7261834Z outputs = self.model.decoder( 2025-08-14T21:53:14.7262208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7262609Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7262944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7263297Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7263675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:53:14.7264153Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:53:14.7264289Z 2025-08-14T21:53:14.7264387Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7264730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7265035Z return mod(**inputs) 2025-08-14T21:53:14.7265390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7265772Z outputs = self.model.decoder( 2025-08-14T21:53:14.7266138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7266521Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7266875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7267231Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7267623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7268032Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7268436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:14.7268888Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:14.7269083Z 2025-08-14T21:53:14.7269185Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7269529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7269841Z return mod(**inputs) 2025-08-14T21:53:14.7270187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7270577Z outputs = self.model.decoder( 2025-08-14T21:53:14.7270950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7271329Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7271654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7272003Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7272387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7272798Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7273200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:14.7273596Z key_states = self.k_proj(current_states) 2025-08-14T21:53:14.7273725Z 2025-08-14T21:53:14.7273827Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7274156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7274460Z return mod(**inputs) 2025-08-14T21:53:14.7274803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7275175Z outputs = self.model.decoder( 2025-08-14T21:53:14.7275534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7275933Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7276261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7276592Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7276961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7277375Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7277761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:14.7278135Z value_states = self.v_proj(current_states) 2025-08-14T21:53:14.7278275Z 2025-08-14T21:53:14.7278352Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7278556Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7278746Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7278942Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7279160Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7279499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7279799Z return mod(**inputs) 2025-08-14T21:53:14.7280174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7280570Z outputs = self.model.decoder( 2025-08-14T21:53:14.7280932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7281306Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7281629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7281969Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7282335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7282731Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7283127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:14.7283521Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:14.7283936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:14.7284401Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:14.7284575Z 2025-08-14T21:53:14.7284678Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7285012Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7285409Z return mod(**inputs) 2025-08-14T21:53:14.7285793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7286196Z outputs = self.model.decoder( 2025-08-14T21:53:14.7286580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7286982Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7287348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7287728Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7288138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7288587Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7289018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:14.7289459Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:14.7289886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:14.7290366Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:14.7290554Z 2025-08-14T21:53:14.7290671Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7291040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7291381Z return mod(**inputs) 2025-08-14T21:53:14.7291773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7292194Z outputs = self.model.decoder( 2025-08-14T21:53:14.7292598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7293017Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7293379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7293746Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7294177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7294634Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7295080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:14.7295505Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:14.7295659Z 2025-08-14T21:53:14.7295769Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7296152Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7296496Z return mod(**inputs) 2025-08-14T21:53:14.7296894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7297326Z outputs = self.model.decoder( 2025-08-14T21:53:14.7297798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7298217Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7298580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7298955Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7299357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:14.7299816Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:14.7299989Z 2025-08-14T21:53:14.7300088Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7300441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7300752Z return mod(**inputs) 2025-08-14T21:53:14.7301119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7301512Z outputs = self.model.decoder( 2025-08-14T21:53:14.7301914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7302322Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7302686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7303062Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7303469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:14.7303957Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:14.7304357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:53:14.7304705Z return self.act(input) 2025-08-14T21:53:14.7304812Z 2025-08-14T21:53:14.7304911Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7305296Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7305645Z return mod(**inputs) 2025-08-14T21:53:14.7306031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7306456Z outputs = self.model.decoder( 2025-08-14T21:53:14.7306900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7307399Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7307738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7308106Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7308553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:53:14.7308975Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:53:14.7309117Z 2025-08-14T21:53:14.7309241Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7309621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7309959Z return mod(**inputs) 2025-08-14T21:53:14.7310351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7310784Z outputs = self.model.decoder( 2025-08-14T21:53:14.7311187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7311592Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7311926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7312287Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7312684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7313086Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7313495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:14.7313962Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:14.7314162Z 2025-08-14T21:53:14.7314273Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7314619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7314949Z return mod(**inputs) 2025-08-14T21:53:14.7315303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7315684Z outputs = self.model.decoder( 2025-08-14T21:53:14.7316052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7316443Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7316786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7317142Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7317538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7317949Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7318396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:14.7318785Z key_states = self.k_proj(current_states) 2025-08-14T21:53:14.7318927Z 2025-08-14T21:53:14.7319029Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7319410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7319734Z return mod(**inputs) 2025-08-14T21:53:14.7320092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7320487Z outputs = self.model.decoder( 2025-08-14T21:53:14.7320868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7321254Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7321600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7321963Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7322371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7322782Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7323243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:14.7323646Z value_states = self.v_proj(current_states) 2025-08-14T21:53:14.7323784Z 2025-08-14T21:53:14.7323872Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7324080Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7324288Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7324495Z cudagraph partition due to non gpu ops 2025-08-14T21:53:14.7324717Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7325071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7325520Z return mod(**inputs) 2025-08-14T21:53:14.7325935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7326377Z outputs = self.model.decoder( 2025-08-14T21:53:14.7326828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7327273Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7327641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7328029Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7328462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7328906Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7329370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:14.7329837Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:14.7330312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:14.7330821Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:14.7331034Z 2025-08-14T21:53:14.7331150Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7331550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7331906Z return mod(**inputs) 2025-08-14T21:53:14.7332306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7332758Z outputs = self.model.decoder( 2025-08-14T21:53:14.7333160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7333562Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7333926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7334336Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7334765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7335213Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7335662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:14.7336110Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:14.7336590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:14.7337082Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:14.7337262Z 2025-08-14T21:53:14.7337393Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7337945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7338355Z return mod(**inputs) 2025-08-14T21:53:14.7338764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7339201Z outputs = self.model.decoder( 2025-08-14T21:53:14.7339623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7340046Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7340427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7340822Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7341275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:14.7341714Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:14.7342113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:14.7342496Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:14.7342628Z 2025-08-14T21:53:14.7342726Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7343063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7343368Z return mod(**inputs) 2025-08-14T21:53:14.7343718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7344085Z outputs = self.model.decoder( 2025-08-14T21:53:14.7344445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7344817Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7345141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7345494Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7345878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:14.7346300Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:14.7346465Z 2025-08-14T21:53:14.7346564Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7346920Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7347266Z return mod(**inputs) 2025-08-14T21:53:14.7347612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7347975Z outputs = self.model.decoder( 2025-08-14T21:53:14.7348335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7348734Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7349051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7349383Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7349758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:14.7350167Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:14.7350534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:53:14.7350877Z return self.act(input) 2025-08-14T21:53:14.7350980Z 2025-08-14T21:53:14.7351084Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7351444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7351760Z return mod(**inputs) 2025-08-14T21:53:14.7352178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1678, in forward 2025-08-14T21:53:14.7352560Z outputs = self.model.decoder( 2025-08-14T21:53:14.7352918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:14.7353291Z layer_outputs = decoder_layer( 2025-08-14T21:53:14.7353621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:14.7353964Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:14.7354333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:53:14.7354718Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:53:14.7354851Z 2025-08-14T21:53:14.7354959Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7355294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7355601Z return mod(**inputs) 2025-08-14T21:53:14.7355949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1694, in forward 2025-08-14T21:53:14.7356331Z logits = self.lm_head(outputs[0]) 2025-08-14T21:53:14.7356454Z 2025-08-14T21:53:14.7356550Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:14.7356890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:14.7357194Z return mod(**inputs) 2025-08-14T21:53:14.7357534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1700, in forward 2025-08-14T21:53:14.7357978Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:53:14.7358170Z 2025-08-14T21:53:22.8253446Z Compilation time (from dynamo_timed): 11.396213489 2025-08-14T21:53:22.8587913Z pass 2025-08-14T21:53:22.8588335Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:53:22.8589067Z TIMING: _recursive_pre_grad_passes:0.00573 _recursive_joint_graph_passes:0.24183 _recursive_post_grad_passes:0.05447 async_compile.wait:0.78419 code_gen:7.4444 inductor_compile:8.41824 backend_compile:10.17486 gc:0.00089 entire_frame_compile:11.39621 total_wall_time:11.39621 2025-08-14T21:53:22.8589920Z STATS: call_* op count: 198 | FakeTensorMode.__torch_dispatch__:7102 | FakeTensor.__torch_dispatch__:2588 | ProxyTorchDispatchMode.__torch_dispatch__:2533 2025-08-14T21:53:22.8590759Z Dynamo produced 1 graphs covering 198 ops with 0 graph breaks (0 unique) 2025-08-14T21:53:27.9727961Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:53:27.9729214Z from pkg_resources import resource_filename 2025-08-14T21:53:28.5713332Z 2025-08-14T21:53:30.9451104Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:53:30.9455116Z loading model: 0it [00:02, ?it/s] 2025-08-14T21:53:30.9469272Z cpu eval PLBartForConditionalGeneration 2025-08-14T21:53:32.1836431Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:53:32.7148092Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:53:33.2446597Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:53:42.5523072Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5523893Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5524278Z return mod(**inputs) 2025-08-14T21:53:42.5528207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1357, in forward 2025-08-14T21:53:42.5528854Z decoder_input_ids = shift_tokens_right(labels, self.config.pad_token_id) 2025-08-14T21:53:42.5529440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1084, in shift_tokens_right 2025-08-14T21:53:42.5530520Z index_of_eos = (prev_output_tokens.ne(pad_token_id).sum(dim=1) - 1).unsqueeze(-1) 2025-08-14T21:53:42.5530798Z 2025-08-14T21:53:42.5531506Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5531767Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5531975Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5532184Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5532410Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5532612Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5538518Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5544332Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5548985Z return mod(**inputs) 2025-08-14T21:53:42.5553918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5556766Z outputs = self.model( 2025-08-14T21:53:42.5557447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5558078Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5558675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5559115Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5559487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5559850Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5560248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5560659Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5561062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:42.5561511Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:42.5561919Z 2025-08-14T21:53:42.5562034Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5562444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5562862Z return mod(**inputs) 2025-08-14T21:53:42.5563280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5563795Z outputs = self.model( 2025-08-14T21:53:42.5564188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5564577Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5564965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5565506Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5565903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5566299Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5566775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5567191Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5567615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:42.5568008Z key_states = self.k_proj(current_states) 2025-08-14T21:53:42.5568149Z 2025-08-14T21:53:42.5568255Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5568615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5568931Z return mod(**inputs) 2025-08-14T21:53:42.5569296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5569696Z outputs = self.model( 2025-08-14T21:53:42.5570066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5570445Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5570828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5571210Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5571544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5572192Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5572579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5572982Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5573372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:42.5573765Z value_states = self.v_proj(current_states) 2025-08-14T21:53:42.5573903Z 2025-08-14T21:53:42.5573994Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5574208Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5574407Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5574615Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5574850Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5575198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5575516Z return mod(**inputs) 2025-08-14T21:53:42.5575875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5576246Z outputs = self.model( 2025-08-14T21:53:42.5576657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5577043Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5577433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5577834Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5578176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5578527Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5578908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5579315Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5579721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.5580139Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.5580576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:42.5581053Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:42.5581242Z 2025-08-14T21:53:42.5581340Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5581699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5582005Z return mod(**inputs) 2025-08-14T21:53:42.5582367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5582744Z outputs = self.model( 2025-08-14T21:53:42.5583100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5583478Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5583854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5584236Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5584558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5584903Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5585277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5585661Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5586036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.5586429Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.5586851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:42.5587281Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:42.5587430Z 2025-08-14T21:53:42.5587529Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5587871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5588193Z return mod(**inputs) 2025-08-14T21:53:42.5588543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5588923Z outputs = self.model( 2025-08-14T21:53:42.5589282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5589664Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5590034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5590437Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5590774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5591118Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5591529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5591926Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5592323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:42.5592707Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:42.5592850Z 2025-08-14T21:53:42.5592951Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5593301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5593625Z return mod(**inputs) 2025-08-14T21:53:42.5593993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5594370Z outputs = self.model( 2025-08-14T21:53:42.5594744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5595143Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5595521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5595898Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5596234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5596572Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5596964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:53:42.5597401Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.5597570Z 2025-08-14T21:53:42.5597670Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5598023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5598343Z return mod(**inputs) 2025-08-14T21:53:42.5598710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5599086Z outputs = self.model( 2025-08-14T21:53:42.5599450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5599838Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5600214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5600598Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5600939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5601295Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5601678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:53:42.5602106Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.5602489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:53:42.5602825Z return self.act(input) 2025-08-14T21:53:42.5602930Z 2025-08-14T21:53:42.5603030Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5603385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5603728Z return mod(**inputs) 2025-08-14T21:53:42.5604079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5604463Z outputs = self.model( 2025-08-14T21:53:42.5604828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5605260Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5605725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5606136Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5606510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5606881Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5607282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-08-14T21:53:42.5607723Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:53:42.5607862Z 2025-08-14T21:53:42.5607987Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5608351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5608675Z return mod(**inputs) 2025-08-14T21:53:42.5609052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5609435Z outputs = self.model( 2025-08-14T21:53:42.5609786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5610167Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5610545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5610919Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5611263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5611605Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5611982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5612360Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5612743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:42.5613186Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:42.5613378Z 2025-08-14T21:53:42.5613482Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5613815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5614123Z return mod(**inputs) 2025-08-14T21:53:42.5614474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5614834Z outputs = self.model( 2025-08-14T21:53:42.5615182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5615557Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5615924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5616288Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5616616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5616955Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5617315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5617720Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5618101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:42.5618479Z key_states = self.k_proj(current_states) 2025-08-14T21:53:42.5618626Z 2025-08-14T21:53:42.5618723Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5619061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5619368Z return mod(**inputs) 2025-08-14T21:53:42.5619717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5620077Z outputs = self.model( 2025-08-14T21:53:42.5620428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5620803Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5621163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5621534Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5621878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5622241Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5622616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5623004Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5623390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:42.5623775Z value_states = self.v_proj(current_states) 2025-08-14T21:53:42.5623922Z 2025-08-14T21:53:42.5624005Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5624224Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5624425Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5624616Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5624840Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5625199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5625515Z return mod(**inputs) 2025-08-14T21:53:42.5625867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5626237Z outputs = self.model( 2025-08-14T21:53:42.5626588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5626959Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5627327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5627699Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5628023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5628367Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5628741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5629128Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5629505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.5629899Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.5630317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:42.5630796Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:42.5630968Z 2025-08-14T21:53:42.5631067Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5631407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5631717Z return mod(**inputs) 2025-08-14T21:53:42.5632080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5632449Z outputs = self.model( 2025-08-14T21:53:42.5632796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5633162Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5633521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5633897Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5634239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5634573Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5634964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5635357Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5635757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.5636150Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.5636577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:42.5637016Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:42.5637169Z 2025-08-14T21:53:42.5637273Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5637606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5638125Z return mod(**inputs) 2025-08-14T21:53:42.5638485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5638855Z outputs = self.model( 2025-08-14T21:53:42.5639209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5639588Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5639962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5640330Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5640668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5641017Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5641389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5641785Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5642178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:42.5642566Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:42.5642695Z 2025-08-14T21:53:42.5642795Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5643146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5643464Z return mod(**inputs) 2025-08-14T21:53:42.5643823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5644207Z outputs = self.model( 2025-08-14T21:53:42.5644601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5644983Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5645393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5645826Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5646176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5646538Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5646929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:53:42.5647357Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.5647527Z 2025-08-14T21:53:42.5647639Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5647993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5648303Z return mod(**inputs) 2025-08-14T21:53:42.5648692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5649076Z outputs = self.model( 2025-08-14T21:53:42.5649454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5649839Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5650213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5650587Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5650914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5651266Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5651649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:53:42.5652062Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.5652435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:53:42.5652770Z return self.act(input) 2025-08-14T21:53:42.5652876Z 2025-08-14T21:53:42.5652983Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5653324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5653639Z return mod(**inputs) 2025-08-14T21:53:42.5653997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5654366Z outputs = self.model( 2025-08-14T21:53:42.5654727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5655110Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5655483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5655854Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5656193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5656542Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5656924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-08-14T21:53:42.5657302Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:53:42.5657444Z 2025-08-14T21:53:42.5657544Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5657901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5658243Z return mod(**inputs) 2025-08-14T21:53:42.5658595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5658966Z outputs = self.model( 2025-08-14T21:53:42.5659341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5659706Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5660070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5660441Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5660766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5661107Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5661486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5661875Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5662279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:42.5662745Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:42.5662950Z 2025-08-14T21:53:42.5663054Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5663404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5663709Z return mod(**inputs) 2025-08-14T21:53:42.5664081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5664457Z outputs = self.model( 2025-08-14T21:53:42.5664800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5665175Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5665541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5665914Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5666235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5666575Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5666951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5667338Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5667716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:42.5668098Z key_states = self.k_proj(current_states) 2025-08-14T21:53:42.5668227Z 2025-08-14T21:53:42.5668333Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5668666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5668976Z return mod(**inputs) 2025-08-14T21:53:42.5669335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5669707Z outputs = self.model( 2025-08-14T21:53:42.5670055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5670430Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5670799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5671163Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5671520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5671864Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5672239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5672636Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5673020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:42.5673402Z value_states = self.v_proj(current_states) 2025-08-14T21:53:42.5673535Z 2025-08-14T21:53:42.5673622Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5673821Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5674051Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5674266Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5674480Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5674823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5675129Z return mod(**inputs) 2025-08-14T21:53:42.5675488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5675860Z outputs = self.model( 2025-08-14T21:53:42.5676226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5676602Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5676968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5677333Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5677662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5677996Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5678374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5678748Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5679124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.5679510Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.5679934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:42.5680386Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:42.5680559Z 2025-08-14T21:53:42.5680664Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5680999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5681309Z return mod(**inputs) 2025-08-14T21:53:42.5681660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5682020Z outputs = self.model( 2025-08-14T21:53:42.5682371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5682750Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5683124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5683495Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5683830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5684179Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5684565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5684979Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5685434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.5685882Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.5686323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:42.5686813Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:42.5686991Z 2025-08-14T21:53:42.5687094Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5687455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5687765Z return mod(**inputs) 2025-08-14T21:53:42.5688123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5688500Z outputs = self.model( 2025-08-14T21:53:42.5688861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5689243Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5689623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5690031Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5690355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5690702Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5691086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5691486Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5691868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:42.5692253Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:42.5692382Z 2025-08-14T21:53:42.5692493Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5692834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5693147Z return mod(**inputs) 2025-08-14T21:53:42.5693500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5693869Z outputs = self.model( 2025-08-14T21:53:42.5694220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5694598Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5694977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5695363Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5695695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5696054Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5696434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:53:42.5696839Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.5697012Z 2025-08-14T21:53:42.5697110Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5697456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5697769Z return mod(**inputs) 2025-08-14T21:53:42.5698114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5698509Z outputs = self.model( 2025-08-14T21:53:42.5698858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5699230Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5699624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5699993Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5700321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5700656Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5701028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:53:42.5701435Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.5701798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:53:42.5702110Z return self.act(input) 2025-08-14T21:53:42.5702220Z 2025-08-14T21:53:42.5702339Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5702680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5703011Z return mod(**inputs) 2025-08-14T21:53:42.5703365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5703734Z outputs = self.model( 2025-08-14T21:53:42.5704084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5704451Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5704816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5705184Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5705505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5705847Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5706225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-08-14T21:53:42.5706603Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:53:42.5706731Z 2025-08-14T21:53:42.5706829Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5707168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5707474Z return mod(**inputs) 2025-08-14T21:53:42.5707812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5708191Z outputs = self.model( 2025-08-14T21:53:42.5708541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5708912Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5709275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5709638Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5709961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5710294Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5710647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5711021Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5711402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:42.5711844Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:42.5712039Z 2025-08-14T21:53:42.5712135Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5712487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5712794Z return mod(**inputs) 2025-08-14T21:53:42.5713139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5713510Z outputs = self.model( 2025-08-14T21:53:42.5713860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5714240Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5714593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5714954Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5715273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5715610Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5715990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5716371Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5716744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:42.5717109Z key_states = self.k_proj(current_states) 2025-08-14T21:53:42.5717239Z 2025-08-14T21:53:42.5717336Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5717666Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5717959Z return mod(**inputs) 2025-08-14T21:53:42.5718301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5718658Z outputs = self.model( 2025-08-14T21:53:42.5719021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5719383Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5719744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5720108Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5720422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5720758Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5721121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5721508Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5721890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:42.5722275Z value_states = self.v_proj(current_states) 2025-08-14T21:53:42.5722418Z 2025-08-14T21:53:42.5722498Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5722703Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5722897Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5723098Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5723322Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5723658Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5723971Z return mod(**inputs) 2025-08-14T21:53:42.5724328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5724709Z outputs = self.model( 2025-08-14T21:53:42.5725044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5725495Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5725922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5726311Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5726662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5727029Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5727409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5727796Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5728190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.5728615Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.5729072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:42.5729574Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:42.5729766Z 2025-08-14T21:53:42.5729869Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5730233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5730550Z return mod(**inputs) 2025-08-14T21:53:42.5730920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5731314Z outputs = self.model( 2025-08-14T21:53:42.5731678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5732070Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5732457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5732854Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5733192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5733550Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5733943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5734369Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5734765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.5735185Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.5735624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:42.5736081Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:42.5736242Z 2025-08-14T21:53:42.5736345Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5736701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5737007Z return mod(**inputs) 2025-08-14T21:53:42.5737349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5737879Z outputs = self.model( 2025-08-14T21:53:42.5738242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5738665Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5739027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5739403Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5739737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5740117Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5740503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5740898Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5741305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:42.5741681Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:42.5741820Z 2025-08-14T21:53:42.5741919Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5742265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5742575Z return mod(**inputs) 2025-08-14T21:53:42.5742949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5743362Z outputs = self.model( 2025-08-14T21:53:42.5743715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5744080Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5744445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5744815Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5745147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5745485Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5745865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:53:42.5746292Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.5746459Z 2025-08-14T21:53:42.5746562Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5746909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5747238Z return mod(**inputs) 2025-08-14T21:53:42.5747596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5747965Z outputs = self.model( 2025-08-14T21:53:42.5748334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5748709Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5749074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5749440Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5749770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5750114Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5750476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:53:42.5750891Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.5751255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:53:42.5751570Z return self.act(input) 2025-08-14T21:53:42.5751671Z 2025-08-14T21:53:42.5751768Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5752133Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5752435Z return mod(**inputs) 2025-08-14T21:53:42.5752772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5753164Z outputs = self.model( 2025-08-14T21:53:42.5753519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5753897Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5754257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5754632Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5754960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5755300Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5755689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-08-14T21:53:42.5756056Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:53:42.5756209Z 2025-08-14T21:53:42.5756316Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5756659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5756966Z return mod(**inputs) 2025-08-14T21:53:42.5757306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5757665Z outputs = self.model( 2025-08-14T21:53:42.5757999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5758363Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5758724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5759078Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5759398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5759739Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5760112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5760497Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5760885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:42.5761331Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:42.5761529Z 2025-08-14T21:53:42.5761640Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5761982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5762300Z return mod(**inputs) 2025-08-14T21:53:42.5762657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5763028Z outputs = self.model( 2025-08-14T21:53:42.5763386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5763778Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5764155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5764523Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5764859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5765244Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5765703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5766120Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5766542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:42.5766974Z key_states = self.k_proj(current_states) 2025-08-14T21:53:42.5767104Z 2025-08-14T21:53:42.5767203Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5767551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5767869Z return mod(**inputs) 2025-08-14T21:53:42.5768226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5768600Z outputs = self.model( 2025-08-14T21:53:42.5768962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5769343Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5769736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5770123Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5770484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5770841Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5771220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5771622Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5772021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:42.5772418Z value_states = self.v_proj(current_states) 2025-08-14T21:53:42.5772565Z 2025-08-14T21:53:42.5772646Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5772859Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5773064Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5773263Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5773492Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5773851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5774167Z return mod(**inputs) 2025-08-14T21:53:42.5774532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5774914Z outputs = self.model( 2025-08-14T21:53:42.5775272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5775655Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5776030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5776418Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5776749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5777107Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5777492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5777946Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5778363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.5778763Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.5779215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:42.5779673Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:42.5779847Z 2025-08-14T21:53:42.5779946Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5780315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5780628Z return mod(**inputs) 2025-08-14T21:53:42.5780976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5781356Z outputs = self.model( 2025-08-14T21:53:42.5781713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5782092Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5782455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5782830Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5783183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5783522Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5783920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5784299Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5784679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.5785063Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.5785481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:42.5785903Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:42.5786048Z 2025-08-14T21:53:42.5786151Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5786479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5786784Z return mod(**inputs) 2025-08-14T21:53:42.5787141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5787502Z outputs = self.model( 2025-08-14T21:53:42.5787853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5788222Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5788586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5788952Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5789274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5789607Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5789982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5790375Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5790750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:42.5791123Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:42.5791247Z 2025-08-14T21:53:42.5791344Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5791685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5791982Z return mod(**inputs) 2025-08-14T21:53:42.5792362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5792724Z outputs = self.model( 2025-08-14T21:53:42.5793092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5793493Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5793860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5794245Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5794589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5794932Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5795295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:53:42.5795703Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.5795860Z 2025-08-14T21:53:42.5795964Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5796356Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5796655Z return mod(**inputs) 2025-08-14T21:53:42.5797021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5797386Z outputs = self.model( 2025-08-14T21:53:42.5797720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5798087Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5798454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5798826Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5799154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5799496Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5799872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:53:42.5800278Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.5800645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:53:42.5800969Z return self.act(input) 2025-08-14T21:53:42.5801073Z 2025-08-14T21:53:42.5801185Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5801509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5801809Z return mod(**inputs) 2025-08-14T21:53:42.5802157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5802524Z outputs = self.model( 2025-08-14T21:53:42.5802867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5803243Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5803612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5803978Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5804309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5804648Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5805028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-08-14T21:53:42.5805460Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:53:42.5805636Z 2025-08-14T21:53:42.5805737Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5806111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5806433Z return mod(**inputs) 2025-08-14T21:53:42.5806814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5807249Z outputs = self.model( 2025-08-14T21:53:42.5807624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5807998Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5808369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5808747Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5809072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5809423Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5809819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5810225Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5810624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:42.5811066Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:42.5811267Z 2025-08-14T21:53:42.5811364Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5811703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5812006Z return mod(**inputs) 2025-08-14T21:53:42.5812356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5812723Z outputs = self.model( 2025-08-14T21:53:42.5813060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5813436Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5813804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5814182Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5814502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5814847Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5815221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5815605Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5815982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:42.5847935Z key_states = self.k_proj(current_states) 2025-08-14T21:53:42.5848299Z 2025-08-14T21:53:42.5848466Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5848853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5849190Z return mod(**inputs) 2025-08-14T21:53:42.5849594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5849993Z outputs = self.model( 2025-08-14T21:53:42.5850362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5850759Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5851164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5851732Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5852076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5852492Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5852894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5853298Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5853708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:42.5854175Z value_states = self.v_proj(current_states) 2025-08-14T21:53:42.5854312Z 2025-08-14T21:53:42.5854402Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5854606Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5854809Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5855011Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5855231Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5855618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5855938Z return mod(**inputs) 2025-08-14T21:53:42.5856322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5856705Z outputs = self.model( 2025-08-14T21:53:42.5857058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5857435Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5857798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5858178Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5858506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5858866Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5859248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5859638Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5860016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.5860410Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.5860832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:42.5861289Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:42.5861462Z 2025-08-14T21:53:42.5861565Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5861909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5862216Z return mod(**inputs) 2025-08-14T21:53:42.5862558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5862932Z outputs = self.model( 2025-08-14T21:53:42.5863286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5863658Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5864020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5864388Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5864719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5865095Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5865475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5865866Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5866276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.5866665Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.5867088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:42.5867521Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:42.5867674Z 2025-08-14T21:53:42.5867783Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5868119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5868426Z return mod(**inputs) 2025-08-14T21:53:42.5868779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5869945Z outputs = self.model( 2025-08-14T21:53:42.5870347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5870738Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5871099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5871455Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5871785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5872132Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5872495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 496, in forward 2025-08-14T21:53:42.5872885Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:53:42.5873268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:42.5873647Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:42.5873784Z 2025-08-14T21:53:42.5873882Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5874215Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5874516Z return mod(**inputs) 2025-08-14T21:53:42.5874850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5875199Z outputs = self.model( 2025-08-14T21:53:42.5875540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5875910Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5876256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5876618Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5876941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5877275Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5877629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:53:42.5878040Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.5878200Z 2025-08-14T21:53:42.5878304Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5878646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5878968Z return mod(**inputs) 2025-08-14T21:53:42.5879316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5879682Z outputs = self.model( 2025-08-14T21:53:42.5880028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5880414Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5880766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5881123Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5881437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5881768Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5882141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 507, in forward 2025-08-14T21:53:42.5882543Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.5882922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:53:42.5883251Z return self.act(input) 2025-08-14T21:53:42.5883355Z 2025-08-14T21:53:42.5883480Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5883828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5884140Z return mod(**inputs) 2025-08-14T21:53:42.5884503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5884880Z outputs = self.model( 2025-08-14T21:53:42.5885229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1189, in forward 2025-08-14T21:53:42.5885710Z encoder_outputs = self.encoder( 2025-08-14T21:53:42.5886100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 669, in forward 2025-08-14T21:53:42.5886484Z layer_outputs = encoder_layer( 2025-08-14T21:53:42.5886828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5887184Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5887574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 509, in forward 2025-08-14T21:53:42.5887959Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:53:42.5888102Z 2025-08-14T21:53:42.5888205Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5888556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5888866Z return mod(**inputs) 2025-08-14T21:53:42.5889228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5889606Z outputs = self.model( 2025-08-14T21:53:42.5889965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5890341Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5890721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5891107Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5891435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5891780Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5892167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.5892608Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.5893007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:42.5893470Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:42.5893694Z 2025-08-14T21:53:42.5893796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5894143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5894450Z return mod(**inputs) 2025-08-14T21:53:42.5894807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5895184Z outputs = self.model( 2025-08-14T21:53:42.5895535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5895919Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5896295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5896695Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5897031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5897397Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5897784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.5898190Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.5898583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:42.5898968Z key_states = self.k_proj(current_states) 2025-08-14T21:53:42.5899100Z 2025-08-14T21:53:42.5899208Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5899554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5899860Z return mod(**inputs) 2025-08-14T21:53:42.5900209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5900578Z outputs = self.model( 2025-08-14T21:53:42.5900919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5901289Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5901655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5902021Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5902350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5902690Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5903061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.5903449Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.5903843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:42.5904224Z value_states = self.v_proj(current_states) 2025-08-14T21:53:42.5904356Z 2025-08-14T21:53:42.5904442Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5904642Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5904843Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5905038Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5905250Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5905613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5905921Z return mod(**inputs) 2025-08-14T21:53:42.5906263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5906633Z outputs = self.model( 2025-08-14T21:53:42.5907006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5907385Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5907746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5908125Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5908456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5908789Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5909171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.5909569Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.5909985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.5910403Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.5910825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:42.5911280Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:42.5911450Z 2025-08-14T21:53:42.5911555Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5911889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5912197Z return mod(**inputs) 2025-08-14T21:53:42.5912545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5912910Z outputs = self.model( 2025-08-14T21:53:42.5913260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5913631Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5913997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5914361Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5914692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5915035Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5915414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.5915810Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.5916196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.5916597Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.5917008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:42.5917438Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:42.5917590Z 2025-08-14T21:53:42.5917689Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5918026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5918333Z return mod(**inputs) 2025-08-14T21:53:42.5918672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5919071Z outputs = self.model( 2025-08-14T21:53:42.5919416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5919781Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5920141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5920535Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5920864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5921205Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5921574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.5921971Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.5922360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:42.5922735Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:42.5922871Z 2025-08-14T21:53:42.5922969Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5923326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5923652Z return mod(**inputs) 2025-08-14T21:53:42.5923992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5924358Z outputs = self.model( 2025-08-14T21:53:42.5924708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5925073Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5925555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5925990Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5926360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5926735Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5927147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.5927570Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.5927973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:42.5928412Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:42.5928612Z 2025-08-14T21:53:42.5928712Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5929058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5929358Z return mod(**inputs) 2025-08-14T21:53:42.5929713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5930083Z outputs = self.model( 2025-08-14T21:53:42.5930435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5930797Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5931161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5931538Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5931862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5932202Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5932599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.5932996Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.5933388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:42.5933796Z key_states = self.k_proj(current_states) 2025-08-14T21:53:42.5933939Z 2025-08-14T21:53:42.5934036Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5934369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5934671Z return mod(**inputs) 2025-08-14T21:53:42.5935020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5935389Z outputs = self.model( 2025-08-14T21:53:42.5935728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5936107Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5936472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5936860Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5937206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5937550Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5938055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.5938463Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.5938861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:42.5939251Z value_states = self.v_proj(current_states) 2025-08-14T21:53:42.5939386Z 2025-08-14T21:53:42.5939473Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5939675Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5939877Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5940080Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5940300Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5940651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5940960Z return mod(**inputs) 2025-08-14T21:53:42.5941307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5941668Z outputs = self.model( 2025-08-14T21:53:42.5942020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5942396Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5942764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5943142Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5943472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5943817Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5944185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.5944586Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.5944984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.5945384Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.5945803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:42.5946306Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:42.5946481Z 2025-08-14T21:53:42.5946587Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5946933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5947270Z return mod(**inputs) 2025-08-14T21:53:42.5947617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5947987Z outputs = self.model( 2025-08-14T21:53:42.5948330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5948709Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5949080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5949451Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5949773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5950142Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5950544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.5950946Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.5951350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.5951452Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.5951724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:42.5951826Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:42.5951831Z 2025-08-14T21:53:42.5951939Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5952128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5952194Z return mod(**inputs) 2025-08-14T21:53:42.5952450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5952515Z outputs = self.model( 2025-08-14T21:53:42.5952767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5952837Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5953083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5953162Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5953373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5953456Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5953700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.5953801Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.5954053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:42.5954132Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:42.5954135Z 2025-08-14T21:53:42.5954232Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5954426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5954489Z return mod(**inputs) 2025-08-14T21:53:42.5954749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5954834Z outputs = self.model( 2025-08-14T21:53:42.5955071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5955151Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5955406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5955478Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5955679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5955750Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5955989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:42.5956098Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.5956103Z 2025-08-14T21:53:42.5956200Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5956391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5956482Z return mod(**inputs) 2025-08-14T21:53:42.5956743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5956807Z outputs = self.model( 2025-08-14T21:53:42.5957044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5957118Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5957352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5957417Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5957627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5957698Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5957940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:42.5958052Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.5958251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:53:42.5958323Z return self.act(input) 2025-08-14T21:53:42.5958327Z 2025-08-14T21:53:42.5958423Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5958614Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5958676Z return mod(**inputs) 2025-08-14T21:53:42.5958918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5958992Z outputs = self.model( 2025-08-14T21:53:42.5959232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5959303Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5959554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5959623Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5959837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5959910Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5960159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:53:42.5960243Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:53:42.5960262Z 2025-08-14T21:53:42.5960358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5960548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5960611Z return mod(**inputs) 2025-08-14T21:53:42.5960848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5960937Z outputs = self.model( 2025-08-14T21:53:42.5961176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5961243Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5961488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5961555Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5961764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5961839Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5962081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.5962198Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.5962463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:42.5962607Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:42.5962619Z 2025-08-14T21:53:42.5962716Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5962902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5962973Z return mod(**inputs) 2025-08-14T21:53:42.5963216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5963282Z outputs = self.model( 2025-08-14T21:53:42.5963534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5963605Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5963860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5963929Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5964135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5964219Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5964461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.5964556Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.5964805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:42.5964883Z key_states = self.k_proj(current_states) 2025-08-14T21:53:42.5964886Z 2025-08-14T21:53:42.5964991Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5965181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5965245Z return mod(**inputs) 2025-08-14T21:53:42.5965573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5965646Z outputs = self.model( 2025-08-14T21:53:42.5965904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5965977Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5966233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5966339Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5966566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5966646Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5966936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.5967034Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.5967293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:42.5967391Z value_states = self.v_proj(current_states) 2025-08-14T21:53:42.5967395Z 2025-08-14T21:53:42.5967473Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5967558Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5967634Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5967706Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5967813Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5968022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5968098Z return mod(**inputs) 2025-08-14T21:53:42.5968365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5968431Z outputs = self.model( 2025-08-14T21:53:42.5968681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5968751Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5968993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5969070Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5969277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5969359Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5969601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.5969694Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.5969942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.5970035Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.5970315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:42.5970441Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:42.5970445Z 2025-08-14T21:53:42.5970544Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5970736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5970801Z return mod(**inputs) 2025-08-14T21:53:42.5971045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5971119Z outputs = self.model( 2025-08-14T21:53:42.5971360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5971439Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5971684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5971752Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5971961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5972055Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5972301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.5972395Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.5972657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.5972755Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.5973030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:42.5973133Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:42.5973144Z 2025-08-14T21:53:42.5973242Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5973428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5973499Z return mod(**inputs) 2025-08-14T21:53:42.5973744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5973826Z outputs = self.model( 2025-08-14T21:53:42.5974081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5974168Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5974424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5974494Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5974702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5974784Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5975027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.5975122Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.5975374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:42.5975453Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:42.5975456Z 2025-08-14T21:53:42.5975563Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5975754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5975816Z return mod(**inputs) 2025-08-14T21:53:42.5976067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5976133Z outputs = self.model( 2025-08-14T21:53:42.5976373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5976453Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5976695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5976773Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5976984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5977058Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5977308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.5977409Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.5977671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:42.5977812Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:42.5977838Z 2025-08-14T21:53:42.5977935Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5978129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5978192Z return mod(**inputs) 2025-08-14T21:53:42.5978470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5978541Z outputs = self.model( 2025-08-14T21:53:42.5978782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5978860Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5979108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5979179Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5979399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5979480Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5979746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.5979848Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.5980105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:42.5980189Z key_states = self.k_proj(current_states) 2025-08-14T21:53:42.5980193Z 2025-08-14T21:53:42.5980288Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5980476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5980546Z return mod(**inputs) 2025-08-14T21:53:42.5980788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5980863Z outputs = self.model( 2025-08-14T21:53:42.5981107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5981179Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5981429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5981498Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5981710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5981784Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5982025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.5982135Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.5982377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:42.5982459Z value_states = self.v_proj(current_states) 2025-08-14T21:53:42.5982462Z 2025-08-14T21:53:42.5982551Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5982627Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5982709Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5982781Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.5982876Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5983076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5983139Z return mod(**inputs) 2025-08-14T21:53:42.5983386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5983478Z outputs = self.model( 2025-08-14T21:53:42.5983724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5983800Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5984047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5984134Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5984359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5984432Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5984668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.5984775Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.5985011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.5985109Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.5985389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:42.5985513Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:42.5985517Z 2025-08-14T21:53:42.5985633Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5985816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5985885Z return mod(**inputs) 2025-08-14T21:53:42.5986121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5986185Z outputs = self.model( 2025-08-14T21:53:42.5986429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5986499Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5986734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5986812Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5987017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5987096Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5987332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.5987428Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.5987685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.5987774Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.5988053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:42.5988150Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:42.5988154Z 2025-08-14T21:53:42.5988250Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5988442Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5988505Z return mod(**inputs) 2025-08-14T21:53:42.5988741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5988813Z outputs = self.model( 2025-08-14T21:53:42.5989053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5989131Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5989394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5989464Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5989682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5989776Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5990029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.5990130Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.5990372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:42.5990459Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:42.5990462Z 2025-08-14T21:53:42.5990559Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5990748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5990823Z return mod(**inputs) 2025-08-14T21:53:42.5991083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5991160Z outputs = self.model( 2025-08-14T21:53:42.5991422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5991496Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5991752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5991822Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5992028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5992110Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5992352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:42.5992474Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.5992477Z 2025-08-14T21:53:42.5992575Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5992766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5992838Z return mod(**inputs) 2025-08-14T21:53:42.5993082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5993154Z outputs = self.model( 2025-08-14T21:53:42.5993395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5993465Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5993728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5993794Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5993996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5994075Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5994312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:42.5994429Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.5994620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:53:42.5994685Z return self.act(input) 2025-08-14T21:53:42.5994688Z 2025-08-14T21:53:42.5994791Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5994974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5995061Z return mod(**inputs) 2025-08-14T21:53:42.5995297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5995362Z outputs = self.model( 2025-08-14T21:53:42.5995625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5995693Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5995928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5996004Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5996207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5996287Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5996528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:53:42.5996605Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:53:42.5996609Z 2025-08-14T21:53:42.5996727Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5996921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5997036Z return mod(**inputs) 2025-08-14T21:53:42.5997285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5997349Z outputs = self.model( 2025-08-14T21:53:42.5997597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5997664Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.5997911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.5997989Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.5998190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.5998270Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.5998505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.5998596Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.5998837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:42.5998973Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:42.5998977Z 2025-08-14T21:53:42.5999077Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.5999256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.5999318Z return mod(**inputs) 2025-08-14T21:53:42.5999561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.5999624Z outputs = self.model( 2025-08-14T21:53:42.5999861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.5999936Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6000169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6000243Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6000444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6000515Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6000790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.6000880Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.6001113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:42.6001211Z key_states = self.k_proj(current_states) 2025-08-14T21:53:42.6001216Z 2025-08-14T21:53:42.6001308Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6001500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6001559Z return mod(**inputs) 2025-08-14T21:53:42.6001800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6001871Z outputs = self.model( 2025-08-14T21:53:42.6002111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6002187Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6002455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6002528Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6002766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6002843Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6003093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.6003196Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.6003443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:42.6003537Z value_states = self.v_proj(current_states) 2025-08-14T21:53:42.6003541Z 2025-08-14T21:53:42.6003621Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6003699Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6003783Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6003869Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6003966Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6004160Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6004223Z return mod(**inputs) 2025-08-14T21:53:42.6004476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6004542Z outputs = self.model( 2025-08-14T21:53:42.6004788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6004868Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6005118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6005189Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6005482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6005569Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6005832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.6005929Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.6006238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.6006352Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.6006662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:42.6006844Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:42.6006848Z 2025-08-14T21:53:42.6006950Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6007145Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6007239Z return mod(**inputs) 2025-08-14T21:53:42.6007493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6007562Z outputs = self.model( 2025-08-14T21:53:42.6007821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6007895Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6008154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6008230Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6008445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6008544Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6008817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.6008922Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.6009175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.6009267Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.6009554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:42.6009659Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:42.6009663Z 2025-08-14T21:53:42.6009762Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6009965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6010030Z return mod(**inputs) 2025-08-14T21:53:42.6010291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6010357Z outputs = self.model( 2025-08-14T21:53:42.6010610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6010690Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6010939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6011016Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6011232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6011308Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6011568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.6011666Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.6011919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:42.6012006Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:42.6012010Z 2025-08-14T21:53:42.6012110Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6012310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6012374Z return mod(**inputs) 2025-08-14T21:53:42.6012624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6012717Z outputs = self.model( 2025-08-14T21:53:42.6012974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6013049Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6013336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6013409Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6013649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6013729Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6014010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6014130Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6014419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:42.6014581Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:42.6014603Z 2025-08-14T21:53:42.6014714Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6014939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6015020Z return mod(**inputs) 2025-08-14T21:53:42.6015298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6015370Z outputs = self.model( 2025-08-14T21:53:42.6015649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6015727Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6016010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6016088Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6016329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6016424Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6016760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6016879Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6017159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:42.6017243Z key_states = self.k_proj(current_states) 2025-08-14T21:53:42.6017247Z 2025-08-14T21:53:42.6017362Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6017572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6017641Z return mod(**inputs) 2025-08-14T21:53:42.6017922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6017994Z outputs = self.model( 2025-08-14T21:53:42.6018277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6018351Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6018600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6018678Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6018892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6018974Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6019244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6019347Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6019605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:42.6019709Z value_states = self.v_proj(current_states) 2025-08-14T21:53:42.6019712Z 2025-08-14T21:53:42.6019790Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6019877Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6019958Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6020046Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6020153Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6020361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6020442Z return mod(**inputs) 2025-08-14T21:53:42.6020711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6020782Z outputs = self.model( 2025-08-14T21:53:42.6021074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6021184Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6021464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6021539Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6021769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6021859Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6022114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6022216Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6022462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.6022556Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.6022847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:42.6022972Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:42.6022976Z 2025-08-14T21:53:42.6023074Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6023270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6023335Z return mod(**inputs) 2025-08-14T21:53:42.6023590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6023658Z outputs = self.model( 2025-08-14T21:53:42.6023907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6023991Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6024247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6024319Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6024555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6024638Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6024912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6025025Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6025318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.6025428Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.6025731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:42.6025874Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:42.6025878Z 2025-08-14T21:53:42.6025986Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6026192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6026270Z return mod(**inputs) 2025-08-14T21:53:42.6026540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6026612Z outputs = self.model( 2025-08-14T21:53:42.6026896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6026975Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6027275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6027355Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6027601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6027694Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6027966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6028084Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6028351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:42.6028438Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:42.6028442Z 2025-08-14T21:53:42.6028555Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6028765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6028835Z return mod(**inputs) 2025-08-14T21:53:42.6029118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6029189Z outputs = self.model( 2025-08-14T21:53:42.6029468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6029546Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6029817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6029901Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6030134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6030214Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6030505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:42.6030635Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.6030639Z 2025-08-14T21:53:42.6030752Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6030965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6031034Z return mod(**inputs) 2025-08-14T21:53:42.6031318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6031389Z outputs = self.model( 2025-08-14T21:53:42.6031689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6031765Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6032036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6032138Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6032368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6032450Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6032736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:42.6032860Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.6033088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:53:42.6033164Z return self.act(input) 2025-08-14T21:53:42.6033168Z 2025-08-14T21:53:42.6033274Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6033492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6033578Z return mod(**inputs) 2025-08-14T21:53:42.6033882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6033956Z outputs = self.model( 2025-08-14T21:53:42.6034229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6034315Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6034586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6034664Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6034906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6034988Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6035278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:53:42.6035366Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:53:42.6035370Z 2025-08-14T21:53:42.6035478Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6035698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6035768Z return mod(**inputs) 2025-08-14T21:53:42.6036042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6036121Z outputs = self.model( 2025-08-14T21:53:42.6036392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6036488Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6036759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6036838Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6037082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6037164Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6037449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.6037553Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.6037968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:42.6038142Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:42.6038195Z 2025-08-14T21:53:42.6038305Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6038519Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6038599Z return mod(**inputs) 2025-08-14T21:53:42.6038909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6038991Z outputs = self.model( 2025-08-14T21:53:42.6039266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6039347Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6039630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6039709Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6039951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6040038Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6040333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.6040450Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.6040741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:42.6040829Z key_states = self.k_proj(current_states) 2025-08-14T21:53:42.6040832Z 2025-08-14T21:53:42.6040946Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6041157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6041234Z return mod(**inputs) 2025-08-14T21:53:42.6041505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6041577Z outputs = self.model( 2025-08-14T21:53:42.6041857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6041936Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6042209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6042293Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6042522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6042611Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6042880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.6042983Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.6043263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:42.6043352Z value_states = self.v_proj(current_states) 2025-08-14T21:53:42.6043358Z 2025-08-14T21:53:42.6043450Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6043536Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6043619Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6043704Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6043811Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6044019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6044095Z return mod(**inputs) 2025-08-14T21:53:42.6044369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6044468Z outputs = self.model( 2025-08-14T21:53:42.6044740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6044819Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6045097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6045193Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6045490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6045586Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6045854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.6045963Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.6046231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.6046333Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.6046679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:42.6046820Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:42.6046824Z 2025-08-14T21:53:42.6046956Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6047167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6047238Z return mod(**inputs) 2025-08-14T21:53:42.6047520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6047591Z outputs = self.model( 2025-08-14T21:53:42.6047866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6047959Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6048203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6048282Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6048492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6048567Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6048816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.6048907Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.6049154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.6049246Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.6049519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:42.6049630Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:42.6049633Z 2025-08-14T21:53:42.6049731Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6049918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6049988Z return mod(**inputs) 2025-08-14T21:53:42.6050233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6050303Z outputs = self.model( 2025-08-14T21:53:42.6050544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6050613Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6050883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6050952Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6051158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6051257Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6051505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.6051605Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.6051849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:42.6051925Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:42.6051928Z 2025-08-14T21:53:42.6052033Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6052224Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6052291Z return mod(**inputs) 2025-08-14T21:53:42.6052551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6052619Z outputs = self.model( 2025-08-14T21:53:42.6052886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6052957Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6053200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6053277Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6053483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6053564Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6053809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6053909Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6054165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:42.6054309Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:42.6054312Z 2025-08-14T21:53:42.6054414Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6054602Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6054682Z return mod(**inputs) 2025-08-14T21:53:42.6054936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6055000Z outputs = self.model( 2025-08-14T21:53:42.6055244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6055322Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6055569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6055648Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6055856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6055929Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6056178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6056279Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6056534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:42.6056635Z key_states = self.k_proj(current_states) 2025-08-14T21:53:42.6056639Z 2025-08-14T21:53:42.6056736Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6056939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6057019Z return mod(**inputs) 2025-08-14T21:53:42.6057271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6057346Z outputs = self.model( 2025-08-14T21:53:42.6057595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6057673Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6057921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6057993Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6058214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6058288Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6058555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6058692Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6058936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:42.6059024Z value_states = self.v_proj(current_states) 2025-08-14T21:53:42.6059028Z 2025-08-14T21:53:42.6059103Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6059179Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6059260Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6059331Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6059436Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6059625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6059687Z return mod(**inputs) 2025-08-14T21:53:42.6059941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6060006Z outputs = self.model( 2025-08-14T21:53:42.6060249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6060327Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6060569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6060642Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6060849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6060924Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6061175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6061276Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6061516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.6061615Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.6061887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:42.6062016Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:42.6062019Z 2025-08-14T21:53:42.6062115Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6062327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6062398Z return mod(**inputs) 2025-08-14T21:53:42.6062641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6062714Z outputs = self.model( 2025-08-14T21:53:42.6062975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6063045Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6063293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6063362Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6063569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6063652Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6063893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6063999Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6064256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.6064365Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.6064647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:42.6064746Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:42.6064750Z 2025-08-14T21:53:42.6064855Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6065044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6065108Z return mod(**inputs) 2025-08-14T21:53:42.6065359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6065424Z outputs = self.model( 2025-08-14T21:53:42.6065666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6065749Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6065992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6066074Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6066285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6066361Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6066609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6066712Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6066963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:42.6067041Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:42.6067046Z 2025-08-14T21:53:42.6067142Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6067340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6067404Z return mod(**inputs) 2025-08-14T21:53:42.6067645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6067717Z outputs = self.model( 2025-08-14T21:53:42.6067969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6068073Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6068313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6068382Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6068596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6068700Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6068936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:42.6069055Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.6069058Z 2025-08-14T21:53:42.6069153Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6069343Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6069404Z return mod(**inputs) 2025-08-14T21:53:42.6069644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6069714Z outputs = self.model( 2025-08-14T21:53:42.6069967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6070046Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6070304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6070376Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6070595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6070670Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6070913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:42.6071035Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.6071247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:53:42.6071319Z return self.act(input) 2025-08-14T21:53:42.6071324Z 2025-08-14T21:53:42.6071419Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6071604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6071674Z return mod(**inputs) 2025-08-14T21:53:42.6071914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6071977Z outputs = self.model( 2025-08-14T21:53:42.6072222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6072291Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6072537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6072605Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6072810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6072892Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6073131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:53:42.6073213Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:53:42.6073216Z 2025-08-14T21:53:42.6073309Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6073491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6073561Z return mod(**inputs) 2025-08-14T21:53:42.6073803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6073887Z outputs = self.model( 2025-08-14T21:53:42.6074134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6074238Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6074482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6074549Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6074750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6074828Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6075061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.6075159Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.6075394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:42.6075531Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:42.6075550Z 2025-08-14T21:53:42.6075656Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6075850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6075915Z return mod(**inputs) 2025-08-14T21:53:42.6076165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6076227Z outputs = self.model( 2025-08-14T21:53:42.6076475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6076544Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6076785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6076864Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6077076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6077173Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6077416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.6077510Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.6077752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:42.6077825Z key_states = self.k_proj(current_states) 2025-08-14T21:53:42.6077828Z 2025-08-14T21:53:42.6077921Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6078111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6078171Z return mod(**inputs) 2025-08-14T21:53:42.6078418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6078483Z outputs = self.model( 2025-08-14T21:53:42.6078721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6078797Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6079032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6079098Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6079307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6079397Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6079640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.6079731Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.6079969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:42.6080074Z value_states = self.v_proj(current_states) 2025-08-14T21:53:42.6080077Z 2025-08-14T21:53:42.6080154Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6080238Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6080311Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6080384Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6080489Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6080673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6080735Z return mod(**inputs) 2025-08-14T21:53:42.6080982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6081047Z outputs = self.model( 2025-08-14T21:53:42.6081306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6081395Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6081632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6081707Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6081910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6081983Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6082235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.6082329Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.6082578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.6082672Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.6082949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:42.6083079Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:42.6083082Z 2025-08-14T21:53:42.6083314Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6083511Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6083574Z return mod(**inputs) 2025-08-14T21:53:42.6083816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6083890Z outputs = self.model( 2025-08-14T21:53:42.6084133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6084205Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6084458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6084528Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6084748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6084824Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6085073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.6085177Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.6085517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.6085629Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.6085924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:42.6086050Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:42.6086053Z 2025-08-14T21:53:42.6086160Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6086347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6086411Z return mod(**inputs) 2025-08-14T21:53:42.6086669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6086736Z outputs = self.model( 2025-08-14T21:53:42.6087000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6087075Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6087341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6087424Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6087659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6087749Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6088001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.6088095Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.6088363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:42.6088442Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:42.6088446Z 2025-08-14T21:53:42.6088541Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6088733Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6088798Z return mod(**inputs) 2025-08-14T21:53:42.6089051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6089116Z outputs = self.model( 2025-08-14T21:53:42.6089360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6089436Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6089678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6089749Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6089964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6090036Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6090288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6090394Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6090637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:42.6090783Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:42.6090786Z 2025-08-14T21:53:42.6090882Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6091075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6091168Z return mod(**inputs) 2025-08-14T21:53:42.6091419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6091491Z outputs = self.model( 2025-08-14T21:53:42.6091742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6091828Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6092078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6092148Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6092364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6092437Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6092675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6092786Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6093030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:42.6093157Z key_states = self.k_proj(current_states) 2025-08-14T21:53:42.6093162Z 2025-08-14T21:53:42.6093260Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6093469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6093542Z return mod(**inputs) 2025-08-14T21:53:42.6093785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6093849Z outputs = self.model( 2025-08-14T21:53:42.6094100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6094173Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6094422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6094490Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6094698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6094794Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6095036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6095146Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6095385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:42.6095468Z value_states = self.v_proj(current_states) 2025-08-14T21:53:42.6095471Z 2025-08-14T21:53:42.6095558Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6095635Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6095711Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6095794Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6095892Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6096092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6096155Z return mod(**inputs) 2025-08-14T21:53:42.6096401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6096471Z outputs = self.model( 2025-08-14T21:53:42.6096712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6096783Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6097033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6097122Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6097341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6097417Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6097679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6097789Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6098033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.6098124Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.6098403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:42.6098527Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:42.6098530Z 2025-08-14T21:53:42.6098632Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6098836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6098902Z return mod(**inputs) 2025-08-14T21:53:42.6099168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6099233Z outputs = self.model( 2025-08-14T21:53:42.6099499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6099569Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6099802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6099877Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6100079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6100152Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6100393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6100495Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6100735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.6100824Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.6101084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:42.6101188Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:42.6101191Z 2025-08-14T21:53:42.6101285Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6101473Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6101534Z return mod(**inputs) 2025-08-14T21:53:42.6101768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6101838Z outputs = self.model( 2025-08-14T21:53:42.6102072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6102140Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6102379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6102447Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6102656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6102759Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6102993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6103099Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6103350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:42.6103432Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:42.6103436Z 2025-08-14T21:53:42.6103530Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6103710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6103778Z return mod(**inputs) 2025-08-14T21:53:42.6104015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6104080Z outputs = self.model( 2025-08-14T21:53:42.6104324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6104392Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6104653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6104736Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6104941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6105021Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6105257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:42.6105374Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.6105378Z 2025-08-14T21:53:42.6105474Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6105656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6105726Z return mod(**inputs) 2025-08-14T21:53:42.6105964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6106029Z outputs = self.model( 2025-08-14T21:53:42.6106275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6106343Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6106590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6106656Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6106861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6106944Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6107179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:42.6107291Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.6107495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:53:42.6107559Z return self.act(input) 2025-08-14T21:53:42.6107562Z 2025-08-14T21:53:42.6107665Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6107846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6107908Z return mod(**inputs) 2025-08-14T21:53:42.6108154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6108235Z outputs = self.model( 2025-08-14T21:53:42.6108482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6108551Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6108788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6108884Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6109088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6109161Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6109404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:53:42.6109479Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:53:42.6109482Z 2025-08-14T21:53:42.6109582Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6109767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6109829Z return mod(**inputs) 2025-08-14T21:53:42.6110088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6110154Z outputs = self.model( 2025-08-14T21:53:42.6110413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6110490Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6110727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6110801Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6111005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6111076Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6111325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.6111416Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.6111668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:42.6111809Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:42.6111812Z 2025-08-14T21:53:42.6111907Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6112099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6112160Z return mod(**inputs) 2025-08-14T21:53:42.6112407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6112480Z outputs = self.model( 2025-08-14T21:53:42.6112727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6112804Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6113053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6113121Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6113338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6113411Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6113662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.6113756Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.6114005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:42.6114106Z key_states = self.k_proj(current_states) 2025-08-14T21:53:42.6114110Z 2025-08-14T21:53:42.6114203Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6114387Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6114472Z return mod(**inputs) 2025-08-14T21:53:42.6114711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6114782Z outputs = self.model( 2025-08-14T21:53:42.6115017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6115084Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6115326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6115393Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6115602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6115672Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6115922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.6116037Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.6116271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:42.6116351Z value_states = self.v_proj(current_states) 2025-08-14T21:53:42.6116354Z 2025-08-14T21:53:42.6116440Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6116518Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6116601Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6116674Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6116773Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6116967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6117030Z return mod(**inputs) 2025-08-14T21:53:42.6117274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6117350Z outputs = self.model( 2025-08-14T21:53:42.6117587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6117663Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6117903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6117975Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6118187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6118262Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6118495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.6118594Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.6118831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.6118927Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.6119191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:42.6119310Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:42.6119313Z 2025-08-14T21:53:42.6119418Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6119625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6119694Z return mod(**inputs) 2025-08-14T21:53:42.6119931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6120012Z outputs = self.model( 2025-08-14T21:53:42.6120258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6120326Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6120560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6120637Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6120838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6120919Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6121158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.6121246Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.6121509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.6121616Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.6121927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:42.6122025Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:42.6122028Z 2025-08-14T21:53:42.6122123Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6122314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6122375Z return mod(**inputs) 2025-08-14T21:53:42.6122615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6122686Z outputs = self.model( 2025-08-14T21:53:42.6122923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6123000Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6123237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6123306Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6123515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6123587Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6123831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 760, in forward 2025-08-14T21:53:42.6123921Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:53:42.6124157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:42.6124241Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:42.6124245Z 2025-08-14T21:53:42.6124337Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6124549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6124622Z return mod(**inputs) 2025-08-14T21:53:42.6124863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6124933Z outputs = self.model( 2025-08-14T21:53:42.6125175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6125270Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6125618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6125699Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6125934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6126051Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6126340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6126464Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6126738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 400, in forward 2025-08-14T21:53:42.6126880Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:53:42.6126885Z 2025-08-14T21:53:42.6126992Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6127184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6127256Z return mod(**inputs) 2025-08-14T21:53:42.6127521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6127606Z outputs = self.model( 2025-08-14T21:53:42.6127860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6127932Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6128173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6128250Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6128459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6128542Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6128786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6128889Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6129146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 419, in forward 2025-08-14T21:53:42.6129222Z key_states = self.k_proj(current_states) 2025-08-14T21:53:42.6129225Z 2025-08-14T21:53:42.6129328Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6129514Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6129576Z return mod(**inputs) 2025-08-14T21:53:42.6129826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6129891Z outputs = self.model( 2025-08-14T21:53:42.6130133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6130208Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6130452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6130531Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6130737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6130809Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6131057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6131155Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6131406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 420, in forward 2025-08-14T21:53:42.6131507Z value_states = self.v_proj(current_states) 2025-08-14T21:53:42.6131510Z 2025-08-14T21:53:42.6131586Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6131668Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6131761Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6131837Z cudagraph partition due to non gpu ops 2025-08-14T21:53:42.6131946Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6132138Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6132208Z return mod(**inputs) 2025-08-14T21:53:42.6132457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6132522Z outputs = self.model( 2025-08-14T21:53:42.6132777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6132850Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6133122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6133202Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6133427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6133512Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6133761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6133865Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6134120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.6134215Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.6134501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:53:42.6134632Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:53:42.6134637Z 2025-08-14T21:53:42.6134733Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6134931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6134995Z return mod(**inputs) 2025-08-14T21:53:42.6135245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6135317Z outputs = self.model( 2025-08-14T21:53:42.6135564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6135644Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6135893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6135963Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6136191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6136266Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6136508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6136614Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6136857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 438, in forward 2025-08-14T21:53:42.6136956Z attn_output, attn_weights = attention_interface( 2025-08-14T21:53:42.6137230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:53:42.6137349Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:53:42.6137352Z 2025-08-14T21:53:42.6137458Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6137796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6137925Z return mod(**inputs) 2025-08-14T21:53:42.6138201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6138271Z outputs = self.model( 2025-08-14T21:53:42.6138545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6138620Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6138887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6138973Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6139197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6139311Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6139655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 777, in forward 2025-08-14T21:53:42.6139760Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:53:42.6140010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 452, in forward 2025-08-14T21:53:42.6140090Z attn_output = self.out_proj(attn_output) 2025-08-14T21:53:42.6140094Z 2025-08-14T21:53:42.6140200Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6140385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6140449Z return mod(**inputs) 2025-08-14T21:53:42.6140701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6140766Z outputs = self.model( 2025-08-14T21:53:42.6141010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6141093Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6141337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6141412Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6141619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6141691Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6141943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:42.6142056Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.6142060Z 2025-08-14T21:53:42.6142164Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6142352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6142418Z return mod(**inputs) 2025-08-14T21:53:42.6142683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6142748Z outputs = self.model( 2025-08-14T21:53:42.6142991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6143067Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6143309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6143413Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6143622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6143696Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6145079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 792, in forward 2025-08-14T21:53:42.6145192Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:53:42.6145391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:53:42.6145469Z return self.act(input) 2025-08-14T21:53:42.6145472Z 2025-08-14T21:53:42.6145568Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6145768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6145833Z return mod(**inputs) 2025-08-14T21:53:42.6146080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1359, in forward 2025-08-14T21:53:42.6146153Z outputs = self.model( 2025-08-14T21:53:42.6146413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1207, in forward 2025-08-14T21:53:42.6146495Z decoder_outputs = self.decoder( 2025-08-14T21:53:42.6146752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1031, in forward 2025-08-14T21:53:42.6146824Z layer_outputs = decoder_layer( 2025-08-14T21:53:42.6147041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:53:42.6147115Z return super().__call__(*args, **kwargs) 2025-08-14T21:53:42.6147363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 794, in forward 2025-08-14T21:53:42.6147447Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:53:42.6147450Z 2025-08-14T21:53:42.6147543Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6147731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6147793Z return mod(**inputs) 2025-08-14T21:53:42.6148033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1377, in forward 2025-08-14T21:53:42.6148117Z lm_logits = self.lm_head(outputs[0]) 2025-08-14T21:53:42.6148120Z 2025-08-14T21:53:42.6148212Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:53:42.6148399Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:53:42.6148459Z return mod(**inputs) 2025-08-14T21:53:42.6148698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/plbart/modeling_plbart.py", line 1383, in forward 2025-08-14T21:53:42.6148860Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:53:42.6148866Z 2025-08-14T21:53:52.0005017Z Compilation time (from dynamo_timed): 17.03614773 2025-08-14T21:53:52.0261819Z pass 2025-08-14T21:53:52.0266450Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:53:52.0271543Z TIMING: _recursive_pre_grad_passes:0.00935 _recursive_joint_graph_passes:0.4336 _recursive_post_grad_passes:0.11009 async_compile.wait:0.74308 code_gen:8.77801 inductor_compile:10.29865 backend_compile:14.20854 gc:0.00214 entire_frame_compile:17.03615 total_wall_time:17.03615 2025-08-14T21:53:52.0272889Z STATS: call_* op count: 517 | FakeTensorMode.__torch_dispatch__:17501 | FakeTensor.__torch_dispatch__:6218 | ProxyTorchDispatchMode.__torch_dispatch__:6406 2025-08-14T21:53:52.0273397Z Dynamo produced 1 graphs covering 517 ops with 0 graph breaks (0 unique) 2025-08-14T21:53:57.4922955Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:53:57.4924162Z from pkg_resources import resource_filename 2025-08-14T21:53:58.2380425Z 2025-08-14T21:54:01.7028886Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:54:01.7031601Z loading model: 0it [00:03, ?it/s] 2025-08-14T21:54:01.7046162Z cpu eval PegasusForCausalLM 2025-08-14T21:54:02.0672667Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:54:02.2115855Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:54:02.3525895Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:54:09.7140750Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7146532Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7152078Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7152862Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7153208Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7153977Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7160324Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7160614Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7160831Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7161030Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7161240Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7161446Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7161694Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7164112Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7164664Z return mod(**inputs) 2025-08-14T21:54:09.7165141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7165737Z outputs = self.model.decoder( 2025-08-14T21:54:09.7166190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7166629Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7167054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7167429Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7168001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7168447Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7168873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:09.7169348Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:09.7169564Z 2025-08-14T21:54:09.7169681Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7170057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7170382Z return mod(**inputs) 2025-08-14T21:54:09.7170767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7171171Z outputs = self.model.decoder( 2025-08-14T21:54:09.7171565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7171957Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7173433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7173815Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7174238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7174770Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7175214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:09.7175649Z key_states = self.k_proj(current_states) 2025-08-14T21:54:09.7175792Z 2025-08-14T21:54:09.7175908Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7176295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7176624Z return mod(**inputs) 2025-08-14T21:54:09.7177003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7177401Z outputs = self.model.decoder( 2025-08-14T21:54:09.7177826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7178229Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7178601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7178973Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7179385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7179812Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7180227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:09.7180647Z value_states = self.v_proj(current_states) 2025-08-14T21:54:09.7180791Z 2025-08-14T21:54:09.7180886Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7181099Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7181308Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7181724Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7181962Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7182326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7182651Z return mod(**inputs) 2025-08-14T21:54:09.7183221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7183640Z outputs = self.model.decoder( 2025-08-14T21:54:09.7184049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7184450Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7184802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7185154Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7185557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7185983Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7186407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7186813Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7187247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:09.7187723Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:09.7187935Z 2025-08-14T21:54:09.7188051Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7188409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7188742Z return mod(**inputs) 2025-08-14T21:54:09.7189122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7189547Z outputs = self.model.decoder( 2025-08-14T21:54:09.7189946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7190335Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7190673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7191016Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7191405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7191815Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7192232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7192642Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7193097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:09.7193547Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:09.7193704Z 2025-08-14T21:54:09.7193805Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7194156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7194473Z return mod(**inputs) 2025-08-14T21:54:09.7194840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7195239Z outputs = self.model.decoder( 2025-08-14T21:54:09.7195633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7196036Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7196383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7196751Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7197155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7197568Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7197965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:09.7198363Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:09.7198495Z 2025-08-14T21:54:09.7198602Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7198950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7199259Z return mod(**inputs) 2025-08-14T21:54:09.7199628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7200019Z outputs = self.model.decoder( 2025-08-14T21:54:09.7200400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7200802Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7201147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7201511Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7201931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7202383Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7202560Z 2025-08-14T21:54:09.7202673Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7203046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7203371Z return mod(**inputs) 2025-08-14T21:54:09.7203746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7204146Z outputs = self.model.decoder( 2025-08-14T21:54:09.7204583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7205000Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7205440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7205842Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7206281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7206761Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7207174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:09.7207527Z return self.act(input) 2025-08-14T21:54:09.7207639Z 2025-08-14T21:54:09.7207744Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7208107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7208432Z return mod(**inputs) 2025-08-14T21:54:09.7208811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7209207Z outputs = self.model.decoder( 2025-08-14T21:54:09.7209596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7209995Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7210333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7210695Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7211093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:09.7211497Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:09.7211633Z 2025-08-14T21:54:09.7211737Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7212095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7212418Z return mod(**inputs) 2025-08-14T21:54:09.7212763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7213140Z outputs = self.model.decoder( 2025-08-14T21:54:09.7213511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7213892Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7214212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7214553Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7214931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7215332Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7215718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:09.7216194Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:09.7216388Z 2025-08-14T21:54:09.7216493Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7216823Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7217155Z return mod(**inputs) 2025-08-14T21:54:09.7217511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7217891Z outputs = self.model.decoder( 2025-08-14T21:54:09.7218255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7218637Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7218970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7219316Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7219692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7220206Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7220629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:09.7221012Z key_states = self.k_proj(current_states) 2025-08-14T21:54:09.7221150Z 2025-08-14T21:54:09.7221250Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7221595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7221908Z return mod(**inputs) 2025-08-14T21:54:09.7222259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7222645Z outputs = self.model.decoder( 2025-08-14T21:54:09.7223026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7223412Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7223742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7224084Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7224587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7224995Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7225406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:09.7225816Z value_states = self.v_proj(current_states) 2025-08-14T21:54:09.7225951Z 2025-08-14T21:54:09.7226042Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7226245Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7226449Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7226651Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7226871Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7227220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7227534Z return mod(**inputs) 2025-08-14T21:54:09.7227888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7228271Z outputs = self.model.decoder( 2025-08-14T21:54:09.7228644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7229019Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7229371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7229718Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7230100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7230522Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7230911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7231307Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7231722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:09.7232174Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:09.7232347Z 2025-08-14T21:54:09.7232443Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7232785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7233094Z return mod(**inputs) 2025-08-14T21:54:09.7233462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7233844Z outputs = self.model.decoder( 2025-08-14T21:54:09.7234237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7234623Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7234946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7235287Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7235677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7236081Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7236471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7236870Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7237289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:09.7237944Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:09.7238113Z 2025-08-14T21:54:09.7238212Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7238561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7238874Z return mod(**inputs) 2025-08-14T21:54:09.7239229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7239621Z outputs = self.model.decoder( 2025-08-14T21:54:09.7240004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7240384Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7240725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7241077Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7241461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7241855Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7242252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:09.7242643Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:09.7242838Z 2025-08-14T21:54:09.7242945Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7243278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7243590Z return mod(**inputs) 2025-08-14T21:54:09.7243948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7244367Z outputs = self.model.decoder( 2025-08-14T21:54:09.7244761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7245156Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7245578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7245931Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7246318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7246740Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7246903Z 2025-08-14T21:54:09.7247007Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7247379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7247695Z return mod(**inputs) 2025-08-14T21:54:09.7248082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7248458Z outputs = self.model.decoder( 2025-08-14T21:54:09.7248838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7249216Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7249543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7249881Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7250262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7250692Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7251060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:09.7251390Z return self.act(input) 2025-08-14T21:54:09.7251505Z 2025-08-14T21:54:09.7251602Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7251943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7252274Z return mod(**inputs) 2025-08-14T21:54:09.7252632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7253019Z outputs = self.model.decoder( 2025-08-14T21:54:09.7253382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7253768Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7254103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7254451Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7254826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:09.7255212Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:09.7255345Z 2025-08-14T21:54:09.7255454Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7255801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7256105Z return mod(**inputs) 2025-08-14T21:54:09.7256492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7256871Z outputs = self.model.decoder( 2025-08-14T21:54:09.7257236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7257673Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7258010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7258365Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7258750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7259163Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7259572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:09.7260025Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:09.7260217Z 2025-08-14T21:54:09.7260315Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7260672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7260986Z return mod(**inputs) 2025-08-14T21:54:09.7261355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7261733Z outputs = self.model.decoder( 2025-08-14T21:54:09.7262099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7262472Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7262793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7263137Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7263517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7263905Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7264298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:09.7264683Z key_states = self.k_proj(current_states) 2025-08-14T21:54:09.7264810Z 2025-08-14T21:54:09.7264912Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7265237Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7265540Z return mod(**inputs) 2025-08-14T21:54:09.7265893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7266273Z outputs = self.model.decoder( 2025-08-14T21:54:09.7266638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7267014Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7267342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7267676Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7268051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7268456Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7268854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:09.7269234Z value_states = self.v_proj(current_states) 2025-08-14T21:54:09.7269375Z 2025-08-14T21:54:09.7269484Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7269691Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7269885Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7270088Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7270312Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7270695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7270998Z return mod(**inputs) 2025-08-14T21:54:09.7271356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7271751Z outputs = self.model.decoder( 2025-08-14T21:54:09.7272113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7272493Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7272822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7273162Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7273533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7273956Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7274375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7274775Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7275189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:09.7275651Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:09.7275827Z 2025-08-14T21:54:09.7275933Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7276269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7276577Z return mod(**inputs) 2025-08-14T21:54:09.7276933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7277316Z outputs = self.model.decoder( 2025-08-14T21:54:09.7277682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7278070Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7278390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7278722Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7279086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7279487Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7279886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7280280Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7280700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:09.7281147Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:09.7281295Z 2025-08-14T21:54:09.7281399Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7281728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7282039Z return mod(**inputs) 2025-08-14T21:54:09.7282395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7282798Z outputs = self.model.decoder( 2025-08-14T21:54:09.7283161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7283540Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7283867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7284221Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7284606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7285007Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7285518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:09.7285934Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:09.7286078Z 2025-08-14T21:54:09.7286182Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7286536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7286849Z return mod(**inputs) 2025-08-14T21:54:09.7287242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7287630Z outputs = self.model.decoder( 2025-08-14T21:54:09.7288032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7288409Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7288748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7289079Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7289450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7289851Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7290020Z 2025-08-14T21:54:09.7290115Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7290445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7290741Z return mod(**inputs) 2025-08-14T21:54:09.7291098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7291476Z outputs = self.model.decoder( 2025-08-14T21:54:09.7291870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7292304Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7292652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7292992Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7293368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7293787Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7294157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:09.7294482Z return self.act(input) 2025-08-14T21:54:09.7294588Z 2025-08-14T21:54:09.7294686Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7295031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7295330Z return mod(**inputs) 2025-08-14T21:54:09.7295681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7296050Z outputs = self.model.decoder( 2025-08-14T21:54:09.7296449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7296829Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7297152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7297514Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7297892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:09.7298278Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:09.7298407Z 2025-08-14T21:54:09.7298503Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7298842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7299150Z return mod(**inputs) 2025-08-14T21:54:09.7299494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7299872Z outputs = self.model.decoder( 2025-08-14T21:54:09.7300243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7300642Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7300981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7301329Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7301708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-14T21:54:09.7302091Z hidden_states = residual + hidden_states 2025-08-14T21:54:09.7302218Z 2025-08-14T21:54:09.7302314Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7302656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7302973Z return mod(**inputs) 2025-08-14T21:54:09.7303317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7303698Z outputs = self.model.decoder( 2025-08-14T21:54:09.7304075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7304456Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7304774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7305115Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7305492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7305887Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7306285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:09.7306733Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:09.7306929Z 2025-08-14T21:54:09.7307033Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7307371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7307678Z return mod(**inputs) 2025-08-14T21:54:09.7308061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7308445Z outputs = self.model.decoder( 2025-08-14T21:54:09.7308810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7309192Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7309524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7309887Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7310258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7310672Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7311069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:09.7311438Z key_states = self.k_proj(current_states) 2025-08-14T21:54:09.7311573Z 2025-08-14T21:54:09.7311668Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7312001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7312300Z return mod(**inputs) 2025-08-14T21:54:09.7312639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7313015Z outputs = self.model.decoder( 2025-08-14T21:54:09.7313374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7313757Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7314100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7314431Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7314798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7315177Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7315559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:09.7315940Z value_states = self.v_proj(current_states) 2025-08-14T21:54:09.7316073Z 2025-08-14T21:54:09.7316155Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7316349Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7316548Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7316744Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7316954Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7317291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7317596Z return mod(**inputs) 2025-08-14T21:54:09.7317941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7318325Z outputs = self.model.decoder( 2025-08-14T21:54:09.7318699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7319079Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7319407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7319743Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7320121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7320524Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7320912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7321311Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7321731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:09.7322177Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:09.7322403Z 2025-08-14T21:54:09.7322502Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7322845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7323158Z return mod(**inputs) 2025-08-14T21:54:09.7323520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7323930Z outputs = self.model.decoder( 2025-08-14T21:54:09.7324299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7324673Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7324996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7325422Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7325850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7326271Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7326691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7327115Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7327567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:09.7327998Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:09.7328169Z 2025-08-14T21:54:09.7328266Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7328605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7328907Z return mod(**inputs) 2025-08-14T21:54:09.7329261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7329643Z outputs = self.model.decoder( 2025-08-14T21:54:09.7330008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7330378Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7330706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7331048Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7331427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7331823Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7332215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:09.7332604Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:09.7332731Z 2025-08-14T21:54:09.7332828Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7333170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7333480Z return mod(**inputs) 2025-08-14T21:54:09.7333834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7334211Z outputs = self.model.decoder( 2025-08-14T21:54:09.7334581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7334962Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7335289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7335629Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7336013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7336472Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7336632Z 2025-08-14T21:54:09.7336732Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7337103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7337413Z return mod(**inputs) 2025-08-14T21:54:09.7338017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7338417Z outputs = self.model.decoder( 2025-08-14T21:54:09.7338796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7339216Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7339546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7339904Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7340291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7340783Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7341173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:09.7341507Z return self.act(input) 2025-08-14T21:54:09.7341615Z 2025-08-14T21:54:09.7341724Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7342106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7342405Z return mod(**inputs) 2025-08-14T21:54:09.7342762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7343136Z outputs = self.model.decoder( 2025-08-14T21:54:09.7343498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7343873Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7344212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7344563Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7344944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:09.7345335Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:09.7345465Z 2025-08-14T21:54:09.7345569Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7345904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7346201Z return mod(**inputs) 2025-08-14T21:54:09.7346543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7346908Z outputs = self.model.decoder( 2025-08-14T21:54:09.7347260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7347632Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7347951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7348281Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7348641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7349038Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7349436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:09.7349914Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:09.7350113Z 2025-08-14T21:54:09.7350209Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7350549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7350894Z return mod(**inputs) 2025-08-14T21:54:09.7351236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7351608Z outputs = self.model.decoder( 2025-08-14T21:54:09.7351970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7352339Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7352650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7352984Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7353351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7353755Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7354172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:09.7354564Z key_states = self.k_proj(current_states) 2025-08-14T21:54:09.7354692Z 2025-08-14T21:54:09.7354797Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7355129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7355437Z return mod(**inputs) 2025-08-14T21:54:09.7355797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7356172Z outputs = self.model.decoder( 2025-08-14T21:54:09.7356546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7356924Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7357258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7357600Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7357980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7358383Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7358785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:09.7359168Z value_states = self.v_proj(current_states) 2025-08-14T21:54:09.7359309Z 2025-08-14T21:54:09.7359387Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7359595Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7359796Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7360000Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7360223Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7360567Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7360874Z return mod(**inputs) 2025-08-14T21:54:09.7361229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7361607Z outputs = self.model.decoder( 2025-08-14T21:54:09.7361980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7362366Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7362738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7363087Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7363479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7363905Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7364312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7364725Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7365173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:09.7365741Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:09.7365932Z 2025-08-14T21:54:09.7366045Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7366404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7366774Z return mod(**inputs) 2025-08-14T21:54:09.7367172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7367561Z outputs = self.model.decoder( 2025-08-14T21:54:09.7367967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7368362Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7368694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7369042Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7369429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7369838Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7370242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7370645Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7371069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:09.7371508Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:09.7371665Z 2025-08-14T21:54:09.7371775Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7372111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7372426Z return mod(**inputs) 2025-08-14T21:54:09.7372782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7373157Z outputs = self.model.decoder( 2025-08-14T21:54:09.7373530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7373908Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7374247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7374589Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7374980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7375376Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7375756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:09.7376134Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:09.7376297Z 2025-08-14T21:54:09.7376393Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7376731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7377029Z return mod(**inputs) 2025-08-14T21:54:09.7377392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7377787Z outputs = self.model.decoder( 2025-08-14T21:54:09.7378148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7378507Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7378828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7379161Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7379519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7379934Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7380095Z 2025-08-14T21:54:09.7380190Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7380533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7380827Z return mod(**inputs) 2025-08-14T21:54:09.7381220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7381595Z outputs = self.model.decoder( 2025-08-14T21:54:09.7381954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7382317Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7382636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7382967Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7383328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7383736Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7384097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:09.7384417Z return self.act(input) 2025-08-14T21:54:09.7384521Z 2025-08-14T21:54:09.7384617Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7384959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7385262Z return mod(**inputs) 2025-08-14T21:54:09.7385601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7385975Z outputs = self.model.decoder( 2025-08-14T21:54:09.7386335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7386717Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7387030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7387364Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7387738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:09.7388112Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:09.7388238Z 2025-08-14T21:54:09.7388331Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7388663Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7388963Z return mod(**inputs) 2025-08-14T21:54:09.7389329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7389716Z outputs = self.model.decoder( 2025-08-14T21:54:09.7390089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7390492Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7390831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7391170Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7391548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-14T21:54:09.7391920Z hidden_states = residual + hidden_states 2025-08-14T21:54:09.7392056Z 2025-08-14T21:54:09.7392154Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7392500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7392812Z return mod(**inputs) 2025-08-14T21:54:09.7393163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7393563Z outputs = self.model.decoder( 2025-08-14T21:54:09.7393952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7394331Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7394652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7394992Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7395369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7395765Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7396161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:09.7396609Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:09.7396802Z 2025-08-14T21:54:09.7396906Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7397244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7397559Z return mod(**inputs) 2025-08-14T21:54:09.7397911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7398299Z outputs = self.model.decoder( 2025-08-14T21:54:09.7398662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7399048Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7399383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7399719Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7400102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7400502Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7400900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:09.7401278Z key_states = self.k_proj(current_states) 2025-08-14T21:54:09.7401413Z 2025-08-14T21:54:09.7401509Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7401849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7402153Z return mod(**inputs) 2025-08-14T21:54:09.7402508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7402920Z outputs = self.model.decoder( 2025-08-14T21:54:09.7403297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7403687Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7404020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7404366Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7404757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7405160Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7405775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:09.7406202Z value_states = self.v_proj(current_states) 2025-08-14T21:54:09.7406348Z 2025-08-14T21:54:09.7406432Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7406655Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7406911Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7407118Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7407334Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7407694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7408013Z return mod(**inputs) 2025-08-14T21:54:09.7408381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7408787Z outputs = self.model.decoder( 2025-08-14T21:54:09.7409180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7409580Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7409925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7410285Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7410689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7411109Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7411526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7411949Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7412390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:09.7412868Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:09.7413058Z 2025-08-14T21:54:09.7413161Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7413520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7413848Z return mod(**inputs) 2025-08-14T21:54:09.7414215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7414619Z outputs = self.model.decoder( 2025-08-14T21:54:09.7415009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7415401Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7415746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7416106Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7416503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7416945Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7417358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7417779Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7418199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:09.7418625Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:09.7418784Z 2025-08-14T21:54:09.7418882Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7419219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7419518Z return mod(**inputs) 2025-08-14T21:54:09.7419873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7420256Z outputs = self.model.decoder( 2025-08-14T21:54:09.7420663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7421037Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7421393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7421738Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7422107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7422503Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7422899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:09.7423284Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:09.7423412Z 2025-08-14T21:54:09.7423507Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7423850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7424163Z return mod(**inputs) 2025-08-14T21:54:09.7424522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7424892Z outputs = self.model.decoder( 2025-08-14T21:54:09.7425262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7425635Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7425953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7426303Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7426674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7427078Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7427234Z 2025-08-14T21:54:09.7427333Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7427673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7427978Z return mod(**inputs) 2025-08-14T21:54:09.7428329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7428704Z outputs = self.model.decoder( 2025-08-14T21:54:09.7429080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7429448Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7429792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7430123Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7430494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7430918Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7431267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:09.7431582Z return self.act(input) 2025-08-14T21:54:09.7431682Z 2025-08-14T21:54:09.7431782Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7432104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7432405Z return mod(**inputs) 2025-08-14T21:54:09.7432745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7433113Z outputs = self.model.decoder( 2025-08-14T21:54:09.7433465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7433859Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7434213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7434556Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7434926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:09.7435309Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:09.7435438Z 2025-08-14T21:54:09.7435541Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7435872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7436184Z return mod(**inputs) 2025-08-14T21:54:09.7436543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7436921Z outputs = self.model.decoder( 2025-08-14T21:54:09.7437285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7437852Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7438195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7438538Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7438921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7439325Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7439726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:09.7440170Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:09.7440374Z 2025-08-14T21:54:09.7440471Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7440813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7441123Z return mod(**inputs) 2025-08-14T21:54:09.7441471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7441849Z outputs = self.model.decoder( 2025-08-14T21:54:09.7442220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7442592Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7442922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7443336Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7443707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7444091Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7444524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:09.7444912Z key_states = self.k_proj(current_states) 2025-08-14T21:54:09.7445045Z 2025-08-14T21:54:09.7445155Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7445551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7445891Z return mod(**inputs) 2025-08-14T21:54:09.7446271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7446668Z outputs = self.model.decoder( 2025-08-14T21:54:09.7447037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7447517Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7447892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7448219Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7448583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7448972Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7449350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:09.7449727Z value_states = self.v_proj(current_states) 2025-08-14T21:54:09.7449865Z 2025-08-14T21:54:09.7449942Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7450141Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7450327Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7450524Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7450748Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7451079Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7451394Z return mod(**inputs) 2025-08-14T21:54:09.7451747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7452127Z outputs = self.model.decoder( 2025-08-14T21:54:09.7452487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7452874Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7453199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7453520Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7453936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7454342Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7454739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7455123Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7455560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:09.7456037Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:09.7456206Z 2025-08-14T21:54:09.7456337Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7456668Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7456977Z return mod(**inputs) 2025-08-14T21:54:09.7457331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7457729Z outputs = self.model.decoder( 2025-08-14T21:54:09.7458109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7458494Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7458829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7459174Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7459565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7459960Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7460360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7460792Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7461234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:09.7461670Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:09.7461825Z 2025-08-14T21:54:09.7461924Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7462270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7462585Z return mod(**inputs) 2025-08-14T21:54:09.7462943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7463326Z outputs = self.model.decoder( 2025-08-14T21:54:09.7463700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7464082Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7464414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7464752Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7465134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7465534Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7465927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:09.7466317Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:09.7466456Z 2025-08-14T21:54:09.7466555Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7466899Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7467202Z return mod(**inputs) 2025-08-14T21:54:09.7467562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7467947Z outputs = self.model.decoder( 2025-08-14T21:54:09.7468316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7468697Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7469035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7469378Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7469753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7470208Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7470376Z 2025-08-14T21:54:09.7470475Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7470816Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7471143Z return mod(**inputs) 2025-08-14T21:54:09.7471498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7471879Z outputs = self.model.decoder( 2025-08-14T21:54:09.7472244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7472635Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7472962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7473311Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7473671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7474098Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7474476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:09.7474793Z return self.act(input) 2025-08-14T21:54:09.7474894Z 2025-08-14T21:54:09.7474987Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7475319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7475622Z return mod(**inputs) 2025-08-14T21:54:09.7475959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7476327Z outputs = self.model.decoder( 2025-08-14T21:54:09.7476688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7477054Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7477366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7477699Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7478063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:09.7478433Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:09.7478563Z 2025-08-14T21:54:09.7478658Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7478982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7479280Z return mod(**inputs) 2025-08-14T21:54:09.7479617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7479981Z outputs = self.model.decoder( 2025-08-14T21:54:09.7480343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7480708Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7481024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7481355Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7481725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-14T21:54:09.7482091Z hidden_states = residual + hidden_states 2025-08-14T21:54:09.7482222Z 2025-08-14T21:54:09.7482315Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7482672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7482970Z return mod(**inputs) 2025-08-14T21:54:09.7483311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7483701Z outputs = self.model.decoder( 2025-08-14T21:54:09.7484061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7484430Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7484757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7485097Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7485586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7486038Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7486489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:09.7487039Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:09.7487234Z 2025-08-14T21:54:09.7487339Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7487689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7487997Z return mod(**inputs) 2025-08-14T21:54:09.7488346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7488710Z outputs = self.model.decoder( 2025-08-14T21:54:09.7489071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7489444Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7489763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7490086Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7490454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7490846Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7491233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:09.7491617Z key_states = self.k_proj(current_states) 2025-08-14T21:54:09.7491749Z 2025-08-14T21:54:09.7491844Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7492182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7492484Z return mod(**inputs) 2025-08-14T21:54:09.7492843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7493231Z outputs = self.model.decoder( 2025-08-14T21:54:09.7493597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7493970Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7494297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7494642Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7495016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7495417Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7495808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:09.7496220Z value_states = self.v_proj(current_states) 2025-08-14T21:54:09.7496350Z 2025-08-14T21:54:09.7496425Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7496627Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7496824Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7497030Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7497247Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7497583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7497886Z return mod(**inputs) 2025-08-14T21:54:09.7498228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7498608Z outputs = self.model.decoder( 2025-08-14T21:54:09.7502244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7502662Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7503003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7503378Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7503765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7504159Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7504564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7504976Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7505394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:09.7505896Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:09.7506073Z 2025-08-14T21:54:09.7506182Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7506525Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7506844Z return mod(**inputs) 2025-08-14T21:54:09.7507195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7507568Z outputs = self.model.decoder( 2025-08-14T21:54:09.7507936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7508317Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7508634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7508955Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7509326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7509709Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7510093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7510471Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7510876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:09.7511297Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:09.7511448Z 2025-08-14T21:54:09.7511552Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7511879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7512187Z return mod(**inputs) 2025-08-14T21:54:09.7512575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7512947Z outputs = self.model.decoder( 2025-08-14T21:54:09.7513317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7513709Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7514036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7514375Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7514798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7515238Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7515646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:09.7516122Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:09.7516262Z 2025-08-14T21:54:09.7516371Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7516717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7517013Z return mod(**inputs) 2025-08-14T21:54:09.7517351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7517725Z outputs = self.model.decoder( 2025-08-14T21:54:09.7518092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7518459Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7518788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7519134Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7519504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7519923Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7520094Z 2025-08-14T21:54:09.7520193Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7520530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7520828Z return mod(**inputs) 2025-08-14T21:54:09.7521178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7521559Z outputs = self.model.decoder( 2025-08-14T21:54:09.7521923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7522295Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7522617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7522949Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7523309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7523712Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7524070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:09.7524392Z return self.act(input) 2025-08-14T21:54:09.7524496Z 2025-08-14T21:54:09.7524591Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7524925Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7525228Z return mod(**inputs) 2025-08-14T21:54:09.7525669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7526103Z outputs = self.model.decoder( 2025-08-14T21:54:09.7526479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7526877Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7527198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7527534Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7527915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:09.7528292Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:09.7528428Z 2025-08-14T21:54:09.7528526Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7528907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7529220Z return mod(**inputs) 2025-08-14T21:54:09.7529566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7529964Z outputs = self.model.decoder( 2025-08-14T21:54:09.7530341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7530720Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7531045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7531389Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7531768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7532168Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7532574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:09.7532725Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:09.7532729Z 2025-08-14T21:54:09.7532827Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7533015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7533085Z return mod(**inputs) 2025-08-14T21:54:09.7533338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7533408Z outputs = self.model.decoder( 2025-08-14T21:54:09.7533665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7533735Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7533957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7534033Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7534284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7534385Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7534633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:09.7534715Z key_states = self.k_proj(current_states) 2025-08-14T21:54:09.7534718Z 2025-08-14T21:54:09.7534814Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7535000Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7535071Z return mod(**inputs) 2025-08-14T21:54:09.7535319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7535412Z outputs = self.model.decoder( 2025-08-14T21:54:09.7535675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7535761Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7535979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7536052Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7536301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7536402Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7536651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:09.7536763Z value_states = self.v_proj(current_states) 2025-08-14T21:54:09.7536767Z 2025-08-14T21:54:09.7536846Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7536921Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7537002Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7537090Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7537189Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7537403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7537466Z return mod(**inputs) 2025-08-14T21:54:09.7537902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7537985Z outputs = self.model.decoder( 2025-08-14T21:54:09.7538236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7538317Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7538529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7538604Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7538865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7538958Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7539267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7539360Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7539634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:09.7539767Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:09.7539772Z 2025-08-14T21:54:09.7539872Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7540066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7540127Z return mod(**inputs) 2025-08-14T21:54:09.7540407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7540478Z outputs = self.model.decoder( 2025-08-14T21:54:09.7540729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7540804Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7541012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7541086Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7541343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7541483Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7541745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7541865Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7542135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:09.7542246Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:09.7542249Z 2025-08-14T21:54:09.7542348Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7542543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7542608Z return mod(**inputs) 2025-08-14T21:54:09.7542893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7542978Z outputs = self.model.decoder( 2025-08-14T21:54:09.7543265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7543335Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7543569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7543640Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7543890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7543981Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7544230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:09.7544317Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:09.7544320Z 2025-08-14T21:54:09.7544417Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7544607Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7544670Z return mod(**inputs) 2025-08-14T21:54:09.7544918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7544993Z outputs = self.model.decoder( 2025-08-14T21:54:09.7545241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7545308Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7545522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7545594Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7545853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7545963Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7545968Z 2025-08-14T21:54:09.7546062Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7546256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7546318Z return mod(**inputs) 2025-08-14T21:54:09.7546560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7546637Z outputs = self.model.decoder( 2025-08-14T21:54:09.7546877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7546949Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7547149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7547240Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7547488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7547611Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7547812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:09.7547878Z return self.act(input) 2025-08-14T21:54:09.7547881Z 2025-08-14T21:54:09.7547973Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7548157Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7548217Z return mod(**inputs) 2025-08-14T21:54:09.7548458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7548553Z outputs = self.model.decoder( 2025-08-14T21:54:09.7548805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7548906Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7549116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7549189Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7549442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:09.7549517Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:09.7549521Z 2025-08-14T21:54:09.7549623Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7549807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7549872Z return mod(**inputs) 2025-08-14T21:54:09.7550127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7550195Z outputs = self.model.decoder( 2025-08-14T21:54:09.7550440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7550516Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7550724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7550804Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7551051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-14T21:54:09.7551127Z hidden_states = residual + hidden_states 2025-08-14T21:54:09.7551131Z 2025-08-14T21:54:09.7551232Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7551419Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7551482Z return mod(**inputs) 2025-08-14T21:54:09.7551741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7551811Z outputs = self.model.decoder( 2025-08-14T21:54:09.7552063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7552131Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7552337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7552420Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7552672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7552790Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7553150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:09.7553293Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:09.7553312Z 2025-08-14T21:54:09.7553417Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7553601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7553669Z return mod(**inputs) 2025-08-14T21:54:09.7553919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7553989Z outputs = self.model.decoder( 2025-08-14T21:54:09.7554243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7554335Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7554546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7554631Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7554898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7555001Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7555258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:09.7555334Z key_states = self.k_proj(current_states) 2025-08-14T21:54:09.7555338Z 2025-08-14T21:54:09.7555452Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7555635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7555708Z return mod(**inputs) 2025-08-14T21:54:09.7555958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7556025Z outputs = self.model.decoder( 2025-08-14T21:54:09.7556283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7556348Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7556549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7556628Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7556873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7556968Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7557216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:09.7557296Z value_states = self.v_proj(current_states) 2025-08-14T21:54:09.7557300Z 2025-08-14T21:54:09.7557382Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7557457Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7557529Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7557607Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7557701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7557890Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7557952Z return mod(**inputs) 2025-08-14T21:54:09.7558199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7558275Z outputs = self.model.decoder( 2025-08-14T21:54:09.7558527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7558613Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7558825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7558902Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7559178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7559269Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7559521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7559634Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7559940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:09.7560135Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:09.7560140Z 2025-08-14T21:54:09.7560254Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7560491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7560576Z return mod(**inputs) 2025-08-14T21:54:09.7560862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7560948Z outputs = self.model.decoder( 2025-08-14T21:54:09.7561238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7561314Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7561565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7561647Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7561926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7562035Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7562309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7562416Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7562719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:09.7562835Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:09.7562839Z 2025-08-14T21:54:09.7562957Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7563170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7563250Z return mod(**inputs) 2025-08-14T21:54:09.7563539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7563617Z outputs = self.model.decoder( 2025-08-14T21:54:09.7563913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7563991Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7564225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7564318Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7564606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7564717Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7565023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:09.7565132Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:09.7565135Z 2025-08-14T21:54:09.7565252Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7565570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7565671Z return mod(**inputs) 2025-08-14T21:54:09.7566071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7566154Z outputs = self.model.decoder( 2025-08-14T21:54:09.7566439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7566515Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7566739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7566845Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7567100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7567236Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7567242Z 2025-08-14T21:54:09.7567341Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7567527Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7567599Z return mod(**inputs) 2025-08-14T21:54:09.7567847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7567919Z outputs = self.model.decoder( 2025-08-14T21:54:09.7568182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7568253Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7568471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7568547Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7568793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7568913Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7569111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:09.7569186Z return self.act(input) 2025-08-14T21:54:09.7569190Z 2025-08-14T21:54:09.7569285Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7569472Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7569542Z return mod(**inputs) 2025-08-14T21:54:09.7569791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7569860Z outputs = self.model.decoder( 2025-08-14T21:54:09.7570115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7570183Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7570395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7570469Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7570714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:09.7570796Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:09.7570800Z 2025-08-14T21:54:09.7570894Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7571107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7571171Z return mod(**inputs) 2025-08-14T21:54:09.7571416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7571493Z outputs = self.model.decoder( 2025-08-14T21:54:09.7571796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7571870Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7572112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7572196Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7572489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7572593Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7572914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:09.7573087Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:09.7573108Z 2025-08-14T21:54:09.7573220Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7573443Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7573514Z return mod(**inputs) 2025-08-14T21:54:09.7573804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7573890Z outputs = self.model.decoder( 2025-08-14T21:54:09.7574180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7574255Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7574496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7574578Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7574859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7574963Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7575237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:09.7575328Z key_states = self.k_proj(current_states) 2025-08-14T21:54:09.7575332Z 2025-08-14T21:54:09.7575437Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7575657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7575728Z return mod(**inputs) 2025-08-14T21:54:09.7576018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7576104Z outputs = self.model.decoder( 2025-08-14T21:54:09.7576388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7576466Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7576707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7576791Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7577079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7577184Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7577467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:09.7577594Z value_states = self.v_proj(current_states) 2025-08-14T21:54:09.7577597Z 2025-08-14T21:54:09.7577683Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7577766Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7577858Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7577959Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7578075Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7578293Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7578360Z return mod(**inputs) 2025-08-14T21:54:09.7578646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7578721Z outputs = self.model.decoder( 2025-08-14T21:54:09.7578999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7580026Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7580299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7580389Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7580684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7580790Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7581076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7581178Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7581488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:09.7581629Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:09.7581636Z 2025-08-14T21:54:09.7581743Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7581960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7582030Z return mod(**inputs) 2025-08-14T21:54:09.7582312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7582397Z outputs = self.model.decoder( 2025-08-14T21:54:09.7582696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7582780Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7583012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7583095Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7583381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7583481Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7583766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7583867Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7584174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:09.7584295Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:09.7584299Z 2025-08-14T21:54:09.7584407Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7584629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7584697Z return mod(**inputs) 2025-08-14T21:54:09.7585001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7585086Z outputs = self.model.decoder( 2025-08-14T21:54:09.7585358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7585445Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7585653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7585727Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7585978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7586069Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7586313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:09.7586419Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:09.7586423Z 2025-08-14T21:54:09.7586522Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7586730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7586802Z return mod(**inputs) 2025-08-14T21:54:09.7587051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7587127Z outputs = self.model.decoder( 2025-08-14T21:54:09.7587377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7587445Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7587660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7587743Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7587990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7588099Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7588103Z 2025-08-14T21:54:09.7588205Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7588393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7588454Z return mod(**inputs) 2025-08-14T21:54:09.7588694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7588769Z outputs = self.model.decoder( 2025-08-14T21:54:09.7589018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7589094Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7589302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7589376Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7589634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7589745Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7589952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:09.7590019Z return self.act(input) 2025-08-14T21:54:09.7590022Z 2025-08-14T21:54:09.7590116Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7590310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7590373Z return mod(**inputs) 2025-08-14T21:54:09.7590620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7590712Z outputs = self.model.decoder( 2025-08-14T21:54:09.7590958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7591033Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7591263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7591334Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7591583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:09.7591657Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:09.7591661Z 2025-08-14T21:54:09.7591763Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7591943Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7592023Z return mod(**inputs) 2025-08-14T21:54:09.7592277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7592343Z outputs = self.model.decoder( 2025-08-14T21:54:09.7592598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7592672Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7592869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7592947Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7593185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-14T21:54:09.7593259Z hidden_states = residual + hidden_states 2025-08-14T21:54:09.7593262Z 2025-08-14T21:54:09.7593364Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7593542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7593602Z return mod(**inputs) 2025-08-14T21:54:09.7593849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7593915Z outputs = self.model.decoder( 2025-08-14T21:54:09.7594163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7594229Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7594425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7594505Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7594761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7594865Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7595116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:09.7595270Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:09.7595274Z 2025-08-14T21:54:09.7595376Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7595559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7595621Z return mod(**inputs) 2025-08-14T21:54:09.7595876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7595944Z outputs = self.model.decoder( 2025-08-14T21:54:09.7596205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7596300Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7596501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7596584Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7596855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7596952Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7597194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:09.7597268Z key_states = self.k_proj(current_states) 2025-08-14T21:54:09.7597271Z 2025-08-14T21:54:09.7597371Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7597551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7597631Z return mod(**inputs) 2025-08-14T21:54:09.7597885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7597951Z outputs = self.model.decoder( 2025-08-14T21:54:09.7598216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7598286Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7598489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7598570Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7598810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7598907Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7599150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:09.7599230Z value_states = self.v_proj(current_states) 2025-08-14T21:54:09.7599234Z 2025-08-14T21:54:09.7599316Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7599390Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7599461Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7599545Z cudagraph partition due to non gpu ops 2025-08-14T21:54:09.7599640Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7599839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7599901Z return mod(**inputs) 2025-08-14T21:54:09.7600150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7600226Z outputs = self.model.decoder( 2025-08-14T21:54:09.7600476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7600545Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7600766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7600841Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7601093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7601184Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7601429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7601528Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7601798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:09.7601957Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:09.7601961Z 2025-08-14T21:54:09.7602057Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7602244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7602331Z return mod(**inputs) 2025-08-14T21:54:09.7602582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7602653Z outputs = self.model.decoder( 2025-08-14T21:54:09.7602909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7602978Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7603194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7603287Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7603536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7603636Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7603897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:09.7603997Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:09.7604265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:09.7604368Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:09.7604371Z 2025-08-14T21:54:09.7604474Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7604664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7604730Z return mod(**inputs) 2025-08-14T21:54:09.7604993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7605064Z outputs = self.model.decoder( 2025-08-14T21:54:09.7605401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7605482Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7605694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7605779Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7606049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:09.7606157Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:09.7606437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:09.7606524Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:09.7606528Z 2025-08-14T21:54:09.7606642Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7606853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7606925Z return mod(**inputs) 2025-08-14T21:54:09.7607214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7607289Z outputs = self.model.decoder( 2025-08-14T21:54:09.7607558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7607627Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7607834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7607940Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7608188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7608300Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7608334Z 2025-08-14T21:54:09.7608431Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7608615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7608686Z return mod(**inputs) 2025-08-14T21:54:09.7608937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7609007Z outputs = self.model.decoder( 2025-08-14T21:54:09.7609263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7609350Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7609564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7609655Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7609907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:09.7610026Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:09.7610224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:09.7610291Z return self.act(input) 2025-08-14T21:54:09.7610302Z 2025-08-14T21:54:09.7610398Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7610603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7610679Z return mod(**inputs) 2025-08-14T21:54:09.7610927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1634, in forward 2025-08-14T21:54:09.7610996Z outputs = self.model.decoder( 2025-08-14T21:54:09.7611251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:09.7611321Z layer_outputs = decoder_layer( 2025-08-14T21:54:09.7611534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:09.7611607Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:09.7611853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:09.7611937Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:09.7611940Z 2025-08-14T21:54:09.7612034Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7612220Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7612292Z return mod(**inputs) 2025-08-14T21:54:09.7612540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1650, in forward 2025-08-14T21:54:09.7612625Z logits = self.lm_head(outputs[0]) 2025-08-14T21:54:09.7612628Z 2025-08-14T21:54:09.7612723Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:09.7612905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:09.7612975Z return mod(**inputs) 2025-08-14T21:54:09.7613223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1656, in forward 2025-08-14T21:54:09.7613369Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:54:09.7613373Z 2025-08-14T21:54:18.8477898Z Compilation time (from dynamo_timed): 15.251705224 2025-08-14T21:54:18.8496499Z pass 2025-08-14T21:54:18.8496881Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:54:18.8497802Z TIMING: _recursive_pre_grad_passes:0.00755 _recursive_joint_graph_passes:0.65273 _recursive_post_grad_passes:0.08356 async_compile.wait:0.73703 code_gen:8.33834 inductor_compile:9.54844 backend_compile:12.69741 gc:0.00012 entire_frame_compile:15.25171 total_wall_time:15.25171 2025-08-14T21:54:18.8502925Z STATS: call_* op count: 369 | FakeTensorMode.__torch_dispatch__:13170 | FakeTensor.__torch_dispatch__:4856 | ProxyTorchDispatchMode.__torch_dispatch__:4803 2025-08-14T21:54:18.8503600Z Dynamo produced 1 graphs covering 369 ops with 0 graph breaks (0 unique) 2025-08-14T21:54:24.1503235Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:54:24.1504138Z from pkg_resources import resource_filename 2025-08-14T21:54:24.7373371Z 2025-08-14T21:54:30.3412898Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:54:30.3417189Z loading model: 0it [00:05, ?it/s] 2025-08-14T21:54:30.3439021Z cpu eval PegasusForConditionalGeneration 2025-08-14T21:54:31.0064264Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:54:31.2935254Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:54:31.5630599Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:54:48.0921202Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.0921517Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.0921912Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.0922318Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.0922675Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.0925713Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.0926030Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.0926286Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.0926534Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.0926796Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.0927025Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.0927250Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.0927488Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.0927887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.0928236Z return mod(**inputs) 2025-08-14T21:54:48.0928668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.0929136Z outputs = self.model( 2025-08-14T21:54:48.0929570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.0930015Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.0930416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.0930826Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.0931187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.0931554Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.0931967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.0932393Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.0934007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.0934484Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.0934738Z 2025-08-14T21:54:48.0934856Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.0935378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.0949809Z return mod(**inputs) 2025-08-14T21:54:48.0950315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.0950768Z outputs = self.model( 2025-08-14T21:54:48.0951197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.0951639Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.0952298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.0952787Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.0953304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.0953794Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.0954262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.0954750Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.0955276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.0955679Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.0955830Z 2025-08-14T21:54:48.0955944Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.0956320Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.0956664Z return mod(**inputs) 2025-08-14T21:54:48.0957047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.0957449Z outputs = self.model( 2025-08-14T21:54:48.0957831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.0958236Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.0958657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.0959073Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.0959421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.0959773Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.0960182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.0960689Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.0961142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.0961736Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.0961896Z 2025-08-14T21:54:48.0961986Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.0962211Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.0962430Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.0962638Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.0962886Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.0963274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.0963615Z return mod(**inputs) 2025-08-14T21:54:48.0964082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.0964578Z outputs = self.model( 2025-08-14T21:54:48.0964993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.0965642Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.0966075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.0966513Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.0966875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.0967235Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.0967631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.0968078Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.0968481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.0968935Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.0969363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.0969818Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.0969996Z 2025-08-14T21:54:48.0970096Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.0970437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.0970750Z return mod(**inputs) 2025-08-14T21:54:48.0971098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.0971470Z outputs = self.model( 2025-08-14T21:54:48.0971824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.0972200Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.0972564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.0972939Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.0973268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.0973607Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.0973982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.0974384Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.0974787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.0975198Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.0975648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.0976102Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.0976260Z 2025-08-14T21:54:48.0976369Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.0976712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.0977030Z return mod(**inputs) 2025-08-14T21:54:48.0977399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.0977788Z outputs = self.model( 2025-08-14T21:54:48.0978155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.0978577Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.0978964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.0979372Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.0979721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.0980069Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.0980451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.0980843Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.0981240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.0981659Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.0981796Z 2025-08-14T21:54:48.0981899Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.0982260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.0982602Z return mod(**inputs) 2025-08-14T21:54:48.0982975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.0983356Z outputs = self.model( 2025-08-14T21:54:48.0983734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.0984137Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.0984518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.0984907Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.0985267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.0985619Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.0986003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.0986452Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.0986624Z 2025-08-14T21:54:48.0986726Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.0987071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.0987377Z return mod(**inputs) 2025-08-14T21:54:48.0987740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.0988125Z outputs = self.model( 2025-08-14T21:54:48.0988487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.0988883Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.0989283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.0989672Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.0990006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.0990358Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.0990746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.0991175Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.0991548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.0991927Z return self.act(input) 2025-08-14T21:54:48.0992037Z 2025-08-14T21:54:48.0992147Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.0992502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.0992816Z return mod(**inputs) 2025-08-14T21:54:48.0993195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.0993585Z outputs = self.model( 2025-08-14T21:54:48.0993941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.0994327Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.0994703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.0995089Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.0995446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.0995815Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.0996273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:54:48.0996697Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.0996855Z 2025-08-14T21:54:48.0996958Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.0997316Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.0997639Z return mod(**inputs) 2025-08-14T21:54:48.0998001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.0998393Z outputs = self.model( 2025-08-14T21:54:48.0998764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.0999160Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.0999553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.0999952Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1000317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1000688Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1001117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1001552Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1001982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1002477Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1002705Z 2025-08-14T21:54:48.1002816Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1003194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1003535Z return mod(**inputs) 2025-08-14T21:54:48.1003932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1004354Z outputs = self.model( 2025-08-14T21:54:48.1004764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1005186Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1005718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1006164Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1006580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1006960Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1007391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1007865Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1008293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1008735Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1008881Z 2025-08-14T21:54:48.1008990Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1009371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1009709Z return mod(**inputs) 2025-08-14T21:54:48.1010125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1010548Z outputs = self.model( 2025-08-14T21:54:48.1010934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1011375Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1011816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1012236Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1012580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1012931Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1013322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1013726Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1014123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1014529Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1014665Z 2025-08-14T21:54:48.1014757Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1014971Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1015184Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1015392Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1015626Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1015982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1016306Z return mod(**inputs) 2025-08-14T21:54:48.1016686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1017063Z outputs = self.model( 2025-08-14T21:54:48.1017442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1017831Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1018222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1018620Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1018965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1019342Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1019754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1020191Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1020627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1021099Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1021523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1021993Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1022199Z 2025-08-14T21:54:48.1022301Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1022648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1022955Z return mod(**inputs) 2025-08-14T21:54:48.1023319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1023702Z outputs = self.model( 2025-08-14T21:54:48.1024075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1024466Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1024864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1025302Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1025650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1026015Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1026410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1026824Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1027230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1027638Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1028104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1028573Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1028741Z 2025-08-14T21:54:48.1028853Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1029240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1029584Z return mod(**inputs) 2025-08-14T21:54:48.1029978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1030393Z outputs = self.model( 2025-08-14T21:54:48.1030798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1031223Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1031636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1032057Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1032440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1032823Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1033262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1033701Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1034140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1034565Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1034716Z 2025-08-14T21:54:48.1034825Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1035228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1035572Z return mod(**inputs) 2025-08-14T21:54:48.1035959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1036369Z outputs = self.model( 2025-08-14T21:54:48.1036852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1037268Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1037888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1038319Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1038691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1039081Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1039613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.1040062Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1040233Z 2025-08-14T21:54:48.1040384Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1040740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1041063Z return mod(**inputs) 2025-08-14T21:54:48.1041449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1041831Z outputs = self.model( 2025-08-14T21:54:48.1042199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1042592Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1042985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1043374Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1043717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1044074Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1044466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.1044983Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1045446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.1045813Z return self.act(input) 2025-08-14T21:54:48.1045930Z 2025-08-14T21:54:48.1046038Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1046424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1046766Z return mod(**inputs) 2025-08-14T21:54:48.1047153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1047563Z outputs = self.model( 2025-08-14T21:54:48.1047939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1048337Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1048729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1049142Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1049500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1049878Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1050324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:54:48.1050759Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.1050896Z 2025-08-14T21:54:48.1051009Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1051384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1051708Z return mod(**inputs) 2025-08-14T21:54:48.1052082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1052468Z outputs = self.model( 2025-08-14T21:54:48.1052841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1053238Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1053640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1054044Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1054389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1054782Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1055183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1055593Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1056005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1056506Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1056733Z 2025-08-14T21:54:48.1056845Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1057204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1057530Z return mod(**inputs) 2025-08-14T21:54:48.1057905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1058313Z outputs = self.model( 2025-08-14T21:54:48.1058713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1059137Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1059543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1059962Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1060315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1060678Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1061071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1061466Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1061866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1062258Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1062387Z 2025-08-14T21:54:48.1062487Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1062842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1063166Z return mod(**inputs) 2025-08-14T21:54:48.1063541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1063947Z outputs = self.model( 2025-08-14T21:54:48.1064346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1064790Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1065203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1065634Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1065999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1066373Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1066782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1067230Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1067680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1068144Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1068293Z 2025-08-14T21:54:48.1068382Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1068607Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1068829Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1069055Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1069310Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1069715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1070059Z return mod(**inputs) 2025-08-14T21:54:48.1070450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1070868Z outputs = self.model( 2025-08-14T21:54:48.1071268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1071666Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1072055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1072446Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1072800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1073155Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1073547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1073958Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1074389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1074830Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1075303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1075810Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1076002Z 2025-08-14T21:54:48.1076115Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1076496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1076840Z return mod(**inputs) 2025-08-14T21:54:48.1077254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1077665Z outputs = self.model( 2025-08-14T21:54:48.1078062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1078484Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1078888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1079329Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1079695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1080072Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1080526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1080962Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1081391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1081831Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1082288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1082789Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1082963Z 2025-08-14T21:54:48.1083077Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1083453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1083821Z return mod(**inputs) 2025-08-14T21:54:48.1084217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1084627Z outputs = self.model( 2025-08-14T21:54:48.1085011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1085522Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1085947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1086383Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1086767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1087165Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1087594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1088026Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1088466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1088896Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1089040Z 2025-08-14T21:54:48.1089158Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1089531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1089867Z return mod(**inputs) 2025-08-14T21:54:48.1090231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1090607Z outputs = self.model( 2025-08-14T21:54:48.1090973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1091364Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1091745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1092119Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1092458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1092809Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1093196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.1093651Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1093832Z 2025-08-14T21:54:48.1093934Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1094289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1094606Z return mod(**inputs) 2025-08-14T21:54:48.1095002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1095398Z outputs = self.model( 2025-08-14T21:54:48.1095776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1096154Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1096536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1096919Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1097282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1097635Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1098042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.1098473Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1098842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.1099174Z return self.act(input) 2025-08-14T21:54:48.1099282Z 2025-08-14T21:54:48.1099390Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1099739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1100046Z return mod(**inputs) 2025-08-14T21:54:48.1100409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1100791Z outputs = self.model( 2025-08-14T21:54:48.1101149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1101538Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1101913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1102295Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1102621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1102968Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1103354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:54:48.1103737Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.1103879Z 2025-08-14T21:54:48.1103980Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1104330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1104646Z return mod(**inputs) 2025-08-14T21:54:48.1105009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1105405Z outputs = self.model( 2025-08-14T21:54:48.1105778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1106174Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1106554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1106947Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1107295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1107666Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1108065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 327, in forward 2025-08-14T21:54:48.1108485Z hidden_states = residual + hidden_states 2025-08-14T21:54:48.1108619Z 2025-08-14T21:54:48.1108728Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1109077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1109399Z return mod(**inputs) 2025-08-14T21:54:48.1109772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1110155Z outputs = self.model( 2025-08-14T21:54:48.1110525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1110940Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1111337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1111743Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1112094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1112449Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1112841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1113320Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1113726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1114204Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1114410Z 2025-08-14T21:54:48.1114512Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1114879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1115221Z return mod(**inputs) 2025-08-14T21:54:48.1115621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1116039Z outputs = self.model( 2025-08-14T21:54:48.1116433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1116855Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1117236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1117622Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1117967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1118326Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1118712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1119121Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1119527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1119920Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1120050Z 2025-08-14T21:54:48.1120163Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1120510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1120823Z return mod(**inputs) 2025-08-14T21:54:48.1121181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1121587Z outputs = self.model( 2025-08-14T21:54:48.1121950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1122336Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1122733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1123117Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1123455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1123806Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1124187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1124591Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1125016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1125505Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1125662Z 2025-08-14T21:54:48.1125770Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1125993Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1126217Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1126430Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1126679Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1127053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1127375Z return mod(**inputs) 2025-08-14T21:54:48.1127732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1128104Z outputs = self.model( 2025-08-14T21:54:48.1128474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1128865Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1129237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1129613Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1129936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1130275Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1130652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1131044Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1131424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1131827Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1132246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1132697Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1132873Z 2025-08-14T21:54:48.1132969Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1133308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1133612Z return mod(**inputs) 2025-08-14T21:54:48.1133959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1134332Z outputs = self.model( 2025-08-14T21:54:48.1134685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1135091Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1135452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1135832Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1136188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1136533Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1136923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1137322Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1137905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1138334Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1138812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1139336Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1139493Z 2025-08-14T21:54:48.1139633Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1139979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1140301Z return mod(**inputs) 2025-08-14T21:54:48.1140664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1141044Z outputs = self.model( 2025-08-14T21:54:48.1141408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1141792Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1142176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1142560Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1142896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1143247Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1143634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1144028Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1144420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1144894Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1145023Z 2025-08-14T21:54:48.1145123Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1145469Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1145780Z return mod(**inputs) 2025-08-14T21:54:48.1146137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1146519Z outputs = self.model( 2025-08-14T21:54:48.1146886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1147266Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1147634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1148019Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1148352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1148699Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1149113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.1149519Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1149678Z 2025-08-14T21:54:48.1149785Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1150147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1150440Z return mod(**inputs) 2025-08-14T21:54:48.1150794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1151170Z outputs = self.model( 2025-08-14T21:54:48.1151518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1151898Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1152272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1152639Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1152973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1153320Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1153696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.1154105Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1154475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.1154817Z return self.act(input) 2025-08-14T21:54:48.1154926Z 2025-08-14T21:54:48.1155037Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1155393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1155718Z return mod(**inputs) 2025-08-14T21:54:48.1156095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1156493Z outputs = self.model( 2025-08-14T21:54:48.1156862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1157262Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1157634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1158002Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1158333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1158676Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1159058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:54:48.1159439Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.1159577Z 2025-08-14T21:54:48.1159675Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1160015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1160316Z return mod(**inputs) 2025-08-14T21:54:48.1160674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1161049Z outputs = self.model( 2025-08-14T21:54:48.1161404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1161773Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1162143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1162539Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1162865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1163198Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1163629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1164025Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1164406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1164865Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1165075Z 2025-08-14T21:54:48.1165176Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1165656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1165992Z return mod(**inputs) 2025-08-14T21:54:48.1166408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1166868Z outputs = self.model( 2025-08-14T21:54:48.1167284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1167697Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1168093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1168484Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1168824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1169180Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1169579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1169994Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1170393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1170793Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1170924Z 2025-08-14T21:54:48.1171033Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1171382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1171705Z return mod(**inputs) 2025-08-14T21:54:48.1172075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1172464Z outputs = self.model( 2025-08-14T21:54:48.1172830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1173226Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1173607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1173997Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1174338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1174695Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1175092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1175495Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1175905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1176343Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1176487Z 2025-08-14T21:54:48.1176577Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1176786Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1176993Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1177204Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1177448Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1177808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1178134Z return mod(**inputs) 2025-08-14T21:54:48.1178498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1178881Z outputs = self.model( 2025-08-14T21:54:48.1179246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1179633Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1180034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1180422Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1180775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1181134Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1181518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1181925Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1182324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1182763Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1183206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1183679Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1183859Z 2025-08-14T21:54:48.1183970Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1184319Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1184640Z return mod(**inputs) 2025-08-14T21:54:48.1185015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1185405Z outputs = self.model( 2025-08-14T21:54:48.1185772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1186167Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1186548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1186932Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1187277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1187629Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1188034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1188426Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1188816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1189230Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1189652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1190091Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1190249Z 2025-08-14T21:54:48.1190345Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1190685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1190982Z return mod(**inputs) 2025-08-14T21:54:48.1191361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1191732Z outputs = self.model( 2025-08-14T21:54:48.1192085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1192462Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1192836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1193212Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1193553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1193900Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1194300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1194702Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1195094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1195487Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1195626Z 2025-08-14T21:54:48.1195725Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1196075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1196380Z return mod(**inputs) 2025-08-14T21:54:48.1196745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1197126Z outputs = self.model( 2025-08-14T21:54:48.1197479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1197871Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1198250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1198640Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1198975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1199330Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1199719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.1200152Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1200325Z 2025-08-14T21:54:48.1200423Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1200772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1201088Z return mod(**inputs) 2025-08-14T21:54:48.1201447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1201835Z outputs = self.model( 2025-08-14T21:54:48.1202200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1202591Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1202969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1203360Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1203739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1204081Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1204467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.1204924Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1205380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.1205737Z return self.act(input) 2025-08-14T21:54:48.1205857Z 2025-08-14T21:54:48.1205963Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1206366Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1206716Z return mod(**inputs) 2025-08-14T21:54:48.1207100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1207487Z outputs = self.model( 2025-08-14T21:54:48.1207854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1208253Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1208639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1209028Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1209369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1209712Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1210099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:54:48.1210496Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.1210631Z 2025-08-14T21:54:48.1210733Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1211085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1211401Z return mod(**inputs) 2025-08-14T21:54:48.1211766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1212141Z outputs = self.model( 2025-08-14T21:54:48.1212509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1212911Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1213294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1213670Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1214012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1214360Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1214738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 327, in forward 2025-08-14T21:54:48.1215130Z hidden_states = residual + hidden_states 2025-08-14T21:54:48.1215270Z 2025-08-14T21:54:48.1215369Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1215725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1216049Z return mod(**inputs) 2025-08-14T21:54:48.1216412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1216793Z outputs = self.model( 2025-08-14T21:54:48.1217152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1217568Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1217961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1218341Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1218677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1219016Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1219392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1219784Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1220164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1220605Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1220817Z 2025-08-14T21:54:48.1220925Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1221259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1221580Z return mod(**inputs) 2025-08-14T21:54:48.1221935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1222322Z outputs = self.model( 2025-08-14T21:54:48.1222681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1223067Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1223446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1223829Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1224153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1224500Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1224888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1225284Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1225678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1226070Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1226198Z 2025-08-14T21:54:48.1226304Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1226645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1226961Z return mod(**inputs) 2025-08-14T21:54:48.1227326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1227706Z outputs = self.model( 2025-08-14T21:54:48.1228074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1228473Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1228863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1229248Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1229596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1229953Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1230355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1230746Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1231164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1231558Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1231692Z 2025-08-14T21:54:48.1231773Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1232006Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1232212Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1232414Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1232632Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1232986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1233294Z return mod(**inputs) 2025-08-14T21:54:48.1233640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1234013Z outputs = self.model( 2025-08-14T21:54:48.1234394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1234778Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1235161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1235544Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1235874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1236209Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1236592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1236985Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1237372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1237949Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1238380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1238837Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1239014Z 2025-08-14T21:54:48.1239120Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1239457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1239770Z return mod(**inputs) 2025-08-14T21:54:48.1240136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1240510Z outputs = self.model( 2025-08-14T21:54:48.1240868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1241251Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1241620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1242006Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1242351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1242709Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1243095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1243498Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1244575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1244988Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1245524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1246030Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1246208Z 2025-08-14T21:54:48.1246331Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1246717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1247036Z return mod(**inputs) 2025-08-14T21:54:48.1247439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1247868Z outputs = self.model( 2025-08-14T21:54:48.1248227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1248619Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1249034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1249465Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1249835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1250191Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1250577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1250976Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1251364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1251752Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1251881Z 2025-08-14T21:54:48.1251987Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1252325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1252640Z return mod(**inputs) 2025-08-14T21:54:48.1253002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1253389Z outputs = self.model( 2025-08-14T21:54:48.1253741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1254126Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1254503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1254875Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1255228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1255564Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1255940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.1256350Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1256538Z 2025-08-14T21:54:48.1256634Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1257011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1257309Z return mod(**inputs) 2025-08-14T21:54:48.1257644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1258004Z outputs = self.model( 2025-08-14T21:54:48.1258349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1258713Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1259071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1259463Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1259787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1260111Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1260499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.1260909Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1261269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.1261576Z return self.act(input) 2025-08-14T21:54:48.1261684Z 2025-08-14T21:54:48.1261779Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1262126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1262425Z return mod(**inputs) 2025-08-14T21:54:48.1262769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1263147Z outputs = self.model( 2025-08-14T21:54:48.1263495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1263853Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1264212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1264584Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1264899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1265238Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1265615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:54:48.1265999Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.1266127Z 2025-08-14T21:54:48.1266234Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1266571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1266866Z return mod(**inputs) 2025-08-14T21:54:48.1267202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1267567Z outputs = self.model( 2025-08-14T21:54:48.1267913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1268281Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1268635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1269001Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1269318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1269651Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1270010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1270396Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1270772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1271199Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1271394Z 2025-08-14T21:54:48.1271489Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1271827Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1272156Z return mod(**inputs) 2025-08-14T21:54:48.1272506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1272892Z outputs = self.model( 2025-08-14T21:54:48.1273269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1273648Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1274012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1274386Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1274718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1275052Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1275456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1275853Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1276271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1276649Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1276782Z 2025-08-14T21:54:48.1276879Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1277219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1277528Z return mod(**inputs) 2025-08-14T21:54:48.1277875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1278247Z outputs = self.model( 2025-08-14T21:54:48.1278604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1278985Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1279352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1279724Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1280053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1280384Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1280763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1281153Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1281533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1281920Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1282060Z 2025-08-14T21:54:48.1282137Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1282341Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1282532Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1282731Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1282950Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1283282Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1283589Z return mod(**inputs) 2025-08-14T21:54:48.1283944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1284314Z outputs = self.model( 2025-08-14T21:54:48.1284658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1285038Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1285535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1285968Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1286338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1286770Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1287181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1287585Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1287987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1288403Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1288837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1289294Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1289477Z 2025-08-14T21:54:48.1289573Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1289931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1290236Z return mod(**inputs) 2025-08-14T21:54:48.1290593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1290966Z outputs = self.model( 2025-08-14T21:54:48.1291322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1291693Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1292064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1292441Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1292764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1293106Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1293487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1293881Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1294260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1294656Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1295079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1295515Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1295667Z 2025-08-14T21:54:48.1295763Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1296099Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1296406Z return mod(**inputs) 2025-08-14T21:54:48.1296753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1297124Z outputs = self.model( 2025-08-14T21:54:48.1297482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1297858Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1298218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1298595Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1298949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1299294Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1299663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1300071Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1300456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1300832Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1300967Z 2025-08-14T21:54:48.1301064Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1301410Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1301706Z return mod(**inputs) 2025-08-14T21:54:48.1302070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1302491Z outputs = self.model( 2025-08-14T21:54:48.1302844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1303227Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1303588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1303948Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1304265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1304588Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1304964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.1305377Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1305541Z 2025-08-14T21:54:48.1305645Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1305976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1306280Z return mod(**inputs) 2025-08-14T21:54:48.1306633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1306997Z outputs = self.model( 2025-08-14T21:54:48.1307352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1307752Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1308107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1308464Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1308786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1309116Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1309476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.1309887Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1310242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.1310551Z return self.act(input) 2025-08-14T21:54:48.1310651Z 2025-08-14T21:54:48.1310744Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1311072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1311368Z return mod(**inputs) 2025-08-14T21:54:48.1311709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1312091Z outputs = self.model( 2025-08-14T21:54:48.1312432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1312800Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1313168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1313535Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1313854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1314184Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1314543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:54:48.1314919Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.1315048Z 2025-08-14T21:54:48.1315171Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1315507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1315816Z return mod(**inputs) 2025-08-14T21:54:48.1316183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1316560Z outputs = self.model( 2025-08-14T21:54:48.1316909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1317287Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1317653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1318029Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1318350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1318686Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1319056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 327, in forward 2025-08-14T21:54:48.1319438Z hidden_states = residual + hidden_states 2025-08-14T21:54:48.1319573Z 2025-08-14T21:54:48.1319669Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1319861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1319922Z return mod(**inputs) 2025-08-14T21:54:48.1320176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1320248Z outputs = self.model( 2025-08-14T21:54:48.1320499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1320573Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1320831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1320903Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1321124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1321198Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1321449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1321545Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1321796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1321947Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1321986Z 2025-08-14T21:54:48.1322087Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1322274Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1322343Z return mod(**inputs) 2025-08-14T21:54:48.1322598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1322684Z outputs = self.model( 2025-08-14T21:54:48.1322949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1323021Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1323281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1323350Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1323579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1323666Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1323925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1324017Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1324271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1324347Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1324351Z 2025-08-14T21:54:48.1324454Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1324641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1324704Z return mod(**inputs) 2025-08-14T21:54:48.1324962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1325030Z outputs = self.model( 2025-08-14T21:54:48.1325354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1325445Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1325724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1325809Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1326036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1326120Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1326416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1326511Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1326796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1326880Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1326884Z 2025-08-14T21:54:48.1326968Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1327055Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1327134Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1327211Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1327325Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1327516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1327588Z return mod(**inputs) 2025-08-14T21:54:48.1327840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1327905Z outputs = self.model( 2025-08-14T21:54:48.1328189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1328260Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1328512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1328595Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1328804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1328886Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1329131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1329216Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1329470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1329579Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1329870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1330015Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1330020Z 2025-08-14T21:54:48.1330119Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1330313Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1330376Z return mod(**inputs) 2025-08-14T21:54:48.1330632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1330699Z outputs = self.model( 2025-08-14T21:54:48.1330948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1331028Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1331272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1331340Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1331554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1331630Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1331883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1331967Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1332215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1332313Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1332594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1332705Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1332709Z 2025-08-14T21:54:48.1332807Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1332997Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1333066Z return mod(**inputs) 2025-08-14T21:54:48.1333331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1333392Z outputs = self.model( 2025-08-14T21:54:48.1333645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1333711Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1333967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1334057Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1334267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1334347Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1334618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1334703Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1334956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1335031Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1335035Z 2025-08-14T21:54:48.1335137Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1335335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1335400Z return mod(**inputs) 2025-08-14T21:54:48.1335657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1335738Z outputs = self.model( 2025-08-14T21:54:48.1336000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1336069Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1336323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1336395Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1336596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1336668Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1336916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.1337027Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1337031Z 2025-08-14T21:54:48.1337132Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1337314Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1337375Z return mod(**inputs) 2025-08-14T21:54:48.1337786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1337860Z outputs = self.model( 2025-08-14T21:54:48.1338123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1338193Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1338443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1338526Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1338735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1338811Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1339068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.1339183Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1339402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.1339467Z return self.act(input) 2025-08-14T21:54:48.1339470Z 2025-08-14T21:54:48.1339567Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1339760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1339870Z return mod(**inputs) 2025-08-14T21:54:48.1340122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1340185Z outputs = self.model( 2025-08-14T21:54:48.1340431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1340532Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1340778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1340847Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1341063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1341137Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1341415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:54:48.1341498Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.1341502Z 2025-08-14T21:54:48.1341597Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1341820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1341885Z return mod(**inputs) 2025-08-14T21:54:48.1342135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1342213Z outputs = self.model( 2025-08-14T21:54:48.1342455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1342530Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1342768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1342836Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1343046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1343119Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1343369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1343452Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1343697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1343842Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1343846Z 2025-08-14T21:54:48.1343939Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1344129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1344194Z return mod(**inputs) 2025-08-14T21:54:48.1344436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1344509Z outputs = self.model( 2025-08-14T21:54:48.1344761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1344830Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1345087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1345155Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1345368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1345442Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1345753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1345844Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1346100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1346189Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1346200Z 2025-08-14T21:54:48.1346295Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1346476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1346544Z return mod(**inputs) 2025-08-14T21:54:48.1346786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1346849Z outputs = self.model( 2025-08-14T21:54:48.1347116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1347185Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1347433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1347516Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1347720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1347797Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1348037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1348118Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1348367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1348445Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1348451Z 2025-08-14T21:54:48.1348531Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1348603Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1348673Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1348752Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1348846Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1349027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1349094Z return mod(**inputs) 2025-08-14T21:54:48.1349334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1349403Z outputs = self.model( 2025-08-14T21:54:48.1349643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1349709Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1349957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1350023Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1350223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1350303Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1350543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1350630Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1350868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1350958Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1351231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1351370Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1351374Z 2025-08-14T21:54:48.1351476Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1351661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1351738Z return mod(**inputs) 2025-08-14T21:54:48.1351995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1352058Z outputs = self.model( 2025-08-14T21:54:48.1352302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1352377Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1352620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1352719Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1352922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1353009Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1353266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1353349Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1353601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1353690Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1353961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1354070Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1354076Z 2025-08-14T21:54:48.1354169Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1354361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1354423Z return mod(**inputs) 2025-08-14T21:54:48.1354671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1354740Z outputs = self.model( 2025-08-14T21:54:48.1354984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1355052Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1355300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1355367Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1355578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1355650Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1355894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1355983Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1356227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1356300Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1356310Z 2025-08-14T21:54:48.1356403Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1356583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1356653Z return mod(**inputs) 2025-08-14T21:54:48.1356897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1356987Z outputs = self.model( 2025-08-14T21:54:48.1357235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1357303Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1357575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1357642Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1357850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1357931Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1358186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.1358294Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1358316Z 2025-08-14T21:54:48.1358420Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1358605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1358689Z return mod(**inputs) 2025-08-14T21:54:48.1358934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1358996Z outputs = self.model( 2025-08-14T21:54:48.1359243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1359311Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1359555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1359622Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1359826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1359908Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1360147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.1360255Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1360453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.1360517Z return self.act(input) 2025-08-14T21:54:48.1360520Z 2025-08-14T21:54:48.1360621Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1360797Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1360856Z return mod(**inputs) 2025-08-14T21:54:48.1361104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1361168Z outputs = self.model( 2025-08-14T21:54:48.1361410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1361485Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1361722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1361796Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1361997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1362067Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1362311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:54:48.1362386Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.1362403Z 2025-08-14T21:54:48.1362505Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1362688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1362748Z return mod(**inputs) 2025-08-14T21:54:48.1362996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1363809Z outputs = self.model( 2025-08-14T21:54:48.1364054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1364129Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1364370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1364444Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1364670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1364747Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1365003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 327, in forward 2025-08-14T21:54:48.1365098Z hidden_states = residual + hidden_states 2025-08-14T21:54:48.1365103Z 2025-08-14T21:54:48.1365208Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1365484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1365551Z return mod(**inputs) 2025-08-14T21:54:48.1365843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1365915Z outputs = self.model( 2025-08-14T21:54:48.1366193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1366285Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1366561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1366638Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1366855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1366932Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1367195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1367285Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1367543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1367698Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1367704Z 2025-08-14T21:54:48.1367813Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1368008Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1368071Z return mod(**inputs) 2025-08-14T21:54:48.1368383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1368466Z outputs = self.model( 2025-08-14T21:54:48.1368720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1368798Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1369045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1369114Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1369332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1369423Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1369670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1369765Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1370033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1370119Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1370123Z 2025-08-14T21:54:48.1370220Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1370405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1370474Z return mod(**inputs) 2025-08-14T21:54:48.1370744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1370818Z outputs = self.model( 2025-08-14T21:54:48.1371070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1371154Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1371411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1371478Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1371684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1371765Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1372013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1372103Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1372355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1372436Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1372440Z 2025-08-14T21:54:48.1372525Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1372600Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1372679Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1372752Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1372849Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1373047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1373109Z return mod(**inputs) 2025-08-14T21:54:48.1373356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1373428Z outputs = self.model( 2025-08-14T21:54:48.1373679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1373757Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1374003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1374072Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1374286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1374360Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1374607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1374700Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1374945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1375064Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1375340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1375467Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1375486Z 2025-08-14T21:54:48.1375591Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1375778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1375849Z return mod(**inputs) 2025-08-14T21:54:48.1376099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1376163Z outputs = self.model( 2025-08-14T21:54:48.1376420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1376507Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1376758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1376837Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1377066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1377150Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1377396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1377480Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1377732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1377823Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1378099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1378210Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1378214Z 2025-08-14T21:54:48.1378312Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1378510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1378572Z return mod(**inputs) 2025-08-14T21:54:48.1378822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1378893Z outputs = self.model( 2025-08-14T21:54:48.1379141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1379217Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1379463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1379532Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1379746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1379820Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1380068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1380161Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1380408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1380489Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1380493Z 2025-08-14T21:54:48.1380590Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1380776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1380869Z return mod(**inputs) 2025-08-14T21:54:48.1381120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1381192Z outputs = self.model( 2025-08-14T21:54:48.1381463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1381531Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1381778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1381844Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1382044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1382125Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1382385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.1382503Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1382506Z 2025-08-14T21:54:48.1382613Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1382800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1382870Z return mod(**inputs) 2025-08-14T21:54:48.1383111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1383182Z outputs = self.model( 2025-08-14T21:54:48.1383421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1383489Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1383738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1383807Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1384008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1384090Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1384330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.1384444Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1384635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.1384698Z return self.act(input) 2025-08-14T21:54:48.1384701Z 2025-08-14T21:54:48.1384802Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1384983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1385053Z return mod(**inputs) 2025-08-14T21:54:48.1385296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1385357Z outputs = self.model( 2025-08-14T21:54:48.1385607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1385675Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1385915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1385989Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1386193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1386274Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1386530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:54:48.1386622Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.1386625Z 2025-08-14T21:54:48.1386728Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1386911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1386989Z return mod(**inputs) 2025-08-14T21:54:48.1387240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1387303Z outputs = self.model( 2025-08-14T21:54:48.1387552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1387619Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1387859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1387949Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1388154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1388248Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1388492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1388575Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1388819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1388957Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1388961Z 2025-08-14T21:54:48.1389053Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1389244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1389307Z return mod(**inputs) 2025-08-14T21:54:48.1389556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1389620Z outputs = self.model( 2025-08-14T21:54:48.1389866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1389943Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1390182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1390255Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1390452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1390522Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1390790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1390874Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1391114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1391195Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1391198Z 2025-08-14T21:54:48.1391291Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1391476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1391536Z return mod(**inputs) 2025-08-14T21:54:48.1391777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1391846Z outputs = self.model( 2025-08-14T21:54:48.1392085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1392175Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1392415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1392483Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1392707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1392778Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1393014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1393104Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1393342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1393428Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1393448Z 2025-08-14T21:54:48.1393523Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1393595Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1393676Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1393768Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1393864Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1394050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1394110Z return mod(**inputs) 2025-08-14T21:54:48.1394356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1394419Z outputs = self.model( 2025-08-14T21:54:48.1394664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1394739Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1394986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1395055Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1395277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1395356Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1395622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1395706Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1395948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1396045Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1396316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1396448Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1396452Z 2025-08-14T21:54:48.1396547Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1396734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1396805Z return mod(**inputs) 2025-08-14T21:54:48.1397052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1397117Z outputs = self.model( 2025-08-14T21:54:48.1397367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1397436Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1397686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1397776Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1397988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1398069Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1398321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1398437Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1398687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1398776Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1399053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1399155Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1399184Z 2025-08-14T21:54:48.1399283Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1399476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1399552Z return mod(**inputs) 2025-08-14T21:54:48.1399811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1399876Z outputs = self.model( 2025-08-14T21:54:48.1400121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1400198Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1400443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1400518Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1400725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1400799Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1401056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1401143Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1401388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1401475Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1401479Z 2025-08-14T21:54:48.1401578Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1401774Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1401838Z return mod(**inputs) 2025-08-14T21:54:48.1402094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1402169Z outputs = self.model( 2025-08-14T21:54:48.1402421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1402501Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1402760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1402828Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1403042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1403114Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1403357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.1403475Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1403501Z 2025-08-14T21:54:48.1403598Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1403788Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1403853Z return mod(**inputs) 2025-08-14T21:54:48.1404117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1404188Z outputs = self.model( 2025-08-14T21:54:48.1404438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1404505Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1404757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1404824Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1405047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1405124Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1405467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.1405601Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1405823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.1405908Z return self.act(input) 2025-08-14T21:54:48.1405912Z 2025-08-14T21:54:48.1406022Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1406245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1406318Z return mod(**inputs) 2025-08-14T21:54:48.1406578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1406649Z outputs = self.model( 2025-08-14T21:54:48.1406914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1406989Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1407253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1407326Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1407547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1407632Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1407884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:54:48.1407971Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.1407976Z 2025-08-14T21:54:48.1408076Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1408266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1408340Z return mod(**inputs) 2025-08-14T21:54:48.1408590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1408660Z outputs = self.model( 2025-08-14T21:54:48.1408916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1408987Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1409241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1409311Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1409521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1409621Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1409868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 327, in forward 2025-08-14T21:54:48.1409945Z hidden_states = residual + hidden_states 2025-08-14T21:54:48.1409973Z 2025-08-14T21:54:48.1410069Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1410254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1410324Z return mod(**inputs) 2025-08-14T21:54:48.1410572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1410636Z outputs = self.model( 2025-08-14T21:54:48.1410890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1410973Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1411230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1411297Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1411520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1411603Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1411853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1411939Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1412196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1412336Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1412340Z 2025-08-14T21:54:48.1412445Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1412630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1412692Z return mod(**inputs) 2025-08-14T21:54:48.1412952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1413016Z outputs = self.model( 2025-08-14T21:54:48.1413278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1413347Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1413595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1413668Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1413875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1413948Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1414204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1414289Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1414547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1414622Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1414627Z 2025-08-14T21:54:48.1414721Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1414913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1414973Z return mod(**inputs) 2025-08-14T21:54:48.1415230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1415311Z outputs = self.model( 2025-08-14T21:54:48.1415560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1415638Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1415900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1415968Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1416180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1416253Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1416510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1416593Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1416854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1416946Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1416950Z 2025-08-14T21:54:48.1417024Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1417146Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1417225Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1417297Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1417403Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1417590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1417652Z return mod(**inputs) 2025-08-14T21:54:48.1417909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1417972Z outputs = self.model( 2025-08-14T21:54:48.1418234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1418310Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1418551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1418623Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1418825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1418897Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1419143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1419226Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1419474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1419566Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1419833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1419963Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1419968Z 2025-08-14T21:54:48.1420063Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1420257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1420318Z return mod(**inputs) 2025-08-14T21:54:48.1420560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1420629Z outputs = self.model( 2025-08-14T21:54:48.1420872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1420959Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1421208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1421275Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1421486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1421582Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1421819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1421908Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1422146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1422234Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1422519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1422623Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1422626Z 2025-08-14T21:54:48.1422743Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1422929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1422989Z return mod(**inputs) 2025-08-14T21:54:48.1423240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1423302Z outputs = self.model( 2025-08-14T21:54:48.1423551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1423618Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1423855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1423929Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1424128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1424202Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1424457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 312, in forward 2025-08-14T21:54:48.1424540Z hidden_states, attn_weights = self.self_attn( 2025-08-14T21:54:48.1424794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1424872Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1424875Z 2025-08-14T21:54:48.1424970Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1425162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1425224Z return mod(**inputs) 2025-08-14T21:54:48.1425480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1425547Z outputs = self.model( 2025-08-14T21:54:48.1425800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1425879Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1426142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1426211Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1426423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1426494Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1426747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.1426877Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1426880Z 2025-08-14T21:54:48.1426978Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1427187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1427248Z return mod(**inputs) 2025-08-14T21:54:48.1427509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1427571Z outputs = self.model( 2025-08-14T21:54:48.1427811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1427883Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1428135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1428204Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1428414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1428499Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1428749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 323, in forward 2025-08-14T21:54:48.1428857Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1429053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.1429125Z return self.act(input) 2025-08-14T21:54:48.1429129Z 2025-08-14T21:54:48.1429226Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1429420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1429484Z return mod(**inputs) 2025-08-14T21:54:48.1429737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1429807Z outputs = self.model( 2025-08-14T21:54:48.1430061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1279, in forward 2025-08-14T21:54:48.1430130Z encoder_outputs = self.encoder( 2025-08-14T21:54:48.1430386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 841, in forward 2025-08-14T21:54:48.1430455Z layer_outputs = encoder_layer( 2025-08-14T21:54:48.1430673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1430748Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1430998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 325, in forward 2025-08-14T21:54:48.1431082Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.1431086Z 2025-08-14T21:54:48.1431180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1431371Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1431440Z return mod(**inputs) 2025-08-14T21:54:48.1431697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1431768Z outputs = self.model( 2025-08-14T21:54:48.1432019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1432086Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1432347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1432437Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1432651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1432725Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1432982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1433097Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1433336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1433472Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1433482Z 2025-08-14T21:54:48.1433572Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1433750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1433834Z return mod(**inputs) 2025-08-14T21:54:48.1434077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1434139Z outputs = self.model( 2025-08-14T21:54:48.1434427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1434495Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1434751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1434817Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1435024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1435102Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1435351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1435447Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1435703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1435779Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1435783Z 2025-08-14T21:54:48.1435885Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1436071Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1436133Z return mod(**inputs) 2025-08-14T21:54:48.1436390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1436453Z outputs = self.model( 2025-08-14T21:54:48.1436711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1436780Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1437035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1437109Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1437314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1437386Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1437809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1437909Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1438164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1438246Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1438308Z 2025-08-14T21:54:48.1438387Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1438471Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1438544Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1438617Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1438745Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1438930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1438999Z return mod(**inputs) 2025-08-14T21:54:48.1439246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1439310Z outputs = self.model( 2025-08-14T21:54:48.1439566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1439636Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1439914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1439993Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1440227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1440311Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1440560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1440652Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1440909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1441000Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1441281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1441405Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1441408Z 2025-08-14T21:54:48.1441505Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1441697Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1441760Z return mod(**inputs) 2025-08-14T21:54:48.1442007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1442078Z outputs = self.model( 2025-08-14T21:54:48.1442322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1442395Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1442645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1442714Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1442926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1442999Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1443254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1443345Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1443588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1443683Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1443951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1444055Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1444089Z 2025-08-14T21:54:48.1444186Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1444370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1444439Z return mod(**inputs) 2025-08-14T21:54:48.1444704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1444769Z outputs = self.model( 2025-08-14T21:54:48.1445022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1445090Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1445393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1445467Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1445721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1445810Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1446078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1446179Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1446447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1446528Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1446532Z 2025-08-14T21:54:48.1446642Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1446838Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1446902Z return mod(**inputs) 2025-08-14T21:54:48.1447176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1447243Z outputs = self.model( 2025-08-14T21:54:48.1447513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1447583Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1447831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1447905Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1448112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1448185Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1448438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1448541Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1448792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1448934Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1448938Z 2025-08-14T21:54:48.1449034Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1449227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1449287Z return mod(**inputs) 2025-08-14T21:54:48.1449543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1449607Z outputs = self.model( 2025-08-14T21:54:48.1449852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1449926Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1450192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1450259Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1450473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1450563Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1450816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1450917Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1451166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1451248Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1451251Z 2025-08-14T21:54:48.1451347Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1451565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1451627Z return mod(**inputs) 2025-08-14T21:54:48.1451886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1451957Z outputs = self.model( 2025-08-14T21:54:48.1452197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1452264Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1452513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1452579Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1452786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1452859Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1453099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1453205Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1453444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1453524Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1453535Z 2025-08-14T21:54:48.1453608Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1453681Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1453758Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1453828Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1453921Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1454109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1454172Z return mod(**inputs) 2025-08-14T21:54:48.1454411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1454479Z outputs = self.model( 2025-08-14T21:54:48.1454730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1454811Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1455061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1455130Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1455362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1455435Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1455685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1455802Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1456048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1456163Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1456443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1456564Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1456575Z 2025-08-14T21:54:48.1456669Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1456849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1456916Z return mod(**inputs) 2025-08-14T21:54:48.1457176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1457240Z outputs = self.model( 2025-08-14T21:54:48.1457503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1457574Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1457827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1457894Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1458093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1458170Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1458409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1458508Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1458755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1458845Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1459118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1459216Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1459220Z 2025-08-14T21:54:48.1459313Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1459504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1459565Z return mod(**inputs) 2025-08-14T21:54:48.1459811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1459875Z outputs = self.model( 2025-08-14T21:54:48.1460116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1460190Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1460428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1460495Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1460703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1460774Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1461019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1461114Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1461355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1461454Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1461458Z 2025-08-14T21:54:48.1461551Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1461738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1461815Z return mod(**inputs) 2025-08-14T21:54:48.1462054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1462122Z outputs = self.model( 2025-08-14T21:54:48.1462364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1462431Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1462693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1462761Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1462968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1463061Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1463306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1463420Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1463424Z 2025-08-14T21:54:48.1463517Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1463705Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1463767Z return mod(**inputs) 2025-08-14T21:54:48.1464009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1464081Z outputs = self.model( 2025-08-14T21:54:48.1464320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1464387Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1464632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1464697Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1464901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1464973Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1465214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1465329Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1465524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.1465587Z return self.act(input) 2025-08-14T21:54:48.1465597Z 2025-08-14T21:54:48.1465691Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1465875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1465942Z return mod(**inputs) 2025-08-14T21:54:48.1466185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1466247Z outputs = self.model( 2025-08-14T21:54:48.1466498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1466566Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1466815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1466901Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1467103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1467182Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1467422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:48.1467514Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.1467517Z 2025-08-14T21:54:48.1467617Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1467799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1467868Z return mod(**inputs) 2025-08-14T21:54:48.1468109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1468173Z outputs = self.model( 2025-08-14T21:54:48.1468437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1468506Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1468759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1468837Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1469040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1469118Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1469358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1469448Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1469698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1469837Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1469841Z 2025-08-14T21:54:48.1469942Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1470123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1470184Z return mod(**inputs) 2025-08-14T21:54:48.1470437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1470498Z outputs = self.model( 2025-08-14T21:54:48.1470740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1470813Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1471055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1471129Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1471330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1471403Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1471651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1471740Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1471989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1472062Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1472065Z 2025-08-14T21:54:48.1472157Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1472346Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1472425Z return mod(**inputs) 2025-08-14T21:54:48.1472673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1472743Z outputs = self.model( 2025-08-14T21:54:48.1472988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1473079Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1473321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1473387Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1473595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1473666Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1473928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1474021Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1474281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1474371Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1474374Z 2025-08-14T21:54:48.1474447Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1474519Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1474598Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1474666Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1474763Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1474944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1475003Z return mod(**inputs) 2025-08-14T21:54:48.1475251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1475315Z outputs = self.model( 2025-08-14T21:54:48.1475554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1475628Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1475865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1475935Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1476134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1476204Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1476448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1476538Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1476785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1476873Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1477140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1477266Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1477269Z 2025-08-14T21:54:48.1477361Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1477541Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1477607Z return mod(**inputs) 2025-08-14T21:54:48.1477848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1477934Z outputs = self.model( 2025-08-14T21:54:48.1478180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1478247Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1478497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1478580Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1478789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1478862Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1479106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1479200Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1479464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1479555Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1479849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1479952Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1479955Z 2025-08-14T21:54:48.1480057Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1480236Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1480296Z return mod(**inputs) 2025-08-14T21:54:48.1480543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1480606Z outputs = self.model( 2025-08-14T21:54:48.1480852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1480921Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1481161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1481236Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1481439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1481512Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1481761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1481849Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1482100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1482174Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1482181Z 2025-08-14T21:54:48.1482274Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1482464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1482526Z return mod(**inputs) 2025-08-14T21:54:48.1482766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1482835Z outputs = self.model( 2025-08-14T21:54:48.1483072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1483146Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1483382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1483447Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1483653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1483742Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1483989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1484105Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1484350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1484492Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1484495Z 2025-08-14T21:54:48.1484589Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1484778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1484838Z return mod(**inputs) 2025-08-14T21:54:48.1485124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1485197Z outputs = self.model( 2025-08-14T21:54:48.1485607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1485689Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1485963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1486032Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1486257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1486345Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1486597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1486709Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1486971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1487043Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1487058Z 2025-08-14T21:54:48.1487154Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1487334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1487403Z return mod(**inputs) 2025-08-14T21:54:48.1487666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1487734Z outputs = self.model( 2025-08-14T21:54:48.1488005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1488077Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1488353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1488423Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1488643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1488728Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1488988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1489091Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1489360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1489444Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1489448Z 2025-08-14T21:54:48.1489532Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1489634Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1489710Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1489793Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1489892Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1490088Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1490175Z return mod(**inputs) 2025-08-14T21:54:48.1490439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1490512Z outputs = self.model( 2025-08-14T21:54:48.1490774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1490846Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1491126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1491198Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1491418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1491518Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1491782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1491893Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1492154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1492249Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1492548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1492683Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1492687Z 2025-08-14T21:54:48.1492796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1492989Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1493056Z return mod(**inputs) 2025-08-14T21:54:48.1493328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1493395Z outputs = self.model( 2025-08-14T21:54:48.1493663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1493735Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1493996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1494072Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1494293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1494371Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1494640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1494747Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1495017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1495122Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1495394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1495501Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1495507Z 2025-08-14T21:54:48.1495624Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1495833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1495897Z return mod(**inputs) 2025-08-14T21:54:48.1496145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1496231Z outputs = self.model( 2025-08-14T21:54:48.1496470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1496537Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1496781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1496848Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1497060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1497151Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1497392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1497512Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1497757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1497832Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1497845Z 2025-08-14T21:54:48.1497938Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1498116Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1498183Z return mod(**inputs) 2025-08-14T21:54:48.1498424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1498488Z outputs = self.model( 2025-08-14T21:54:48.1498733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1498798Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1499046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1499113Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1499311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1499389Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1499629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1499738Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1499748Z 2025-08-14T21:54:48.1499845Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1500025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1500094Z return mod(**inputs) 2025-08-14T21:54:48.1500334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1500398Z outputs = self.model( 2025-08-14T21:54:48.1500643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1500711Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1500959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1501026Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1501227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1501325Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1501564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1501673Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1501892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.1501958Z return self.act(input) 2025-08-14T21:54:48.1501961Z 2025-08-14T21:54:48.1502061Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1502242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1502302Z return mod(**inputs) 2025-08-14T21:54:48.1502554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1502618Z outputs = self.model( 2025-08-14T21:54:48.1502911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1502981Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1503236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1503310Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1503513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1503585Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1503835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:48.1503909Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.1503912Z 2025-08-14T21:54:48.1504011Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1504194Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1504255Z return mod(**inputs) 2025-08-14T21:54:48.1504511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1504573Z outputs = self.model( 2025-08-14T21:54:48.1504815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1504890Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1505133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1505205Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1505406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1505477Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1505730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-14T21:54:48.1505805Z hidden_states = residual + hidden_states 2025-08-14T21:54:48.1505808Z 2025-08-14T21:54:48.1505908Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1506091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1506150Z return mod(**inputs) 2025-08-14T21:54:48.1506400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1506463Z outputs = self.model( 2025-08-14T21:54:48.1506705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1506781Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1507042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1507114Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1507317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1507414Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1507661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1507752Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1507999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1508134Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1508138Z 2025-08-14T21:54:48.1508230Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1508434Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1508496Z return mod(**inputs) 2025-08-14T21:54:48.1508754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1508828Z outputs = self.model( 2025-08-14T21:54:48.1509072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1509147Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1509390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1509456Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1509665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1509739Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1509988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1510082Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1510327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1510408Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1510411Z 2025-08-14T21:54:48.1510502Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1510682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1510751Z return mod(**inputs) 2025-08-14T21:54:48.1510990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1511062Z outputs = self.model( 2025-08-14T21:54:48.1511304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1511370Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1511619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1511694Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1511896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1511974Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1512214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1512309Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1512548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1512643Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1512646Z 2025-08-14T21:54:48.1512726Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1512800Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1512878Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1512966Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1513061Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1513250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1513309Z return mod(**inputs) 2025-08-14T21:54:48.1513555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1513626Z outputs = self.model( 2025-08-14T21:54:48.1513881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1513959Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1514198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1514281Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1514494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1514564Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1514806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1514903Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1515146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1515243Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1515510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1515630Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1515636Z 2025-08-14T21:54:48.1515739Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1515922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1515989Z return mod(**inputs) 2025-08-14T21:54:48.1516231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1516294Z outputs = self.model( 2025-08-14T21:54:48.1516541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1516607Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1516850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1516924Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1517129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1517210Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1517451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1517540Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1517793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1517881Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1518155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1518274Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1518278Z 2025-08-14T21:54:48.1518372Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1518562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1518638Z return mod(**inputs) 2025-08-14T21:54:48.1518880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1518951Z outputs = self.model( 2025-08-14T21:54:48.1519190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1519263Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1519503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1519588Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1519801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1519886Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1520136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1520225Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1520468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1520551Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1520554Z 2025-08-14T21:54:48.1520648Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1520830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1520900Z return mod(**inputs) 2025-08-14T21:54:48.1521141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1521209Z outputs = self.model( 2025-08-14T21:54:48.1521452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1521519Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1521768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1521832Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1522036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1522114Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1522354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1522461Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1522704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1522841Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1522845Z 2025-08-14T21:54:48.1522945Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1523129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1523195Z return mod(**inputs) 2025-08-14T21:54:48.1523437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1523500Z outputs = self.model( 2025-08-14T21:54:48.1523750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1523843Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1524097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1524167Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1524388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1524469Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1524717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1524817Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1525070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1525147Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1525194Z 2025-08-14T21:54:48.1525364Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1525566Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1525655Z return mod(**inputs) 2025-08-14T21:54:48.1525925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1525993Z outputs = self.model( 2025-08-14T21:54:48.1526255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1526339Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1526611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1526690Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1526898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1526971Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1527228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1527333Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1527594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1527677Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1527681Z 2025-08-14T21:54:48.1527758Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1527843Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1527916Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1527990Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1528096Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1528290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1528360Z return mod(**inputs) 2025-08-14T21:54:48.1528613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1528679Z outputs = self.model( 2025-08-14T21:54:48.1528939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1529009Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1529258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1529336Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1529546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1529658Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1529908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1530011Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1530268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1530381Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1530670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1530794Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1530798Z 2025-08-14T21:54:48.1530897Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1531095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1531178Z return mod(**inputs) 2025-08-14T21:54:48.1531438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1531511Z outputs = self.model( 2025-08-14T21:54:48.1531782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1531861Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1532121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1532190Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1532409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1532482Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1532748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1532849Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1533107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1533207Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1533490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1533592Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1533603Z 2025-08-14T21:54:48.1533701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1533892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1533962Z return mod(**inputs) 2025-08-14T21:54:48.1534225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1534290Z outputs = self.model( 2025-08-14T21:54:48.1534556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1534626Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1534890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1534959Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1535172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1535254Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1535511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1535640Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1535887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1535962Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1535966Z 2025-08-14T21:54:48.1536084Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1536264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1536324Z return mod(**inputs) 2025-08-14T21:54:48.1536572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1536634Z outputs = self.model( 2025-08-14T21:54:48.1536882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1536950Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1537210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1537286Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1537504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1537579Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1538016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1538133Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1538137Z 2025-08-14T21:54:48.1538242Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1538424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1538485Z return mod(**inputs) 2025-08-14T21:54:48.1538741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1538804Z outputs = self.model( 2025-08-14T21:54:48.1539046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1539123Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1539362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1539437Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1539641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1539714Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1539969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1540085Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1540300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.1540365Z return self.act(input) 2025-08-14T21:54:48.1540369Z 2025-08-14T21:54:48.1540465Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1540656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1540719Z return mod(**inputs) 2025-08-14T21:54:48.1540957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1541026Z outputs = self.model( 2025-08-14T21:54:48.1541265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1541338Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1541619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1541685Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1541891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1541986Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1542237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:48.1542311Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.1542315Z 2025-08-14T21:54:48.1542407Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1542593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1542654Z return mod(**inputs) 2025-08-14T21:54:48.1542919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1542991Z outputs = self.model( 2025-08-14T21:54:48.1543236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1543335Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1543579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1543645Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1543850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1543922Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1544158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1544258Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1544498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1544641Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1544646Z 2025-08-14T21:54:48.1544740Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1544916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1544983Z return mod(**inputs) 2025-08-14T21:54:48.1545221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1545290Z outputs = self.model( 2025-08-14T21:54:48.1545527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1545593Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1545846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1545912Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1546132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1546213Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1546450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1546546Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1546783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1546855Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1546858Z 2025-08-14T21:54:48.1546956Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1547154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1547220Z return mod(**inputs) 2025-08-14T21:54:48.1547460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1547538Z outputs = self.model( 2025-08-14T21:54:48.1547786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1547853Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1548090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1548161Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1548363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1548459Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1548704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1548794Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1549069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1549150Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1549154Z 2025-08-14T21:54:48.1549236Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1549308Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1549380Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1549458Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1549551Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1549737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1549808Z return mod(**inputs) 2025-08-14T21:54:48.1550052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1550122Z outputs = self.model( 2025-08-14T21:54:48.1550366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1550435Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1550683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1550748Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1550950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1551028Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1551271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1551370Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1551614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1551704Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1551979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1552100Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1552103Z 2025-08-14T21:54:48.1552206Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1552391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1552450Z return mod(**inputs) 2025-08-14T21:54:48.1552707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1552788Z outputs = self.model( 2025-08-14T21:54:48.1553030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1553122Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1553365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1553439Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1553638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1553709Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1553955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1554044Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1554311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1554403Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1554689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1554799Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1554803Z 2025-08-14T21:54:48.1554899Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1555086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1555156Z return mod(**inputs) 2025-08-14T21:54:48.1555445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1555516Z outputs = self.model( 2025-08-14T21:54:48.1555762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1555829Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1556078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1556147Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1556352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1556432Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1556680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1556778Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1557027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1557104Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1557107Z 2025-08-14T21:54:48.1557213Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1557400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1557473Z return mod(**inputs) 2025-08-14T21:54:48.1557720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1557783Z outputs = self.model( 2025-08-14T21:54:48.1558035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1558104Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1558351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1558447Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1558654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1558733Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1558979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 416, in forward 2025-08-14T21:54:48.1559072Z hidden_states = residual + hidden_states 2025-08-14T21:54:48.1559075Z 2025-08-14T21:54:48.1559178Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1559367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1559434Z return mod(**inputs) 2025-08-14T21:54:48.1559688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1559750Z outputs = self.model( 2025-08-14T21:54:48.1560025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1560094Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1560361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1560436Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1560642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1560722Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1560968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1561068Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1561323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1561462Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1561466Z 2025-08-14T21:54:48.1561567Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1561755Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1561819Z return mod(**inputs) 2025-08-14T21:54:48.1562080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1562144Z outputs = self.model( 2025-08-14T21:54:48.1562393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1562469Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1562719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1562796Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1563004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1563078Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1563335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1563436Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1563693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1563769Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1563772Z 2025-08-14T21:54:48.1563869Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1564062Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1564144Z return mod(**inputs) 2025-08-14T21:54:48.1564398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1564468Z outputs = self.model( 2025-08-14T21:54:48.1564717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1564809Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1565056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1565123Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1565412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1565494Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1565768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1565884Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1566168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1566271Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1566275Z 2025-08-14T21:54:48.1566361Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1566446Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1566536Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1566618Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1566738Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1566926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1566988Z return mod(**inputs) 2025-08-14T21:54:48.1567250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1567316Z outputs = self.model( 2025-08-14T21:54:48.1567567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1567647Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1567896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1567973Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1568179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1568262Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1568521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1568624Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1568869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1568967Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1569243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1569375Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1569378Z 2025-08-14T21:54:48.1569473Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1569660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1569730Z return mod(**inputs) 2025-08-14T21:54:48.1569978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1570069Z outputs = self.model( 2025-08-14T21:54:48.1570317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1570386Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1570643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1570727Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1570935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1571016Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1571262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1571368Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1571638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1571732Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1572028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1572130Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1572133Z 2025-08-14T21:54:48.1572235Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1572418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1572479Z return mod(**inputs) 2025-08-14T21:54:48.1572730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1572793Z outputs = self.model( 2025-08-14T21:54:48.1573034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1573108Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1573348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1573423Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1573622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1573693Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1573942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1574039Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1574283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1574358Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1574363Z 2025-08-14T21:54:48.1574455Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1574642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1574702Z return mod(**inputs) 2025-08-14T21:54:48.1574943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1575013Z outputs = self.model( 2025-08-14T21:54:48.1575252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1575325Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1575571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1575637Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1575867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1575949Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1576197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1576323Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1576326Z 2025-08-14T21:54:48.1576418Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1576606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1576666Z return mod(**inputs) 2025-08-14T21:54:48.1576902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1576970Z outputs = self.model( 2025-08-14T21:54:48.1577223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1577300Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1577541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1577620Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1577831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1577902Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1578141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1578258Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1578451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.1578522Z return self.act(input) 2025-08-14T21:54:48.1578526Z 2025-08-14T21:54:48.1578621Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1578802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1578871Z return mod(**inputs) 2025-08-14T21:54:48.1579115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1579186Z outputs = self.model( 2025-08-14T21:54:48.1579430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1579497Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1579745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1579815Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1580020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1580101Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1580345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:48.1580426Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.1580430Z 2025-08-14T21:54:48.1580523Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1580704Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1580771Z return mod(**inputs) 2025-08-14T21:54:48.1581014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1581084Z outputs = self.model( 2025-08-14T21:54:48.1581327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1581412Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1581663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1581731Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1581948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1582027Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1582278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1582378Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1582625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1582763Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1582785Z 2025-08-14T21:54:48.1582890Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1583076Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1583162Z return mod(**inputs) 2025-08-14T21:54:48.1583415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1583478Z outputs = self.model( 2025-08-14T21:54:48.1583734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1583812Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1584054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1584129Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1584335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1584414Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1584662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1584754Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1585006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1585082Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1585085Z 2025-08-14T21:54:48.1585188Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1585376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1585437Z return mod(**inputs) 2025-08-14T21:54:48.1585696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1585761Z outputs = self.model( 2025-08-14T21:54:48.1586012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1586090Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1586340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1586415Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1586624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1586696Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1586975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1587063Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1587338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1587425Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1587428Z 2025-08-14T21:54:48.1587502Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1587613Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1587687Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1587759Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1587860Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1588043Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1588105Z return mod(**inputs) 2025-08-14T21:54:48.1588362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1588426Z outputs = self.model( 2025-08-14T21:54:48.1588700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1588771Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1589030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1589108Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1589314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1589394Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1589643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1589735Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1589993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1590083Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1590359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1590490Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1590494Z 2025-08-14T21:54:48.1590589Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1590783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1590846Z return mod(**inputs) 2025-08-14T21:54:48.1591096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1591167Z outputs = self.model( 2025-08-14T21:54:48.1591417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1591496Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1591744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1591813Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1592029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1592102Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1592348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1592468Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1592717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1592814Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1593104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1593206Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1593209Z 2025-08-14T21:54:48.1593315Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1593529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1593596Z return mod(**inputs) 2025-08-14T21:54:48.1593845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1593909Z outputs = self.model( 2025-08-14T21:54:48.1594162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1594230Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1594497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1594577Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1594807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1594893Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1595182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1595272Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1595528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1595605Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1595608Z 2025-08-14T21:54:48.1595713Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1595901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1595963Z return mod(**inputs) 2025-08-14T21:54:48.1596218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1596283Z outputs = self.model( 2025-08-14T21:54:48.1596531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1596608Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1596856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1596928Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1597134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1597206Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1597461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1597563Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1597822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1597962Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1597966Z 2025-08-14T21:54:48.1598060Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1598253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1598316Z return mod(**inputs) 2025-08-14T21:54:48.1598564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1598650Z outputs = self.model( 2025-08-14T21:54:48.1598898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1598974Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1599222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1599310Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1599524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1599597Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1599847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1599955Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1600223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1600311Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1600315Z 2025-08-14T21:54:48.1600412Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1600615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1600692Z return mod(**inputs) 2025-08-14T21:54:48.1600946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1601018Z outputs = self.model( 2025-08-14T21:54:48.1601270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1601339Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1601601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1601682Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1601890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1601968Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1602214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1602323Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1602570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1602649Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1602653Z 2025-08-14T21:54:48.1602735Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1602809Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1602889Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1602962Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1603056Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1603249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1603312Z return mod(**inputs) 2025-08-14T21:54:48.1603559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1603628Z outputs = self.model( 2025-08-14T21:54:48.1603872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1603945Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1604195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1604262Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1604494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1604571Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1604829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1604956Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1605216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1605383Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1605683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1605815Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1605819Z 2025-08-14T21:54:48.1605929Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1606148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1606225Z return mod(**inputs) 2025-08-14T21:54:48.1606514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1606583Z outputs = self.model( 2025-08-14T21:54:48.1606843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1606914Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1607169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1607253Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1607461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1607544Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1607791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1607891Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1608148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1608241Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1608532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1608639Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1608643Z 2025-08-14T21:54:48.1608744Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1608951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1609018Z return mod(**inputs) 2025-08-14T21:54:48.1609282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1609357Z outputs = self.model( 2025-08-14T21:54:48.1609619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1609702Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1609965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1610037Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1610263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1610340Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1610608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1610737Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1610995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1611101Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1611105Z 2025-08-14T21:54:48.1611206Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1611403Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1611474Z return mod(**inputs) 2025-08-14T21:54:48.1611738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1611810Z outputs = self.model( 2025-08-14T21:54:48.1612085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1612160Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1612431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1612518Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1612739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1612829Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1613092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 433, in forward 2025-08-14T21:54:48.1613181Z hidden_states = residual + hidden_states 2025-08-14T21:54:48.1613184Z 2025-08-14T21:54:48.1613288Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1613489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1613569Z return mod(**inputs) 2025-08-14T21:54:48.1613836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1613912Z outputs = self.model( 2025-08-14T21:54:48.1614178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1614255Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1614530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1614600Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1614818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1614932Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1615185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1615313Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1615316Z 2025-08-14T21:54:48.1615417Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1615610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1615687Z return mod(**inputs) 2025-08-14T21:54:48.1615950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1616022Z outputs = self.model( 2025-08-14T21:54:48.1616271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1616344Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1616600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1616709Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1616918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1617000Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1617295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1617413Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1617612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.1617678Z return self.act(input) 2025-08-14T21:54:48.1617681Z 2025-08-14T21:54:48.1617784Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1617975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1618048Z return mod(**inputs) 2025-08-14T21:54:48.1618318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1618383Z outputs = self.model( 2025-08-14T21:54:48.1618656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1618729Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1618981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1619057Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1619263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1619344Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1619596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:48.1619673Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.1619676Z 2025-08-14T21:54:48.1619779Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1619965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1620030Z return mod(**inputs) 2025-08-14T21:54:48.1620284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1620348Z outputs = self.model( 2025-08-14T21:54:48.1620599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1620668Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1620912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1620989Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1621196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1621277Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1621524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1621618Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1621872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1622011Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1622014Z 2025-08-14T21:54:48.1622118Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1622303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1622382Z return mod(**inputs) 2025-08-14T21:54:48.1622639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1622704Z outputs = self.model( 2025-08-14T21:54:48.1622951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1623046Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1623294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1623369Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1623575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1623647Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1623922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1624017Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1624281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1624368Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1624371Z 2025-08-14T21:54:48.1624466Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1624660Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1624720Z return mod(**inputs) 2025-08-14T21:54:48.1624965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1625038Z outputs = self.model( 2025-08-14T21:54:48.1625284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1625360Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1625605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1625674Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1625899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1625971Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1626220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1626318Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1626564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1626653Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1626657Z 2025-08-14T21:54:48.1626733Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1626809Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1626889Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1626960Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1627057Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1627254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1627317Z return mod(**inputs) 2025-08-14T21:54:48.1627575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1627639Z outputs = self.model( 2025-08-14T21:54:48.1627885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1627959Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1628230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1628296Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1628510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1628599Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1628854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1628944Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1629188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1629285Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1629575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1629711Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1629715Z 2025-08-14T21:54:48.1629810Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1630015Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1630087Z return mod(**inputs) 2025-08-14T21:54:48.1630337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1630403Z outputs = self.model( 2025-08-14T21:54:48.1630655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1630723Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1630975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1631044Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1631250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1631336Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1631581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1631676Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1631917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1632005Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1632277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1632376Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1632381Z 2025-08-14T21:54:48.1632482Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1632664Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1632723Z return mod(**inputs) 2025-08-14T21:54:48.1632972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1633034Z outputs = self.model( 2025-08-14T21:54:48.1633273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1633345Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1633587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1633658Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1633860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1633953Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1634204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1634309Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1634555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1634638Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1634642Z 2025-08-14T21:54:48.1634736Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1634927Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1634990Z return mod(**inputs) 2025-08-14T21:54:48.1635249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1635323Z outputs = self.model( 2025-08-14T21:54:48.1635571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1635675Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1635927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1635995Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1636209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1636282Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1636529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1636639Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1636888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1637046Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1637051Z 2025-08-14T21:54:48.1637145Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1637327Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1637397Z return mod(**inputs) 2025-08-14T21:54:48.1637803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1637888Z outputs = self.model( 2025-08-14T21:54:48.1638133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1638202Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1638457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1638523Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1638727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1638811Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1639061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1639171Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1639418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1639495Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1639498Z 2025-08-14T21:54:48.1639603Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1639836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1639905Z return mod(**inputs) 2025-08-14T21:54:48.1640156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1640243Z outputs = self.model( 2025-08-14T21:54:48.1640502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1640568Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1640822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1640893Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1641095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1641173Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1641443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1641545Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1641826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1641923Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1641926Z 2025-08-14T21:54:48.1642008Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1642081Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1642154Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1642236Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1642328Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1642510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1642584Z return mod(**inputs) 2025-08-14T21:54:48.1642826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1642888Z outputs = self.model( 2025-08-14T21:54:48.1643145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1643216Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1643471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1643537Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1643747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1643835Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1644089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1644198Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1644451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1644544Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1644830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1644956Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1644960Z 2025-08-14T21:54:48.1645063Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1645253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1645365Z return mod(**inputs) 2025-08-14T21:54:48.1645650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1645742Z outputs = self.model( 2025-08-14T21:54:48.1646029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1646139Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1646398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1646475Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1646691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1646770Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1647055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1647183Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1647472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1647585Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1647903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1648028Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1648032Z 2025-08-14T21:54:48.1648139Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1648347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1648427Z return mod(**inputs) 2025-08-14T21:54:48.1648706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1648789Z outputs = self.model( 2025-08-14T21:54:48.1649065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1649142Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1649437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1649513Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1649752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1649841Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1650131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1650251Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1650543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1650630Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1650634Z 2025-08-14T21:54:48.1650746Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1650953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1651029Z return mod(**inputs) 2025-08-14T21:54:48.1651319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1651391Z outputs = self.model( 2025-08-14T21:54:48.1651680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1651755Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1652045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1652154Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1652385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1652477Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1652755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1652882Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1652885Z 2025-08-14T21:54:48.1652987Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1653170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1653239Z return mod(**inputs) 2025-08-14T21:54:48.1653480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1653544Z outputs = self.model( 2025-08-14T21:54:48.1653809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1653877Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1654136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1654212Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1654416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1654496Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1654747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1654857Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1655070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.1655136Z return self.act(input) 2025-08-14T21:54:48.1655139Z 2025-08-14T21:54:48.1655239Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1655421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1655483Z return mod(**inputs) 2025-08-14T21:54:48.1655731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1655792Z outputs = self.model( 2025-08-14T21:54:48.1656034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1656109Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1656349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1656424Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1656626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1656697Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1656946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:48.1657021Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.1657025Z 2025-08-14T21:54:48.1657116Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1657303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1657363Z return mod(**inputs) 2025-08-14T21:54:48.1657611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1657674Z outputs = self.model( 2025-08-14T21:54:48.1657934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1658008Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1658248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1658336Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1658538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1658609Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1658858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-14T21:54:48.1658930Z hidden_states = residual + hidden_states 2025-08-14T21:54:48.1658934Z 2025-08-14T21:54:48.1659025Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1659246Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1659307Z return mod(**inputs) 2025-08-14T21:54:48.1659595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1659660Z outputs = self.model( 2025-08-14T21:54:48.1659903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1659978Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1660220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1660286Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1660493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1660565Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1660815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1660908Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1661149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1661292Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1661296Z 2025-08-14T21:54:48.1661389Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1661575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1661635Z return mod(**inputs) 2025-08-14T21:54:48.1661877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1661947Z outputs = self.model( 2025-08-14T21:54:48.1662188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1662255Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1662506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1662572Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1662779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1662850Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1663089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1663188Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1663427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1663524Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1663527Z 2025-08-14T21:54:48.1663619Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1663799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1663882Z return mod(**inputs) 2025-08-14T21:54:48.1664125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1664188Z outputs = self.model( 2025-08-14T21:54:48.1664435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1664500Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1664745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1664827Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1665031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1665111Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1665363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1665464Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1665704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1665783Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1665786Z 2025-08-14T21:54:48.1665867Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1665941Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1666013Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1666092Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1666185Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1666374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1666435Z return mod(**inputs) 2025-08-14T21:54:48.1666679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1666751Z outputs = self.model( 2025-08-14T21:54:48.1666989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1667056Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1667303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1667368Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1667579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1667651Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1667888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1667984Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1668220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1668316Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1668579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1668699Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1668702Z 2025-08-14T21:54:48.1668800Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1669001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1669064Z return mod(**inputs) 2025-08-14T21:54:48.1669318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1669394Z outputs = self.model( 2025-08-14T21:54:48.1669643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1669710Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1669949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1670023Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1670223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1670295Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1670558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1670648Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1670913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1671004Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1671272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1671379Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1671382Z 2025-08-14T21:54:48.1671474Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1671662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1671723Z return mod(**inputs) 2025-08-14T21:54:48.1671967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1672038Z outputs = self.model( 2025-08-14T21:54:48.1672282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1672357Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1672599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1672666Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1672875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1672948Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1673193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1673291Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1673533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1673616Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1673620Z 2025-08-14T21:54:48.1673715Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1673892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1673958Z return mod(**inputs) 2025-08-14T21:54:48.1674201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1674262Z outputs = self.model( 2025-08-14T21:54:48.1674508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1674593Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1674840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1674905Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1675104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1675205Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1675446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1675551Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1675793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1675927Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1675947Z 2025-08-14T21:54:48.1676048Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1676231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1676306Z return mod(**inputs) 2025-08-14T21:54:48.1676554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1676616Z outputs = self.model( 2025-08-14T21:54:48.1676863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1676929Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1677171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1677245Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1677447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1677524Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1677767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1677867Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1678112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1678185Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1678188Z 2025-08-14T21:54:48.1678281Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1678466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1678526Z return mod(**inputs) 2025-08-14T21:54:48.1678776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1678838Z outputs = self.model( 2025-08-14T21:54:48.1679080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1679156Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1679399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1679472Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1679670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1679740Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1679992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1680090Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1680354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1680441Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1680444Z 2025-08-14T21:54:48.1680519Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1680626Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1680699Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1680771Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1680873Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1681053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1681114Z return mod(**inputs) 2025-08-14T21:54:48.1681363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1681423Z outputs = self.model( 2025-08-14T21:54:48.1681689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1681758Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1682055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1682133Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1682334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1682406Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1682670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1682768Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1683024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1683114Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1683382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1683512Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1683517Z 2025-08-14T21:54:48.1683610Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1683799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1683858Z return mod(**inputs) 2025-08-14T21:54:48.1684107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1684177Z outputs = self.model( 2025-08-14T21:54:48.1684425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1684500Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1684743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1684812Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1685022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1685095Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1685400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1685510Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1685752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1685848Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1686187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1686301Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1686307Z 2025-08-14T21:54:48.1686425Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1686657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1686727Z return mod(**inputs) 2025-08-14T21:54:48.1686982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1687048Z outputs = self.model( 2025-08-14T21:54:48.1687315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1687385Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1687648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1687728Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1687954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1688039Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1688291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1688388Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1688649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1688723Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1688726Z 2025-08-14T21:54:48.1688825Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1689009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1689070Z return mod(**inputs) 2025-08-14T21:54:48.1689324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1689388Z outputs = self.model( 2025-08-14T21:54:48.1689636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1689711Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1689959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1690033Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1690241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1690315Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1690569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1690681Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1690686Z 2025-08-14T21:54:48.1690782Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1690975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1691038Z return mod(**inputs) 2025-08-14T21:54:48.1691293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1691356Z outputs = self.model( 2025-08-14T21:54:48.1691601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1691676Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1691946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1692021Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1692231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1692320Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1692572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1692683Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1692886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.1692959Z return self.act(input) 2025-08-14T21:54:48.1692962Z 2025-08-14T21:54:48.1693059Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1693270Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1693333Z return mod(**inputs) 2025-08-14T21:54:48.1693613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1693685Z outputs = self.model( 2025-08-14T21:54:48.1693935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1694012Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1694261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1694328Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1694543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1694617Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1694869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:48.1694955Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.1694958Z 2025-08-14T21:54:48.1695056Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1695251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1695312Z return mod(**inputs) 2025-08-14T21:54:48.1695562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1695631Z outputs = self.model( 2025-08-14T21:54:48.1695878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1695946Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1696203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1696271Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1696483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1696557Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1696805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1696907Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1697156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1697302Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1697306Z 2025-08-14T21:54:48.1697401Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1697603Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1697674Z return mod(**inputs) 2025-08-14T21:54:48.1697921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1698001Z outputs = self.model( 2025-08-14T21:54:48.1698260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1698328Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1698583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1698649Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1698854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1698934Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1699196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1699297Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1699561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1699639Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1699642Z 2025-08-14T21:54:48.1699744Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1699929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1699990Z return mod(**inputs) 2025-08-14T21:54:48.1700247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1700311Z outputs = self.model( 2025-08-14T21:54:48.1700571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1700640Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1700891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1700966Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1701175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1701255Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1701505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1701597Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1701914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1701995Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1701998Z 2025-08-14T21:54:48.1702072Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1702154Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1702227Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1702305Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1702396Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1702580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1702645Z return mod(**inputs) 2025-08-14T21:54:48.1702887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1702949Z outputs = self.model( 2025-08-14T21:54:48.1703199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1703297Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1703549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1703615Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1703833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1703910Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1704148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1704236Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1704491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1704579Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1704868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1704991Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1704994Z 2025-08-14T21:54:48.1705103Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1705297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1705358Z return mod(**inputs) 2025-08-14T21:54:48.1705608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1705673Z outputs = self.model( 2025-08-14T21:54:48.1705918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1705995Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1706245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1706313Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1706529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1706606Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1706857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1706949Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1707192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1707291Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1707560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1707672Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1707676Z 2025-08-14T21:54:48.1707772Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1707955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1708029Z return mod(**inputs) 2025-08-14T21:54:48.1708274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1708339Z outputs = self.model( 2025-08-14T21:54:48.1708590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1708660Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1708914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1709033Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1709241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1709319Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1709569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1709680Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1709920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1709994Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1709997Z 2025-08-14T21:54:48.1710097Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1710275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1710337Z return mod(**inputs) 2025-08-14T21:54:48.1710601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1710666Z outputs = self.model( 2025-08-14T21:54:48.1710931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1711001Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1711245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1711317Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1711517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1711595Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1711836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 416, in forward 2025-08-14T21:54:48.1711912Z hidden_states = residual + hidden_states 2025-08-14T21:54:48.1711915Z 2025-08-14T21:54:48.1712020Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1712204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1712265Z return mod(**inputs) 2025-08-14T21:54:48.1712514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1712575Z outputs = self.model( 2025-08-14T21:54:48.1712822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1712888Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1713131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1713206Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1713410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1713482Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1713735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1713834Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1714085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1714220Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1714223Z 2025-08-14T21:54:48.1714316Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1714502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1714579Z return mod(**inputs) 2025-08-14T21:54:48.1714828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1714889Z outputs = self.model( 2025-08-14T21:54:48.1715128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1715219Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1715457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1715529Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1715733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1715803Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1716075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1716177Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1716434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1716518Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1716523Z 2025-08-14T21:54:48.1716619Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1716808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1716871Z return mod(**inputs) 2025-08-14T21:54:48.1717112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1717184Z outputs = self.model( 2025-08-14T21:54:48.1717427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1717497Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1717749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1717819Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1718028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1718099Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1718340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1718447Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1718689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1718777Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1718781Z 2025-08-14T21:54:48.1718857Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1718934Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1719013Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1719085Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1719182Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1719375Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1719438Z return mod(**inputs) 2025-08-14T21:54:48.1719689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1719752Z outputs = self.model( 2025-08-14T21:54:48.1719993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1720068Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1720329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1720394Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1720606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1720692Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1720941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1721038Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1721279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1721374Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1721655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1721790Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1721793Z 2025-08-14T21:54:48.1721886Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1722089Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1722160Z return mod(**inputs) 2025-08-14T21:54:48.1722403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1722465Z outputs = self.model( 2025-08-14T21:54:48.1722713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1722780Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1723030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1723100Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1723303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1723383Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1723630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1723736Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1723982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1724072Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1724346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1724445Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1724451Z 2025-08-14T21:54:48.1724546Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1724734Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1724797Z return mod(**inputs) 2025-08-14T21:54:48.1725050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1725114Z outputs = self.model( 2025-08-14T21:54:48.1725429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1725518Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1725781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1725860Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1726089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1726185Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1726449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1726570Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1726852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1726955Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1726959Z 2025-08-14T21:54:48.1727053Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1727242Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1727303Z return mod(**inputs) 2025-08-14T21:54:48.1727560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1727635Z outputs = self.model( 2025-08-14T21:54:48.1727873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1727964Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1728209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1728275Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1728491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1728563Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1728806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1728924Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1728930Z 2025-08-14T21:54:48.1729021Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1729212Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1729273Z return mod(**inputs) 2025-08-14T21:54:48.1729517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1729590Z outputs = self.model( 2025-08-14T21:54:48.1729831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1729897Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1730147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1730212Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1730421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1730493Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1730736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1730852Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1731049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.1731119Z return self.act(input) 2025-08-14T21:54:48.1731122Z 2025-08-14T21:54:48.1731215Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1731396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1731464Z return mod(**inputs) 2025-08-14T21:54:48.1731708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1731787Z outputs = self.model( 2025-08-14T21:54:48.1732035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1732102Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1732363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1732428Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1732627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1732706Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1732948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:48.1733029Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.1733034Z 2025-08-14T21:54:48.1733141Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1733323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1733389Z return mod(**inputs) 2025-08-14T21:54:48.1733645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1733709Z outputs = self.model( 2025-08-14T21:54:48.1733953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1734019Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1734262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1734329Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1734530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1734611Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1734856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1734951Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1735208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1735348Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1735352Z 2025-08-14T21:54:48.1735454Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1735641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1735704Z return mod(**inputs) 2025-08-14T21:54:48.1735967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1736033Z outputs = self.model( 2025-08-14T21:54:48.1736308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1736376Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1736621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1736694Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1736901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1736974Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1737232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1737325Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1737750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1737835Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1737839Z 2025-08-14T21:54:48.1737935Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1738175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1738239Z return mod(**inputs) 2025-08-14T21:54:48.1738498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1738563Z outputs = self.model( 2025-08-14T21:54:48.1738810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1738887Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1739155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1739228Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1739487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1739567Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1739828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1739923Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1740178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1740268Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1740271Z 2025-08-14T21:54:48.1740349Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1740434Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1740513Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1740589Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1740693Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1740889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1740956Z return mod(**inputs) 2025-08-14T21:54:48.1741230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1741295Z outputs = self.model( 2025-08-14T21:54:48.1741553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1741624Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1741877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1741955Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1742163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1742239Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1742499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1742592Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1742848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1742940Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1743212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1743344Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1743369Z 2025-08-14T21:54:48.1743468Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1743662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1743724Z return mod(**inputs) 2025-08-14T21:54:48.1743972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1744060Z outputs = self.model( 2025-08-14T21:54:48.1744310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1744378Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1744639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1744706Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1744946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1745024Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1745270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1745384Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1745634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1745731Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1746001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1746102Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1746106Z 2025-08-14T21:54:48.1746212Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1746402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1746463Z return mod(**inputs) 2025-08-14T21:54:48.1762475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1762711Z outputs = self.model( 2025-08-14T21:54:48.1763026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1763114Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1763377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1763451Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1763680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1763762Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1764027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1764129Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1764379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1764469Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1764475Z 2025-08-14T21:54:48.1764583Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1764784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1764862Z return mod(**inputs) 2025-08-14T21:54:48.1765117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1765195Z outputs = self.model( 2025-08-14T21:54:48.1765616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1765694Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1765961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1766068Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1766296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1766375Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1766639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1766754Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1767004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1767192Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1767206Z 2025-08-14T21:54:48.1767309Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1767535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1767612Z return mod(**inputs) 2025-08-14T21:54:48.1767863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1767930Z outputs = self.model( 2025-08-14T21:54:48.1768189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1768260Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1768515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1768588Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1768794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1768873Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1769125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1769224Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1769471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1769545Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1769548Z 2025-08-14T21:54:48.1769649Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1769831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1769894Z return mod(**inputs) 2025-08-14T21:54:48.1770151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1770217Z outputs = self.model( 2025-08-14T21:54:48.1770466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1770546Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1770793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1770867Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1771073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1771149Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1771405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1771523Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1771778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1771860Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1771881Z 2025-08-14T21:54:48.1771961Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1772044Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1772117Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1772195Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1772291Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1772480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1772549Z return mod(**inputs) 2025-08-14T21:54:48.1772812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1772880Z outputs = self.model( 2025-08-14T21:54:48.1773138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1773223Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1773481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1773548Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1773753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1773832Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1774077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1774176Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1774429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1774524Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1774802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1774931Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1774935Z 2025-08-14T21:54:48.1775031Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1775226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1775288Z return mod(**inputs) 2025-08-14T21:54:48.1775546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1775608Z outputs = self.model( 2025-08-14T21:54:48.1775855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1775926Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1776172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1776245Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1776449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1776521Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1776774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1776871Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1777117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1777251Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1777518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1777622Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1777641Z 2025-08-14T21:54:48.1777734Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1777919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1777989Z return mod(**inputs) 2025-08-14T21:54:48.1778235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1778307Z outputs = self.model( 2025-08-14T21:54:48.1778554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1778641Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1778900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1778985Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1779196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1779279Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1779530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1779641Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1779894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1779977Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1779982Z 2025-08-14T21:54:48.1780090Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1780280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1780344Z return mod(**inputs) 2025-08-14T21:54:48.1780603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1780670Z outputs = self.model( 2025-08-14T21:54:48.1780926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1780996Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1781243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1781321Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1781532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1781617Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1781866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 433, in forward 2025-08-14T21:54:48.1781943Z hidden_states = residual + hidden_states 2025-08-14T21:54:48.1781948Z 2025-08-14T21:54:48.1782052Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1782243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1782309Z return mod(**inputs) 2025-08-14T21:54:48.1782565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1782631Z outputs = self.model( 2025-08-14T21:54:48.1782886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1782973Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1783221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1783298Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1783520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1783599Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1783847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1783962Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1783966Z 2025-08-14T21:54:48.1784070Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1784256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1784336Z return mod(**inputs) 2025-08-14T21:54:48.1784591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1784655Z outputs = self.model( 2025-08-14T21:54:48.1784923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1784996Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1785243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1785319Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1785524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1785597Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1785856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1785964Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1786167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.1786232Z return self.act(input) 2025-08-14T21:54:48.1786236Z 2025-08-14T21:54:48.1786327Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1786515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1786577Z return mod(**inputs) 2025-08-14T21:54:48.1786831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1786895Z outputs = self.model( 2025-08-14T21:54:48.1787147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1787225Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1787465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1787530Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1787740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1787813Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1788058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:48.1788134Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.1788137Z 2025-08-14T21:54:48.1788231Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1788418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1788477Z return mod(**inputs) 2025-08-14T21:54:48.1788746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1788808Z outputs = self.model( 2025-08-14T21:54:48.1789053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1789144Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1789386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1789452Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1789659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1789729Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1789977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1790086Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1790330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1790491Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1790497Z 2025-08-14T21:54:48.1790589Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1790776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1790837Z return mod(**inputs) 2025-08-14T21:54:48.1791075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1791147Z outputs = self.model( 2025-08-14T21:54:48.1791385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1791454Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1791704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1791770Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1791975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1792045Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1792282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1792376Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1792612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1792688Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1792693Z 2025-08-14T21:54:48.1792786Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1792965Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1793032Z return mod(**inputs) 2025-08-14T21:54:48.1793271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1793335Z outputs = self.model( 2025-08-14T21:54:48.1793585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1793652Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1793899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1793966Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1794167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1794261Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1794500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1794590Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1794852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1794932Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1794935Z 2025-08-14T21:54:48.1795016Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1795089Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1795160Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1795237Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1795330Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1795531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1795606Z return mod(**inputs) 2025-08-14T21:54:48.1795877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1795950Z outputs = self.model( 2025-08-14T21:54:48.1796201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1796270Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1796523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1796591Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1796810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1796881Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1797124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1797222Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1797466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1797557Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1797836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1797960Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1797963Z 2025-08-14T21:54:48.1798063Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1798243Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1798303Z return mod(**inputs) 2025-08-14T21:54:48.1798556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1798616Z outputs = self.model( 2025-08-14T21:54:48.1798862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1798929Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1799172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1799244Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1799446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1799518Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1799766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1799876Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1800123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1800212Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1800495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1800600Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1800604Z 2025-08-14T21:54:48.1800696Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1800880Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1800940Z return mod(**inputs) 2025-08-14T21:54:48.1801195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1801269Z outputs = self.model( 2025-08-14T21:54:48.1801511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1801615Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1801864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1801931Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1802138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1802209Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1802443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1802534Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1802777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1802856Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1802860Z 2025-08-14T21:54:48.1802953Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1803134Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1803198Z return mod(**inputs) 2025-08-14T21:54:48.1803435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1803498Z outputs = self.model( 2025-08-14T21:54:48.1803743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1803811Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1804055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1804121Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1804326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1804408Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1804655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1804754Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1805007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1805148Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1805151Z 2025-08-14T21:54:48.1805254Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1805544Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1805611Z return mod(**inputs) 2025-08-14T21:54:48.1805869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1805955Z outputs = self.model( 2025-08-14T21:54:48.1806210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1806280Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1806531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1806609Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1806820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1806896Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1807196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1807300Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1807568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1807647Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1807651Z 2025-08-14T21:54:48.1807747Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1807937Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1807997Z return mod(**inputs) 2025-08-14T21:54:48.1808245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1808306Z outputs = self.model( 2025-08-14T21:54:48.1808554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1808628Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1808876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1808945Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1809157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1809229Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1809481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1809580Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1809825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1809915Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1809919Z 2025-08-14T21:54:48.1809995Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1810077Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1810152Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1810224Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1810325Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1810509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1810571Z return mod(**inputs) 2025-08-14T21:54:48.1810827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1810890Z outputs = self.model( 2025-08-14T21:54:48.1811141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1811226Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1811476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1811551Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1811774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1811848Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1812103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1812201Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1812453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1812545Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1812835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1812966Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1812970Z 2025-08-14T21:54:48.1813080Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1813279Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1813340Z return mod(**inputs) 2025-08-14T21:54:48.1813583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1813652Z outputs = self.model( 2025-08-14T21:54:48.1813898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1813967Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1814226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1814293Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1814512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1814588Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1814846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1814954Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1815213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1815308Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1815594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1815699Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1815702Z 2025-08-14T21:54:48.1815807Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1816001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1816065Z return mod(**inputs) 2025-08-14T21:54:48.1816341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1816407Z outputs = self.model( 2025-08-14T21:54:48.1816666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1816734Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1816986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1817087Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1817294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1817375Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1817622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1817740Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1817994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1818072Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1818076Z 2025-08-14T21:54:48.1818170Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1818362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1818426Z return mod(**inputs) 2025-08-14T21:54:48.1818703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1818770Z outputs = self.model( 2025-08-14T21:54:48.1819035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1819112Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1819359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1819437Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1819648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1819727Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1819976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1820091Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1820095Z 2025-08-14T21:54:48.1820199Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1820385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1820455Z return mod(**inputs) 2025-08-14T21:54:48.1820702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1820766Z outputs = self.model( 2025-08-14T21:54:48.1821019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1821088Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1821336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1821414Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1821623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1821704Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1821951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1822065Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1822273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.1822340Z return self.act(input) 2025-08-14T21:54:48.1822344Z 2025-08-14T21:54:48.1822448Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1822635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1822697Z return mod(**inputs) 2025-08-14T21:54:48.1822977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1823040Z outputs = self.model( 2025-08-14T21:54:48.1823281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1823374Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1823618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1823694Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1823902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1823973Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1824222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:48.1824316Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.1824320Z 2025-08-14T21:54:48.1824422Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1824946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1825016Z return mod(**inputs) 2025-08-14T21:54:48.1825277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1825344Z outputs = self.model( 2025-08-14T21:54:48.1825595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1825686Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1825935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1826013Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1826221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1826294Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1826547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 442, in forward 2025-08-14T21:54:48.1826623Z hidden_states = residual + hidden_states 2025-08-14T21:54:48.1826626Z 2025-08-14T21:54:48.1826729Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1826916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1826979Z return mod(**inputs) 2025-08-14T21:54:48.1827230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1827293Z outputs = self.model( 2025-08-14T21:54:48.1827543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1827620Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1827867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1827945Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1828148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1828219Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1828470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1828564Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1828809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1828976Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1828980Z 2025-08-14T21:54:48.1829075Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1829268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1829349Z return mod(**inputs) 2025-08-14T21:54:48.1829597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1829669Z outputs = self.model( 2025-08-14T21:54:48.1829915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1829989Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1830235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1830338Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1830554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1830627Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1830886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1830990Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1831235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1831319Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1831322Z 2025-08-14T21:54:48.1831416Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1831600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1831674Z return mod(**inputs) 2025-08-14T21:54:48.1831922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1831995Z outputs = self.model( 2025-08-14T21:54:48.1832240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1832310Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1832559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1832626Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1832832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1832913Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1833156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1833259Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1833503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1833585Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1833590Z 2025-08-14T21:54:48.1833677Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1833754Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1833828Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1833904Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1833999Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1834192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1834254Z return mod(**inputs) 2025-08-14T21:54:48.1834503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1834592Z outputs = self.model( 2025-08-14T21:54:48.1834846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1834915Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1835190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1835256Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1835476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1835550Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1835806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1835907Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1836188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1836288Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1836579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1836705Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1836709Z 2025-08-14T21:54:48.1836811Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1837001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1837070Z return mod(**inputs) 2025-08-14T21:54:48.1837320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1837383Z outputs = self.model( 2025-08-14T21:54:48.1838347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1838426Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1838679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1838759Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1838967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1839052Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1839309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1839403Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1839701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1839796Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1840069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1840183Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1840188Z 2025-08-14T21:54:48.1840284Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1840480Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1840541Z return mod(**inputs) 2025-08-14T21:54:48.1840791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1840864Z outputs = self.model( 2025-08-14T21:54:48.1841111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1841249Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1841497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1841566Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1841782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1841892Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1842142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1842244Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1842497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1842584Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1842589Z 2025-08-14T21:54:48.1842714Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1842910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1842984Z return mod(**inputs) 2025-08-14T21:54:48.1843263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1843338Z outputs = self.model( 2025-08-14T21:54:48.1843595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1843667Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1843924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1843993Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1844204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1844289Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1844543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1844654Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1844907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1845048Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1845051Z 2025-08-14T21:54:48.1845156Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1845401Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1845478Z return mod(**inputs) 2025-08-14T21:54:48.1845738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1845805Z outputs = self.model( 2025-08-14T21:54:48.1846066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1846139Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1846393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1846473Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1846687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1846770Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1847026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1847128Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1847430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1847506Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1847510Z 2025-08-14T21:54:48.1847617Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1847824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1847888Z return mod(**inputs) 2025-08-14T21:54:48.1848145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1848211Z outputs = self.model( 2025-08-14T21:54:48.1848463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1848540Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1848811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1848892Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1849121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1849198Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1849456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1849556Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1849809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1849889Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1849892Z 2025-08-14T21:54:48.1849967Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1850049Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1850125Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1850196Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1850302Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1850491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1850555Z return mod(**inputs) 2025-08-14T21:54:48.1850811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1850875Z outputs = self.model( 2025-08-14T21:54:48.1851131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1851199Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1851447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1851527Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1851736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1851815Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1852063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1852165Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1852416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1852507Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1852779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1852911Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1852933Z 2025-08-14T21:54:48.1853032Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1853227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1853289Z return mod(**inputs) 2025-08-14T21:54:48.1853535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1853626Z outputs = self.model( 2025-08-14T21:54:48.1853874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1853950Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1854196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1854264Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1854494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1854570Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1854812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1854933Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1855181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1855278Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1855547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1855651Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1855654Z 2025-08-14T21:54:48.1855758Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1855953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1856024Z return mod(**inputs) 2025-08-14T21:54:48.1856288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1856353Z outputs = self.model( 2025-08-14T21:54:48.1856602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1856672Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1856918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1856992Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1857198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1857278Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1857528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1857628Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1857880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1857957Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1857961Z 2025-08-14T21:54:48.1858063Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1858250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1858311Z return mod(**inputs) 2025-08-14T21:54:48.1858563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1858626Z outputs = self.model( 2025-08-14T21:54:48.1858887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1858963Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1859208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1859300Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1859508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1859582Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1859836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1859947Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1859951Z 2025-08-14T21:54:48.1860053Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1860256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1860330Z return mod(**inputs) 2025-08-14T21:54:48.1860597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1860662Z outputs = self.model( 2025-08-14T21:54:48.1860902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1860979Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1861221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1861318Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1861559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1861634Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1861887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1861995Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1862200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.1862266Z return self.act(input) 2025-08-14T21:54:48.1862269Z 2025-08-14T21:54:48.1862362Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1862551Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1862613Z return mod(**inputs) 2025-08-14T21:54:48.1862862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1862934Z outputs = self.model( 2025-08-14T21:54:48.1863185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1863265Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1863514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1863586Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1863804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1863878Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1864126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:48.1864212Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.1864216Z 2025-08-14T21:54:48.1864310Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1864504Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1864596Z return mod(**inputs) 2025-08-14T21:54:48.1864847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1864920Z outputs = self.model( 2025-08-14T21:54:48.1865184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1865258Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1865504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1865572Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1865784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1865858Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1866128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1866234Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1866491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1866641Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1866645Z 2025-08-14T21:54:48.1866738Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1866919Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1866989Z return mod(**inputs) 2025-08-14T21:54:48.1867232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1867306Z outputs = self.model( 2025-08-14T21:54:48.1867556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1867623Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1867887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1867954Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1868156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1868235Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1868473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1868568Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1868809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1868884Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1868887Z 2025-08-14T21:54:48.1868987Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1869171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1869240Z return mod(**inputs) 2025-08-14T21:54:48.1869481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1869544Z outputs = self.model( 2025-08-14T21:54:48.1869790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1869856Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1870092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1870183Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1870386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1870463Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1870705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1870817Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1871069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1871145Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1871149Z 2025-08-14T21:54:48.1871229Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1871303Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1871374Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1871452Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1871563Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1871748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1871818Z return mod(**inputs) 2025-08-14T21:54:48.1872076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1872142Z outputs = self.model( 2025-08-14T21:54:48.1872388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1872455Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1872701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1872769Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1872972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1873054Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1873294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1873393Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1873633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1873722Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1873997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1874117Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1874121Z 2025-08-14T21:54:48.1874214Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1874406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1874468Z return mod(**inputs) 2025-08-14T21:54:48.1874724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1874792Z outputs = self.model( 2025-08-14T21:54:48.1875038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1875116Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1875361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1875438Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1875643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1875716Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1875991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1876083Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1876343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1876455Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1876717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1876823Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1876826Z 2025-08-14T21:54:48.1876922Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1877109Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1877180Z return mod(**inputs) 2025-08-14T21:54:48.1877454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1877528Z outputs = self.model( 2025-08-14T21:54:48.1877792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1877865Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1878116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1878183Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1878391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1878470Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1878775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 407, in forward 2025-08-14T21:54:48.1878877Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:54:48.1879123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1879201Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1879205Z 2025-08-14T21:54:48.1879307Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1879494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1879561Z return mod(**inputs) 2025-08-14T21:54:48.1879804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1879868Z outputs = self.model( 2025-08-14T21:54:48.1880120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1880191Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1880436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1880511Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1880718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1880802Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1881050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 416, in forward 2025-08-14T21:54:48.1881125Z hidden_states = residual + hidden_states 2025-08-14T21:54:48.1881128Z 2025-08-14T21:54:48.1881233Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1881416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1881485Z return mod(**inputs) 2025-08-14T21:54:48.1881751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1881815Z outputs = self.model( 2025-08-14T21:54:48.1882067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1882155Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1882402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1882478Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1882683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1882761Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1883008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1883129Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1883384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 215, in forward 2025-08-14T21:54:48.1883544Z query_states = self.q_proj(hidden_states).view(*q_input_shape).transpose(1, 2) 2025-08-14T21:54:48.1883549Z 2025-08-14T21:54:48.1883661Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1883845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1883907Z return mod(**inputs) 2025-08-14T21:54:48.1884160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1884225Z outputs = self.model( 2025-08-14T21:54:48.1884470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1884550Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1884799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1884874Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1885077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1885152Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1885493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1885611Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1885894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 234, in forward 2025-08-14T21:54:48.1885992Z key_states = self.k_proj(current_states) 2025-08-14T21:54:48.1885998Z 2025-08-14T21:54:48.1886100Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1886306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1886375Z return mod(**inputs) 2025-08-14T21:54:48.1886645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1886729Z outputs = self.model( 2025-08-14T21:54:48.1886979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1887058Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1887309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1887377Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1887596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1887693Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1887942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1888052Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1888312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 235, in forward 2025-08-14T21:54:48.1888400Z value_states = self.v_proj(current_states) 2025-08-14T21:54:48.1888404Z 2025-08-14T21:54:48.1888481Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1888558Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1888638Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1888711Z cudagraph partition due to non gpu ops 2025-08-14T21:54:48.1888809Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1889019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1889084Z return mod(**inputs) 2025-08-14T21:54:48.1889346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1889427Z outputs = self.model( 2025-08-14T21:54:48.1889679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1889757Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1890004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1890081Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1890290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1890365Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1890623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1890723Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1890971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1891069Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1891339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 81, in sdpa_attention_forward 2025-08-14T21:54:48.1891470Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:54:48.1891473Z 2025-08-14T21:54:48.1891567Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1891752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1891823Z return mod(**inputs) 2025-08-14T21:54:48.1892075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1892146Z outputs = self.model( 2025-08-14T21:54:48.1892393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1892462Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1892717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1892785Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1892993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1893076Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1893323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1893448Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1893695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 253, in forward 2025-08-14T21:54:48.1893786Z attn_output, attn_weights = attention_interface( 2025-08-14T21:54:48.1894082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/integrations/sdpa_attention.py", line 91, in sdpa_attention_forward 2025-08-14T21:54:48.1894183Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:54:48.1894187Z 2025-08-14T21:54:48.1894291Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1894478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1894541Z return mod(**inputs) 2025-08-14T21:54:48.1894815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1894883Z outputs = self.model( 2025-08-14T21:54:48.1895131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1895223Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1895471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1895547Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1895757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1895831Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1896088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 424, in forward 2025-08-14T21:54:48.1896191Z hidden_states, cross_attn_weights = self.encoder_attn( 2025-08-14T21:54:48.1896444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 267, in forward 2025-08-14T21:54:48.1896530Z attn_output = self.out_proj(attn_output) 2025-08-14T21:54:48.1896534Z 2025-08-14T21:54:48.1896633Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1896835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1896895Z return mod(**inputs) 2025-08-14T21:54:48.1897138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1897209Z outputs = self.model( 2025-08-14T21:54:48.1897451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1897529Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1897774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1897842Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1898053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1898126Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1898369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1898490Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1898493Z 2025-08-14T21:54:48.1898588Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1898781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1898842Z return mod(**inputs) 2025-08-14T21:54:48.1899088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1899185Z outputs = self.model( 2025-08-14T21:54:48.1899431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1899506Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1899795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1899862Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1900073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1900145Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1900398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 438, in forward 2025-08-14T21:54:48.1900512Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:54:48.1900723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:54:48.1900796Z return self.act(input) 2025-08-14T21:54:48.1900800Z 2025-08-14T21:54:48.1900908Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1901093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1901163Z return mod(**inputs) 2025-08-14T21:54:48.1901405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1471, in forward 2025-08-14T21:54:48.1901477Z outputs = self.model( 2025-08-14T21:54:48.1901719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1297, in forward 2025-08-14T21:54:48.1901786Z decoder_outputs = self.decoder( 2025-08-14T21:54:48.1902038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1115, in forward 2025-08-14T21:54:48.1902108Z layer_outputs = decoder_layer( 2025-08-14T21:54:48.1902312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:54:48.1902397Z return super().__call__(*args, **kwargs) 2025-08-14T21:54:48.1902640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 440, in forward 2025-08-14T21:54:48.1902724Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:54:48.1902728Z 2025-08-14T21:54:48.1902821Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1903002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1903070Z return mod(**inputs) 2025-08-14T21:54:48.1903311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1489, in forward 2025-08-14T21:54:48.1903426Z lm_logits = self.lm_head(outputs[0]) + self.final_logits_bias 2025-08-14T21:54:48.1903437Z 2025-08-14T21:54:48.1903530Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:54:48.1903710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:54:48.1903781Z return mod(**inputs) 2025-08-14T21:54:48.1904026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/pegasus/modeling_pegasus.py", line 1494, in forward 2025-08-14T21:54:48.1904180Z masked_lm_loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:54:48.1904184Z 2025-08-14T21:54:59.7499151Z Compilation time (from dynamo_timed): 26.772311103 2025-08-14T21:54:59.7509679Z pass 2025-08-14T21:54:59.7510127Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:54:59.7511103Z TIMING: _recursive_pre_grad_passes:0.014 _recursive_joint_graph_passes:1.11653 _recursive_post_grad_passes:0.16498 async_compile.wait:0.76869 code_gen:11.19626 inductor_compile:14.07039 backend_compile:21.12194 gc:0.00147 entire_frame_compile:26.77231 total_wall_time:26.77231 2025-08-14T21:54:59.7512501Z STATS: call_* op count: 965 | FakeTensorMode.__torch_dispatch__:33299 | FakeTensor.__torch_dispatch__:11840 | ProxyTorchDispatchMode.__torch_dispatch__:12299 2025-08-14T21:54:59.7513144Z Dynamo produced 1 graphs covering 965 ops with 0 graph breaks (0 unique) 2025-08-14T21:55:05.6273205Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:55:05.6274099Z from pkg_resources import resource_filename 2025-08-14T21:55:06.2780433Z 2025-08-14T21:55:06.2912476Z loading model: 0it [00:00, ?it/s]If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2025-08-14T21:55:06.2917165Z WARNING:transformers.models.roberta.modeling_roberta:If you want to use `RobertaLMHeadModel` as a standalone, add `is_decoder=True.` 2025-08-14T21:55:07.7161702Z We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-08-14T21:55:07.7162637Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-08-14T21:55:07.7163553Z WARNING:transformers.modeling_utils:We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-08-14T21:55:07.7164511Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-08-14T21:55:07.8925734Z 2025-08-14T21:55:07.8926526Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:55:07.8940415Z cpu eval RobertaForCausalLM 2025-08-14T21:55:08.4940841Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:55:08.7747667Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:55:09.0556889Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:55:16.4908275Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.4912698Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.4913054Z return mod(**inputs) 2025-08-14T21:55:16.4913532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.4913940Z outputs = self.roberta( 2025-08-14T21:55:16.4914324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-08-14T21:55:16.4914721Z embedding_output = self.embeddings( 2025-08-14T21:55:16.4915100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-08-14T21:55:16.4915607Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-14T21:55:16.4916178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1576, in create_position_ids_from_input_ids 2025-08-14T21:55:16.4916633Z mask = input_ids.ne(padding_idx).int() 2025-08-14T21:55:16.4916775Z 2025-08-14T21:55:16.4916856Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.4917410Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.4917607Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.4917798Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.4917993Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.4918191Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.4919363Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.4919567Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.4919769Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.4919963Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.4920167Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.4920367Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.4920607Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.4920958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.4921287Z return mod(**inputs) 2025-08-14T21:55:16.4921725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.4922164Z outputs = self.roberta( 2025-08-14T21:55:16.4922610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-08-14T21:55:16.4923016Z embedding_output = self.embeddings( 2025-08-14T21:55:16.4923424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-08-14T21:55:16.4923953Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-14T21:55:16.4924542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1577, in create_position_ids_from_input_ids 2025-08-14T21:55:16.4925117Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-14T21:55:16.4925566Z 2025-08-14T21:55:16.4925685Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.4926082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.4926431Z return mod(**inputs) 2025-08-14T21:55:16.4926835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.4927251Z outputs = self.roberta( 2025-08-14T21:55:16.4927681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-08-14T21:55:16.4928068Z embedding_output = self.embeddings( 2025-08-14T21:55:16.4928455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-08-14T21:55:16.4928975Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-14T21:55:16.4929569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1577, in create_position_ids_from_input_ids 2025-08-14T21:55:16.4930130Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-14T21:55:16.4930375Z 2025-08-14T21:55:16.4930480Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.4930841Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.4931160Z return mod(**inputs) 2025-08-14T21:55:16.4931528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.4931931Z outputs = self.roberta( 2025-08-14T21:55:16.4932315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.4932726Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.4933106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.4933726Z layer_outputs = layer_module( 2025-08-14T21:55:16.4934071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.4934445Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.4934840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.4935317Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.4935702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.4936061Z return func(*args, **kwargs) 2025-08-14T21:55:16.4936455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.4936842Z self_outputs = self.self( 2025-08-14T21:55:16.4937184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.4937567Z return func(*args, **kwargs) 2025-08-14T21:55:16.4938239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:16.4938789Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:16.4939072Z 2025-08-14T21:55:16.4939184Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.4939554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.4939877Z return mod(**inputs) 2025-08-14T21:55:16.4940255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.4940647Z outputs = self.roberta( 2025-08-14T21:55:16.4941027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.4941433Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.4941847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.4942267Z layer_outputs = layer_module( 2025-08-14T21:55:16.4942633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.4943015Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.4943433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.4943867Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.4944278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.4944672Z return func(*args, **kwargs) 2025-08-14T21:55:16.4945085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.4945507Z self_outputs = self.self( 2025-08-14T21:55:16.4945888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.4946274Z return func(*args, **kwargs) 2025-08-14T21:55:16.4946682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:16.4947096Z self.key(current_states) 2025-08-14T21:55:16.4947217Z 2025-08-14T21:55:16.4947336Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.4947717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.4948109Z return mod(**inputs) 2025-08-14T21:55:16.4948467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.4948844Z outputs = self.roberta( 2025-08-14T21:55:16.4949248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.4949621Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.4949990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.4950355Z layer_outputs = layer_module( 2025-08-14T21:55:16.4950680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.4951023Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.4951420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.4951805Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.4952185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.4952548Z return func(*args, **kwargs) 2025-08-14T21:55:16.4952909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.4953289Z self_outputs = self.self( 2025-08-14T21:55:16.4953637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.4953987Z return func(*args, **kwargs) 2025-08-14T21:55:16.4954364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:16.4954742Z self.value(current_states) 2025-08-14T21:55:16.4954854Z 2025-08-14T21:55:16.4954939Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.4955162Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.4955506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.4955815Z return mod(**inputs) 2025-08-14T21:55:16.4956160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.4956535Z outputs = self.roberta( 2025-08-14T21:55:16.4956891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.4957266Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.4957625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.4958001Z layer_outputs = layer_module( 2025-08-14T21:55:16.4958334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.4958683Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.4959064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.4959458Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.4959823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.4960173Z return func(*args, **kwargs) 2025-08-14T21:55:16.4960546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.4960926Z self_outputs = self.self( 2025-08-14T21:55:16.4961281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.4961665Z return func(*args, **kwargs) 2025-08-14T21:55:16.4962047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:16.4962501Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:16.4962703Z 2025-08-14T21:55:16.4962812Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.4963169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.4963481Z return mod(**inputs) 2025-08-14T21:55:16.4963839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.4964211Z outputs = self.roberta( 2025-08-14T21:55:16.4964572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.4964987Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.4965446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.4965851Z layer_outputs = layer_module( 2025-08-14T21:55:16.4966224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.4966620Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.4967019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.4967419Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.4967799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.4968167Z return func(*args, **kwargs) 2025-08-14T21:55:16.4968540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:16.4968988Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:16.4969429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:16.4969826Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.4969962Z 2025-08-14T21:55:16.4970065Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.4970418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.4970737Z return mod(**inputs) 2025-08-14T21:55:16.4971093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.4971475Z outputs = self.roberta( 2025-08-14T21:55:16.4971845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.4972231Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.4972604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.4972991Z layer_outputs = layer_module( 2025-08-14T21:55:16.4973330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.4973673Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.4974067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.4974464Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.4974861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.4975257Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.4975673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.4976137Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.4976565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:16.4976969Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.4977107Z 2025-08-14T21:55:16.4977208Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.4977558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.4977864Z return mod(**inputs) 2025-08-14T21:55:16.4978218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.4978597Z outputs = self.roberta( 2025-08-14T21:55:16.4978980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.4979367Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.4979764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.4980159Z layer_outputs = layer_module( 2025-08-14T21:55:16.4980485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.4980841Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.4981217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.4981602Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.4981974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.4982349Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.4982749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.4983196Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.4983607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:16.4984018Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:16.4984378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:16.4984698Z return self.act(input) 2025-08-14T21:55:16.4984811Z 2025-08-14T21:55:16.4984910Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.4985269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.4985581Z return mod(**inputs) 2025-08-14T21:55:16.4985926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.4986299Z outputs = self.roberta( 2025-08-14T21:55:16.4986654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.4987032Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.4987391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.4987761Z layer_outputs = layer_module( 2025-08-14T21:55:16.4988085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.4988419Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.4988797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.4989208Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.4989593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.4989994Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.4990412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:16.4990891Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:16.4991342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:16.4991725Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.4991863Z 2025-08-14T21:55:16.4991961Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.4992324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.4992633Z return mod(**inputs) 2025-08-14T21:55:16.4993050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.4993433Z outputs = self.roberta( 2025-08-14T21:55:16.4993795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.4994176Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.4994564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.4994954Z layer_outputs = layer_module( 2025-08-14T21:55:16.4995292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.4995652Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.4996054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.4996463Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.4996831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.4997200Z return func(*args, **kwargs) 2025-08-14T21:55:16.4997577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.4997989Z self_outputs = self.self( 2025-08-14T21:55:16.4998349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.4998720Z return func(*args, **kwargs) 2025-08-14T21:55:16.4999100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:16.4999633Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:16.4999906Z 2025-08-14T21:55:16.5000011Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5000377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5000703Z return mod(**inputs) 2025-08-14T21:55:16.5001068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5001464Z outputs = self.roberta( 2025-08-14T21:55:16.5001841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5002238Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5002621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5003038Z layer_outputs = layer_module( 2025-08-14T21:55:16.5003387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5003741Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5004159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5004563Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5004943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5005398Z return func(*args, **kwargs) 2025-08-14T21:55:16.5005813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5006230Z self_outputs = self.self( 2025-08-14T21:55:16.5006640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5007018Z return func(*args, **kwargs) 2025-08-14T21:55:16.5007416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:16.5007815Z self.key(current_states) 2025-08-14T21:55:16.5007929Z 2025-08-14T21:55:16.5008035Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5008398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5008720Z return mod(**inputs) 2025-08-14T21:55:16.5009079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5009466Z outputs = self.roberta( 2025-08-14T21:55:16.5009838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5010240Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5010602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5010968Z layer_outputs = layer_module( 2025-08-14T21:55:16.5011292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5011627Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5011995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5012375Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5012732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5013076Z return func(*args, **kwargs) 2025-08-14T21:55:16.5013435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5013804Z self_outputs = self.self( 2025-08-14T21:55:16.5014140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5014484Z return func(*args, **kwargs) 2025-08-14T21:55:16.5014842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:16.5015212Z self.value(current_states) 2025-08-14T21:55:16.5015323Z 2025-08-14T21:55:16.5015401Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.5015626Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5015963Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5016270Z return mod(**inputs) 2025-08-14T21:55:16.5016615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5017007Z outputs = self.roberta( 2025-08-14T21:55:16.5017366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5017766Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5018133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5018504Z layer_outputs = layer_module( 2025-08-14T21:55:16.5018837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5019178Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5019564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5019960Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5020334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5020683Z return func(*args, **kwargs) 2025-08-14T21:55:16.5021056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5021430Z self_outputs = self.self( 2025-08-14T21:55:16.5021765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5022120Z return func(*args, **kwargs) 2025-08-14T21:55:16.5022480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:16.5022909Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:16.5023082Z 2025-08-14T21:55:16.5023183Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5023523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5023836Z return mod(**inputs) 2025-08-14T21:55:16.5024175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5024539Z outputs = self.roberta( 2025-08-14T21:55:16.5024887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5025255Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5025605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5025968Z layer_outputs = layer_module( 2025-08-14T21:55:16.5026285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5026620Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5026982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5027359Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5027711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5028056Z return func(*args, **kwargs) 2025-08-14T21:55:16.5028426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:16.5028863Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:16.5029296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:16.5029683Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5029844Z 2025-08-14T21:55:16.5029948Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5030307Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5030619Z return mod(**inputs) 2025-08-14T21:55:16.5030971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5031359Z outputs = self.roberta( 2025-08-14T21:55:16.5031710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5032088Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5032466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5032844Z layer_outputs = layer_module( 2025-08-14T21:55:16.5033178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5033538Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5033927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5034344Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5034731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5035115Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5035516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.5035963Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.5036374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:16.5036766Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5036904Z 2025-08-14T21:55:16.5037001Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5037342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5037772Z return mod(**inputs) 2025-08-14T21:55:16.5038149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5038533Z outputs = self.roberta( 2025-08-14T21:55:16.5038891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5039285Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5039672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5040068Z layer_outputs = layer_module( 2025-08-14T21:55:16.5040436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5040802Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5041208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5041618Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5042011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5042406Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5042845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.5043314Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.5043756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:16.5044240Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:16.5044623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:16.5044957Z return self.act(input) 2025-08-14T21:55:16.5045120Z 2025-08-14T21:55:16.5045273Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5045673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5046009Z return mod(**inputs) 2025-08-14T21:55:16.5046407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5046800Z outputs = self.roberta( 2025-08-14T21:55:16.5047173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5047568Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5047991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5048394Z layer_outputs = layer_module( 2025-08-14T21:55:16.5048770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5049127Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5049528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5049934Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5050329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5050723Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5051149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:16.5051648Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:16.5052069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:16.5052461Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5052591Z 2025-08-14T21:55:16.5052701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5053045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5053349Z return mod(**inputs) 2025-08-14T21:55:16.5053701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5054075Z outputs = self.roberta( 2025-08-14T21:55:16.5054428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5054815Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5055186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5055561Z layer_outputs = layer_module( 2025-08-14T21:55:16.5055885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5056226Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5056606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5056981Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5057346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5057696Z return func(*args, **kwargs) 2025-08-14T21:55:16.5058079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5058442Z self_outputs = self.self( 2025-08-14T21:55:16.5058784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5059155Z return func(*args, **kwargs) 2025-08-14T21:55:16.5059514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:16.5060003Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:16.5060257Z 2025-08-14T21:55:16.5060354Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5060693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5060992Z return mod(**inputs) 2025-08-14T21:55:16.5061364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5061739Z outputs = self.roberta( 2025-08-14T21:55:16.5062109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5062479Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5062853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5063233Z layer_outputs = layer_module( 2025-08-14T21:55:16.5063556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5063903Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5064283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5064674Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5065037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5065388Z return func(*args, **kwargs) 2025-08-14T21:55:16.5065742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5066108Z self_outputs = self.self( 2025-08-14T21:55:16.5066438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5066783Z return func(*args, **kwargs) 2025-08-14T21:55:16.5067140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:16.5067502Z self.key(current_states) 2025-08-14T21:55:16.5067616Z 2025-08-14T21:55:16.5067714Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5068051Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5068355Z return mod(**inputs) 2025-08-14T21:55:16.5068704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5069081Z outputs = self.roberta( 2025-08-14T21:55:16.5069430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5069796Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5070157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5070527Z layer_outputs = layer_module( 2025-08-14T21:55:16.5070850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5071217Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5071588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5071974Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5072337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5072767Z return func(*args, **kwargs) 2025-08-14T21:55:16.5073125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5073492Z self_outputs = self.self( 2025-08-14T21:55:16.5073829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5074170Z return func(*args, **kwargs) 2025-08-14T21:55:16.5074539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:16.5074908Z self.value(current_states) 2025-08-14T21:55:16.5075018Z 2025-08-14T21:55:16.5075094Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.5075319Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5075670Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5075968Z return mod(**inputs) 2025-08-14T21:55:16.5076314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5076677Z outputs = self.roberta( 2025-08-14T21:55:16.5077021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5077379Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5077741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5078104Z layer_outputs = layer_module( 2025-08-14T21:55:16.5078417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5078750Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5079118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5079492Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5079833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5080174Z return func(*args, **kwargs) 2025-08-14T21:55:16.5080524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5080887Z self_outputs = self.self( 2025-08-14T21:55:16.5081211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5081550Z return func(*args, **kwargs) 2025-08-14T21:55:16.5081900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:16.5082314Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:16.5082490Z 2025-08-14T21:55:16.5082584Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5082917Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5083215Z return mod(**inputs) 2025-08-14T21:55:16.5083550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5083920Z outputs = self.roberta( 2025-08-14T21:55:16.5084278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5084676Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5085056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5085536Z layer_outputs = layer_module( 2025-08-14T21:55:16.5085901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5086327Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5086763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5087183Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5087553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5087909Z return func(*args, **kwargs) 2025-08-14T21:55:16.5088310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:16.5088756Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:16.5089199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:16.5089605Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5089745Z 2025-08-14T21:55:16.5089846Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5090192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5090497Z return mod(**inputs) 2025-08-14T21:55:16.5090858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5091234Z outputs = self.roberta( 2025-08-14T21:55:16.5091598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5091992Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5092373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5092763Z layer_outputs = layer_module( 2025-08-14T21:55:16.5093096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5093452Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5093844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5094248Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5094639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5095032Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5095451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.5095914Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.5096347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:16.5096743Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5096875Z 2025-08-14T21:55:16.5096984Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5097325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5097703Z return mod(**inputs) 2025-08-14T21:55:16.5098055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5098450Z outputs = self.roberta( 2025-08-14T21:55:16.5098808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5099193Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5099581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5099960Z layer_outputs = layer_module( 2025-08-14T21:55:16.5100282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5100612Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5100980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5101344Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5101731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5102100Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5102504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.5102944Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.5103357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:16.5103765Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:16.5104118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:16.5104440Z return self.act(input) 2025-08-14T21:55:16.5104552Z 2025-08-14T21:55:16.5104659Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5104999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5105294Z return mod(**inputs) 2025-08-14T21:55:16.5105638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5106005Z outputs = self.roberta( 2025-08-14T21:55:16.5106347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5106715Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5107074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5107436Z layer_outputs = layer_module( 2025-08-14T21:55:16.5107744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5108078Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5108447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5108820Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5109182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5109543Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5109939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:16.5110387Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:16.5110816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:16.5111203Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5111330Z 2025-08-14T21:55:16.5111462Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5111801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5112104Z return mod(**inputs) 2025-08-14T21:55:16.5112449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5112826Z outputs = self.roberta( 2025-08-14T21:55:16.5113165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5113535Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5113892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5114246Z layer_outputs = layer_module( 2025-08-14T21:55:16.5114560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5114907Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5115277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5115673Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5116026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5116371Z return func(*args, **kwargs) 2025-08-14T21:55:16.5116712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5117071Z self_outputs = self.self( 2025-08-14T21:55:16.5117414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5117766Z return func(*args, **kwargs) 2025-08-14T21:55:16.5118122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:16.5118631Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:16.5118885Z 2025-08-14T21:55:16.5119000Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5119341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5119643Z return mod(**inputs) 2025-08-14T21:55:16.5119995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5120369Z outputs = self.roberta( 2025-08-14T21:55:16.5120717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5121092Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5121463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5121842Z layer_outputs = layer_module( 2025-08-14T21:55:16.5122169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5122512Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5122892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5123266Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5123629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5123982Z return func(*args, **kwargs) 2025-08-14T21:55:16.5124343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5124735Z self_outputs = self.self( 2025-08-14T21:55:16.5125082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5125561Z return func(*args, **kwargs) 2025-08-14T21:55:16.5125977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:16.5126424Z self.key(current_states) 2025-08-14T21:55:16.5126556Z 2025-08-14T21:55:16.5126667Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5127053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5127399Z return mod(**inputs) 2025-08-14T21:55:16.5127755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5128137Z outputs = self.roberta( 2025-08-14T21:55:16.5128509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5128878Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5129316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5129697Z layer_outputs = layer_module( 2025-08-14T21:55:16.5130020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5130367Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5130745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5131128Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5131488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5131846Z return func(*args, **kwargs) 2025-08-14T21:55:16.5132215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5132590Z self_outputs = self.self( 2025-08-14T21:55:16.5132927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5133281Z return func(*args, **kwargs) 2025-08-14T21:55:16.5133643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:16.5134015Z self.value(current_states) 2025-08-14T21:55:16.5134133Z 2025-08-14T21:55:16.5134211Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.5134439Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5134777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5135078Z return mod(**inputs) 2025-08-14T21:55:16.5135437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5135810Z outputs = self.roberta( 2025-08-14T21:55:16.5136162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5136541Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5136914Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5137287Z layer_outputs = layer_module( 2025-08-14T21:55:16.5137800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5138149Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5138543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5139005Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5139377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5139737Z return func(*args, **kwargs) 2025-08-14T21:55:16.5140120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5140515Z self_outputs = self.self( 2025-08-14T21:55:16.5140864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5141210Z return func(*args, **kwargs) 2025-08-14T21:55:16.5141569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:16.5141989Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:16.5142174Z 2025-08-14T21:55:16.5142299Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5142646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5142952Z return mod(**inputs) 2025-08-14T21:55:16.5143335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5143714Z outputs = self.roberta( 2025-08-14T21:55:16.5144070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5144437Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5144805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5145176Z layer_outputs = layer_module( 2025-08-14T21:55:16.5145494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5145836Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5146216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5146601Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5146957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5147321Z return func(*args, **kwargs) 2025-08-14T21:55:16.5147690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:16.5148118Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:16.5148535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:16.5148923Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5149057Z 2025-08-14T21:55:16.5149162Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5149500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5149821Z return mod(**inputs) 2025-08-14T21:55:16.5150180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5150558Z outputs = self.roberta( 2025-08-14T21:55:16.5150913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5151309Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5151671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5152028Z layer_outputs = layer_module( 2025-08-14T21:55:16.5152351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5152712Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5153090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5153487Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5153866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5154239Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5154646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.5155091Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.5155498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:16.5155901Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5156033Z 2025-08-14T21:55:16.5156140Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5156490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5156801Z return mod(**inputs) 2025-08-14T21:55:16.5157149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5157516Z outputs = self.roberta( 2025-08-14T21:55:16.5157876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5158253Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5158615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5158987Z layer_outputs = layer_module( 2025-08-14T21:55:16.5159313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5159654Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5160023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5160407Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5160781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5161149Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5161542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.5161984Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.5162398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:16.5162809Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:16.5163160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:16.5163482Z return self.act(input) 2025-08-14T21:55:16.5163586Z 2025-08-14T21:55:16.5163693Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5164022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5164329Z return mod(**inputs) 2025-08-14T21:55:16.5164678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5165048Z outputs = self.roberta( 2025-08-14T21:55:16.5165465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5165896Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5166304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5166693Z layer_outputs = layer_module( 2025-08-14T21:55:16.5167045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5167394Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5167793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5168170Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5168548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5168923Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5169355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:16.5169814Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:16.5170261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:16.5170650Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5170780Z 2025-08-14T21:55:16.5170880Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5171225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5171535Z return mod(**inputs) 2025-08-14T21:55:16.5171890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5172256Z outputs = self.roberta( 2025-08-14T21:55:16.5172615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5173002Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5173358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5173728Z layer_outputs = layer_module( 2025-08-14T21:55:16.5174050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5174386Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5174751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5175127Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5175492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5175850Z return func(*args, **kwargs) 2025-08-14T21:55:16.5176208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5176581Z self_outputs = self.self( 2025-08-14T21:55:16.5176977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5177318Z return func(*args, **kwargs) 2025-08-14T21:55:16.5177675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:16.5178174Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:16.5178422Z 2025-08-14T21:55:16.5178528Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5178865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5179205Z return mod(**inputs) 2025-08-14T21:55:16.5179560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5179943Z outputs = self.roberta( 2025-08-14T21:55:16.5180288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5180677Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5181039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5181398Z layer_outputs = layer_module( 2025-08-14T21:55:16.5181716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5182053Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5182437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5182808Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5183161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5183522Z return func(*args, **kwargs) 2025-08-14T21:55:16.5183868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5184237Z self_outputs = self.self( 2025-08-14T21:55:16.5184568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5184910Z return func(*args, **kwargs) 2025-08-14T21:55:16.5185253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:16.5185616Z self.key(current_states) 2025-08-14T21:55:16.5185721Z 2025-08-14T21:55:16.5185826Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5186165Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5186467Z return mod(**inputs) 2025-08-14T21:55:16.5186816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5187191Z outputs = self.roberta( 2025-08-14T21:55:16.5187545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5187937Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5188329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5188720Z layer_outputs = layer_module( 2025-08-14T21:55:16.5189077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5189462Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5189882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5190315Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5190728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5191102Z return func(*args, **kwargs) 2025-08-14T21:55:16.5191483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5191870Z self_outputs = self.self( 2025-08-14T21:55:16.5192246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5192646Z return func(*args, **kwargs) 2025-08-14T21:55:16.5193071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:16.5193488Z self.value(current_states) 2025-08-14T21:55:16.5193620Z 2025-08-14T21:55:16.5193708Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.5193968Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5194340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5194666Z return mod(**inputs) 2025-08-14T21:55:16.5195038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5195425Z outputs = self.roberta( 2025-08-14T21:55:16.5195794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5196190Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5196596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5196983Z layer_outputs = layer_module( 2025-08-14T21:55:16.5197346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5197713Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5198108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5198504Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5198885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5199252Z return func(*args, **kwargs) 2025-08-14T21:55:16.5199625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5200018Z self_outputs = self.self( 2025-08-14T21:55:16.5200376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5200743Z return func(*args, **kwargs) 2025-08-14T21:55:16.5201114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:16.5201567Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:16.5201749Z 2025-08-14T21:55:16.5201861Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5202219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5202537Z return mod(**inputs) 2025-08-14T21:55:16.5202906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5203299Z outputs = self.roberta( 2025-08-14T21:55:16.5203667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5204060Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5204449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5204844Z layer_outputs = layer_module( 2025-08-14T21:55:16.5205177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5205643Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5206080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5206511Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5206938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5207342Z return func(*args, **kwargs) 2025-08-14T21:55:16.5207751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:16.5208238Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:16.5208747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:16.5209194Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5209344Z 2025-08-14T21:55:16.5209462Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5209844Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5210216Z return mod(**inputs) 2025-08-14T21:55:16.5210613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5211050Z outputs = self.roberta( 2025-08-14T21:55:16.5211462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5211894Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5212333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5212757Z layer_outputs = layer_module( 2025-08-14T21:55:16.5213135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5213529Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5213969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5214401Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5214789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5215173Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5215578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.5216039Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.5216467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:16.5216851Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5216984Z 2025-08-14T21:55:16.5217083Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5217433Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5217747Z return mod(**inputs) 2025-08-14T21:55:16.5218099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5218485Z outputs = self.roberta( 2025-08-14T21:55:16.5218855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5219253Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5219635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5220027Z layer_outputs = layer_module( 2025-08-14T21:55:16.5220372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5220719Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5221094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5221489Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5221906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5222279Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5222692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.5223194Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.5223633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:16.5224056Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:16.5224436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:16.5224780Z return self.act(input) 2025-08-14T21:55:16.5224892Z 2025-08-14T21:55:16.5225002Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5225372Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5225690Z return mod(**inputs) 2025-08-14T21:55:16.5226067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5226445Z outputs = self.roberta( 2025-08-14T21:55:16.5226813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5227197Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5227574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5227952Z layer_outputs = layer_module( 2025-08-14T21:55:16.5228288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5228644Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5229034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5229447Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5229844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5230246Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5230647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:16.5231128Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:16.5231579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:16.5231985Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5232123Z 2025-08-14T21:55:16.5232228Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5232588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5232911Z return mod(**inputs) 2025-08-14T21:55:16.5233275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5233666Z outputs = self.roberta( 2025-08-14T21:55:16.5234039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5234428Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5234806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5235198Z layer_outputs = layer_module( 2025-08-14T21:55:16.5235542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5235917Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5236312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5236722Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5237120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5237485Z return func(*args, **kwargs) 2025-08-14T21:55:16.5238043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5238445Z self_outputs = self.self( 2025-08-14T21:55:16.5238830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5239223Z return func(*args, **kwargs) 2025-08-14T21:55:16.5239701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:16.5240269Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:16.5240577Z 2025-08-14T21:55:16.5240699Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5241085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5241410Z return mod(**inputs) 2025-08-14T21:55:16.5241787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5242175Z outputs = self.roberta( 2025-08-14T21:55:16.5242551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5242949Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5243345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5243733Z layer_outputs = layer_module( 2025-08-14T21:55:16.5244084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5244448Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5244839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5245304Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5245705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5246079Z return func(*args, **kwargs) 2025-08-14T21:55:16.5246454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5246854Z self_outputs = self.self( 2025-08-14T21:55:16.5247211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5247575Z return func(*args, **kwargs) 2025-08-14T21:55:16.5247958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:16.5248355Z self.key(current_states) 2025-08-14T21:55:16.5248469Z 2025-08-14T21:55:16.5248581Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5248945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5249256Z return mod(**inputs) 2025-08-14T21:55:16.5249611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5249982Z outputs = self.roberta( 2025-08-14T21:55:16.5250374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5250741Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5251103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5251488Z layer_outputs = layer_module( 2025-08-14T21:55:16.5251812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5252144Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5252513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5252879Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5253234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5253593Z return func(*args, **kwargs) 2025-08-14T21:55:16.5253940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5254300Z self_outputs = self.self( 2025-08-14T21:55:16.5254645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5254990Z return func(*args, **kwargs) 2025-08-14T21:55:16.5255337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:16.5255698Z self.value(current_states) 2025-08-14T21:55:16.5255807Z 2025-08-14T21:55:16.5255893Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.5256116Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5256455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5256765Z return mod(**inputs) 2025-08-14T21:55:16.5257127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5257483Z outputs = self.roberta( 2025-08-14T21:55:16.5257833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5258205Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5258567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5258942Z layer_outputs = layer_module( 2025-08-14T21:55:16.5259273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5259620Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5259992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5260379Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5260743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5261098Z return func(*args, **kwargs) 2025-08-14T21:55:16.5261466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5261828Z self_outputs = self.self( 2025-08-14T21:55:16.5262162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5262496Z return func(*args, **kwargs) 2025-08-14T21:55:16.5262845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:16.5263263Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:16.5263453Z 2025-08-14T21:55:16.5263557Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5263883Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5264185Z return mod(**inputs) 2025-08-14T21:55:16.5264545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5264913Z outputs = self.roberta( 2025-08-14T21:55:16.5265254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5265622Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5265978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5266336Z layer_outputs = layer_module( 2025-08-14T21:55:16.5266671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5267011Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5267398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5267769Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5268119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5268472Z return func(*args, **kwargs) 2025-08-14T21:55:16.5268825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:16.5269253Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:16.5269671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:16.5270062Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5270203Z 2025-08-14T21:55:16.5270298Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5270630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5270938Z return mod(**inputs) 2025-08-14T21:55:16.5271286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5271651Z outputs = self.roberta( 2025-08-14T21:55:16.5272003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5272383Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5272745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5273119Z layer_outputs = layer_module( 2025-08-14T21:55:16.5273451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5273794Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5274165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5274560Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5274939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5275305Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5275708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.5276155Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.5276573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:16.5276991Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5277131Z 2025-08-14T21:55:16.5277230Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5277584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5277910Z return mod(**inputs) 2025-08-14T21:55:16.5278250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5278622Z outputs = self.roberta( 2025-08-14T21:55:16.5278979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5279344Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5279711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5280099Z layer_outputs = layer_module( 2025-08-14T21:55:16.5280429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5280784Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5281172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5281565Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5281952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5282325Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5282734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.5283186Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.5283608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:16.5284027Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:16.5284406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:16.5284748Z return self.act(input) 2025-08-14T21:55:16.5284859Z 2025-08-14T21:55:16.5284963Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5285400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5285735Z return mod(**inputs) 2025-08-14T21:55:16.5286095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5286483Z outputs = self.roberta( 2025-08-14T21:55:16.5286865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5287253Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5287639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5288035Z layer_outputs = layer_module( 2025-08-14T21:55:16.5288383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5288745Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5289133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5289539Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5289939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5290344Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5290766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:16.5291252Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:16.5291697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:16.5292120Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5292267Z 2025-08-14T21:55:16.5292371Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5292732Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5292801Z return mod(**inputs) 2025-08-14T21:55:16.5293069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5293141Z outputs = self.roberta( 2025-08-14T21:55:16.5293420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5293506Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5293782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5293857Z layer_outputs = layer_module( 2025-08-14T21:55:16.5294088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5294168Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5294435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5294518Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5294763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5294847Z return func(*args, **kwargs) 2025-08-14T21:55:16.5295108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5295192Z self_outputs = self.self( 2025-08-14T21:55:16.5295440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5295512Z return func(*args, **kwargs) 2025-08-14T21:55:16.5295783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:16.5295988Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:16.5295991Z 2025-08-14T21:55:16.5296098Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5296285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5296348Z return mod(**inputs) 2025-08-14T21:55:16.5296601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5296666Z outputs = self.roberta( 2025-08-14T21:55:16.5296905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5296983Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5297221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5297298Z layer_outputs = layer_module( 2025-08-14T21:55:16.5297499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5297574Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5297841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5297917Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5298144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5298236Z return func(*args, **kwargs) 2025-08-14T21:55:16.5298484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5298559Z self_outputs = self.self( 2025-08-14T21:55:16.5298790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5298857Z return func(*args, **kwargs) 2025-08-14T21:55:16.5299116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:16.5299188Z self.key(current_states) 2025-08-14T21:55:16.5299207Z 2025-08-14T21:55:16.5299317Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5299510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5299592Z return mod(**inputs) 2025-08-14T21:55:16.5299865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5299931Z outputs = self.roberta( 2025-08-14T21:55:16.5300179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5300257Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5300504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5300581Z layer_outputs = layer_module( 2025-08-14T21:55:16.5300792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5300867Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5301119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5301198Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5301422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5301495Z return func(*args, **kwargs) 2025-08-14T21:55:16.5301749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5301821Z self_outputs = self.self( 2025-08-14T21:55:16.5302043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5302110Z return func(*args, **kwargs) 2025-08-14T21:55:16.5302356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:16.5302424Z self.value(current_states) 2025-08-14T21:55:16.5302427Z 2025-08-14T21:55:16.5302513Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.5302608Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5302794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5302861Z return mod(**inputs) 2025-08-14T21:55:16.5303100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5303164Z outputs = self.roberta( 2025-08-14T21:55:16.5303409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5303494Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5303748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5303816Z layer_outputs = layer_module( 2025-08-14T21:55:16.5304026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5304124Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5304363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5304437Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5304665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5304727Z return func(*args, **kwargs) 2025-08-14T21:55:16.5304989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5305058Z self_outputs = self.self( 2025-08-14T21:55:16.5305282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5305379Z return func(*args, **kwargs) 2025-08-14T21:55:16.5305621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:16.5305750Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:16.5305753Z 2025-08-14T21:55:16.5305847Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5306033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5306103Z return mod(**inputs) 2025-08-14T21:55:16.5306343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5306409Z outputs = self.roberta( 2025-08-14T21:55:16.5306656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5306724Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5306973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5307043Z layer_outputs = layer_module( 2025-08-14T21:55:16.5307249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5307333Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5307576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5307655Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5307890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5307958Z return func(*args, **kwargs) 2025-08-14T21:55:16.5308212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:16.5308335Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:16.5308581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:16.5308668Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5308672Z 2025-08-14T21:55:16.5308768Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5308964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5309027Z return mod(**inputs) 2025-08-14T21:55:16.5309276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5309369Z outputs = self.roberta( 2025-08-14T21:55:16.5309624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5309698Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5309973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5310044Z layer_outputs = layer_module( 2025-08-14T21:55:16.5310264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5310340Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5310589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5310679Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5310947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5311041Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5311339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.5311462Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.5311724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:16.5311798Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5311801Z 2025-08-14T21:55:16.5311896Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5312091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5312154Z return mod(**inputs) 2025-08-14T21:55:16.5312418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5312485Z outputs = self.roberta( 2025-08-14T21:55:16.5312741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5312821Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5313073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5313148Z layer_outputs = layer_module( 2025-08-14T21:55:16.5313365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5313441Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5313703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5313785Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5314034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5314115Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5314404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.5314532Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.5314790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:16.5314899Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:16.5315114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:16.5315183Z return self.act(input) 2025-08-14T21:55:16.5315203Z 2025-08-14T21:55:16.5315316Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5315512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5315575Z return mod(**inputs) 2025-08-14T21:55:16.5315837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5315924Z outputs = self.roberta( 2025-08-14T21:55:16.5316180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5316258Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5316512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5316593Z layer_outputs = layer_module( 2025-08-14T21:55:16.5316853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5316931Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5317189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5317291Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5317540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5317620Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5317961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:16.5318097Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:16.5318351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:16.5318433Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5318436Z 2025-08-14T21:55:16.5318542Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5318735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5318809Z return mod(**inputs) 2025-08-14T21:55:16.5319061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5319128Z outputs = self.roberta( 2025-08-14T21:55:16.5319386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5319457Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5319706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5319782Z layer_outputs = layer_module( 2025-08-14T21:55:16.5319996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5320082Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5320335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5320416Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5320656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5320724Z return func(*args, **kwargs) 2025-08-14T21:55:16.5320983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5321050Z self_outputs = self.self( 2025-08-14T21:55:16.5321285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5321379Z return func(*args, **kwargs) 2025-08-14T21:55:16.5321638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:16.5321841Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:16.5321867Z 2025-08-14T21:55:16.5321967Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5322164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5322235Z return mod(**inputs) 2025-08-14T21:55:16.5322493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5322561Z outputs = self.roberta( 2025-08-14T21:55:16.5322823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5322906Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5323151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5323219Z layer_outputs = layer_module( 2025-08-14T21:55:16.5323440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5323525Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5323770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5323860Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5324086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5324153Z return func(*args, **kwargs) 2025-08-14T21:55:16.5324399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5324465Z self_outputs = self.self( 2025-08-14T21:55:16.5324683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5324768Z return func(*args, **kwargs) 2025-08-14T21:55:16.5325013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:16.5325091Z self.key(current_states) 2025-08-14T21:55:16.5325095Z 2025-08-14T21:55:16.5325195Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5325465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5325548Z return mod(**inputs) 2025-08-14T21:55:16.5325814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5325889Z outputs = self.roberta( 2025-08-14T21:55:16.5326157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5326243Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5326507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5326579Z layer_outputs = layer_module( 2025-08-14T21:55:16.5326792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5326879Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5327136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5327212Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5327449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5327537Z return func(*args, **kwargs) 2025-08-14T21:55:16.5327798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5327867Z self_outputs = self.self( 2025-08-14T21:55:16.5328111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5328189Z return func(*args, **kwargs) 2025-08-14T21:55:16.5328439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:16.5328520Z self.value(current_states) 2025-08-14T21:55:16.5328524Z 2025-08-14T21:55:16.5328607Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.5328704Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5328933Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5329000Z return mod(**inputs) 2025-08-14T21:55:16.5329255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5329348Z outputs = self.roberta( 2025-08-14T21:55:16.5329608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5329689Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5329954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5330023Z layer_outputs = layer_module( 2025-08-14T21:55:16.5330244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5330319Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5330580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5330667Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5330905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5330981Z return func(*args, **kwargs) 2025-08-14T21:55:16.5331250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5331316Z self_outputs = self.self( 2025-08-14T21:55:16.5331551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5331617Z return func(*args, **kwargs) 2025-08-14T21:55:16.5331874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:16.5332003Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:16.5332006Z 2025-08-14T21:55:16.5332102Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5332299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5332363Z return mod(**inputs) 2025-08-14T21:55:16.5332616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5332687Z outputs = self.roberta( 2025-08-14T21:55:16.5332936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5333010Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5333262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5333348Z layer_outputs = layer_module( 2025-08-14T21:55:16.5333566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5333638Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5333884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5334011Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5334237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5334307Z return func(*args, **kwargs) 2025-08-14T21:55:16.5334552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:16.5334671Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:16.5334941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:16.5335019Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5335023Z 2025-08-14T21:55:16.5335124Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5335328Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5335395Z return mod(**inputs) 2025-08-14T21:55:16.5335648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5335714Z outputs = self.roberta( 2025-08-14T21:55:16.5335961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5336038Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5336286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5336364Z layer_outputs = layer_module( 2025-08-14T21:55:16.5336575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5336647Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5336902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5336981Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5337232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5337305Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5337597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.5337857Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.5338139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:16.5338230Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5338243Z 2025-08-14T21:55:16.5338354Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5338564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5338644Z return mod(**inputs) 2025-08-14T21:55:16.5338920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5338993Z outputs = self.roberta( 2025-08-14T21:55:16.5339283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5339353Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5339609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5339725Z layer_outputs = layer_module( 2025-08-14T21:55:16.5339942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5340024Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5340293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5340380Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5340661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5340742Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5341059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.5341213Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.5341489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:16.5341648Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:16.5341877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:16.5341960Z return self.act(input) 2025-08-14T21:55:16.5341964Z 2025-08-14T21:55:16.5342071Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5342280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5342356Z return mod(**inputs) 2025-08-14T21:55:16.5342626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5342697Z outputs = self.roberta( 2025-08-14T21:55:16.5342982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5343059Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5343342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5343422Z layer_outputs = layer_module( 2025-08-14T21:55:16.5343653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5343743Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5344015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5344109Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5344377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5344461Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5344777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:16.5344915Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:16.5345192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:16.5345286Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5345290Z 2025-08-14T21:55:16.5345396Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5345612Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5345680Z return mod(**inputs) 2025-08-14T21:55:16.5345955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5346046Z outputs = self.roberta( 2025-08-14T21:55:16.5346296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5346373Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5346618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5346703Z layer_outputs = layer_module( 2025-08-14T21:55:16.5346923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5346996Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5347244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5347330Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5347576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5347651Z return func(*args, **kwargs) 2025-08-14T21:55:16.5347917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5347986Z self_outputs = self.self( 2025-08-14T21:55:16.5348222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5348288Z return func(*args, **kwargs) 2025-08-14T21:55:16.5348540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:16.5348748Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:16.5348752Z 2025-08-14T21:55:16.5348850Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5349052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5349115Z return mod(**inputs) 2025-08-14T21:55:16.5349366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5349441Z outputs = self.roberta( 2025-08-14T21:55:16.5349698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5349774Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5350017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5350084Z layer_outputs = layer_module( 2025-08-14T21:55:16.5350300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5350372Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5350622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5350708Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5350936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5351013Z return func(*args, **kwargs) 2025-08-14T21:55:16.5351257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5351324Z self_outputs = self.self( 2025-08-14T21:55:16.5351555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5351619Z return func(*args, **kwargs) 2025-08-14T21:55:16.5351868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:16.5351957Z self.key(current_states) 2025-08-14T21:55:16.5351961Z 2025-08-14T21:55:16.5352058Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5352256Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5352345Z return mod(**inputs) 2025-08-14T21:55:16.5352596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5352669Z outputs = self.roberta( 2025-08-14T21:55:16.5352918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5352997Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5353246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5353316Z layer_outputs = layer_module( 2025-08-14T21:55:16.5353552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5353627Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5353885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5353970Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5354194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5354269Z return func(*args, **kwargs) 2025-08-14T21:55:16.5354514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5354582Z self_outputs = self.self( 2025-08-14T21:55:16.5354818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5354888Z return func(*args, **kwargs) 2025-08-14T21:55:16.5355142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:16.5355212Z self.value(current_states) 2025-08-14T21:55:16.5355216Z 2025-08-14T21:55:16.5355457Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.5355561Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5355750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5355812Z return mod(**inputs) 2025-08-14T21:55:16.5356063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5356133Z outputs = self.roberta( 2025-08-14T21:55:16.5356392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5356465Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5356710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5356786Z layer_outputs = layer_module( 2025-08-14T21:55:16.5356991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5357074Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5357321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5357396Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5357633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5357700Z return func(*args, **kwargs) 2025-08-14T21:55:16.5357946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5358039Z self_outputs = self.self( 2025-08-14T21:55:16.5358274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5358364Z return func(*args, **kwargs) 2025-08-14T21:55:16.5358612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:16.5358736Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:16.5358739Z 2025-08-14T21:55:16.5358842Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5359032Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5359103Z return mod(**inputs) 2025-08-14T21:55:16.5359367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5359434Z outputs = self.roberta( 2025-08-14T21:55:16.5359691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5359776Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5360025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5360099Z layer_outputs = layer_module( 2025-08-14T21:55:16.5360307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5360387Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5360633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5360710Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5360944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5361010Z return func(*args, **kwargs) 2025-08-14T21:55:16.5361255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:16.5361383Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:16.5361628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:16.5361712Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5361715Z 2025-08-14T21:55:16.5361809Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5361998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5362067Z return mod(**inputs) 2025-08-14T21:55:16.5362316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5362387Z outputs = self.roberta( 2025-08-14T21:55:16.5362634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5362703Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5362953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5363022Z layer_outputs = layer_module( 2025-08-14T21:55:16.5363228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5363309Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5363553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5364887Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5365135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5365210Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5365579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.5365718Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.5365987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:16.5366068Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5366072Z 2025-08-14T21:55:16.5366174Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5366383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5366449Z return mod(**inputs) 2025-08-14T21:55:16.5366718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5366796Z outputs = self.roberta( 2025-08-14T21:55:16.5367065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5367145Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5367393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5367461Z layer_outputs = layer_module( 2025-08-14T21:55:16.5367675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5367748Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5368002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5368083Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5368327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5368411Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5368688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.5368801Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.5369053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:16.5369161Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:16.5369369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:16.5369438Z return self.act(input) 2025-08-14T21:55:16.5369443Z 2025-08-14T21:55:16.5369539Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5369735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5369799Z return mod(**inputs) 2025-08-14T21:55:16.5370055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5370121Z outputs = self.roberta( 2025-08-14T21:55:16.5370366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5370443Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5370687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5370755Z layer_outputs = layer_module( 2025-08-14T21:55:16.5370991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5371066Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5371317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5371411Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5371659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5371739Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5372015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:16.5372138Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:16.5372410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:16.5372490Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5372493Z 2025-08-14T21:55:16.5372596Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5372805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5372870Z return mod(**inputs) 2025-08-14T21:55:16.5373119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5373183Z outputs = self.roberta( 2025-08-14T21:55:16.5373430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5373496Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5373736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5373814Z layer_outputs = layer_module( 2025-08-14T21:55:16.5374018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5374090Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5374341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5374417Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5374647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5374713Z return func(*args, **kwargs) 2025-08-14T21:55:16.5374955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5375028Z self_outputs = self.self( 2025-08-14T21:55:16.5375252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5375328Z return func(*args, **kwargs) 2025-08-14T21:55:16.5375568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:16.5375761Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:16.5375765Z 2025-08-14T21:55:16.5375867Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5376050Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5376111Z return mod(**inputs) 2025-08-14T21:55:16.5376362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5376425Z outputs = self.roberta( 2025-08-14T21:55:16.5376673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5376766Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5377007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5377082Z layer_outputs = layer_module( 2025-08-14T21:55:16.5377302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5377379Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5377619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5377694Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5377923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5377986Z return func(*args, **kwargs) 2025-08-14T21:55:16.5378245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5378319Z self_outputs = self.self( 2025-08-14T21:55:16.5378555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5378628Z return func(*args, **kwargs) 2025-08-14T21:55:16.5378867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:16.5378933Z self.key(current_states) 2025-08-14T21:55:16.5378936Z 2025-08-14T21:55:16.5379039Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5379227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5379290Z return mod(**inputs) 2025-08-14T21:55:16.5379542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5379608Z outputs = self.roberta( 2025-08-14T21:55:16.5379860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5379930Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5380175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5380260Z layer_outputs = layer_module( 2025-08-14T21:55:16.5380461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5380540Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5380778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5380853Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5381083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5381145Z return func(*args, **kwargs) 2025-08-14T21:55:16.5381383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5381457Z self_outputs = self.self( 2025-08-14T21:55:16.5381673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5381744Z return func(*args, **kwargs) 2025-08-14T21:55:16.5381981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:16.5382049Z self.value(current_states) 2025-08-14T21:55:16.5382052Z 2025-08-14T21:55:16.5382136Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.5382230Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5382432Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5382503Z return mod(**inputs) 2025-08-14T21:55:16.5382743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5382833Z outputs = self.roberta( 2025-08-14T21:55:16.5383072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5383141Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5383389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5383455Z layer_outputs = layer_module( 2025-08-14T21:55:16.5383663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5383753Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5383995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5384080Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5384317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5384384Z return func(*args, **kwargs) 2025-08-14T21:55:16.5384636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5384708Z self_outputs = self.self( 2025-08-14T21:55:16.5384937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5385001Z return func(*args, **kwargs) 2025-08-14T21:55:16.5385243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:16.5385373Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:16.5385376Z 2025-08-14T21:55:16.5385468Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5385659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5385723Z return mod(**inputs) 2025-08-14T21:55:16.5385964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5386036Z outputs = self.roberta( 2025-08-14T21:55:16.5386276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5386342Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5386590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5386658Z layer_outputs = layer_module( 2025-08-14T21:55:16.5386869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5386939Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5387184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5387267Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5387488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5387551Z return func(*args, **kwargs) 2025-08-14T21:55:16.5387800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:16.5387919Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:16.5388184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:16.5388264Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5388267Z 2025-08-14T21:55:16.5388364Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5388577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5388640Z return mod(**inputs) 2025-08-14T21:55:16.5388890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5388954Z outputs = self.roberta( 2025-08-14T21:55:16.5389198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5389275Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5389542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5389611Z layer_outputs = layer_module( 2025-08-14T21:55:16.5389823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5389909Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5390157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5390234Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5390470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5390549Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5390815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.5390934Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.5391175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:16.5391250Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5391254Z 2025-08-14T21:55:16.5391356Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5391540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5391600Z return mod(**inputs) 2025-08-14T21:55:16.5391844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5391906Z outputs = self.roberta( 2025-08-14T21:55:16.5392150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5392218Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5392466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5392541Z layer_outputs = layer_module( 2025-08-14T21:55:16.5392750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5392834Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5393080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5393157Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5393406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5393478Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5393753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.5393891Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.5394149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:16.5394260Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:16.5394597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:16.5394682Z return self.act(input) 2025-08-14T21:55:16.5394685Z 2025-08-14T21:55:16.5394792Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5394985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5395057Z return mod(**inputs) 2025-08-14T21:55:16.5395307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5395398Z outputs = self.roberta( 2025-08-14T21:55:16.5395656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5395726Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5395998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5396079Z layer_outputs = layer_module( 2025-08-14T21:55:16.5396334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5396417Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5396660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5396739Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5396986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5397061Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5397340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:16.5397473Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:16.5397719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:16.5397802Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5397805Z 2025-08-14T21:55:16.5397904Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5398094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5398165Z return mod(**inputs) 2025-08-14T21:55:16.5398414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5398487Z outputs = self.roberta( 2025-08-14T21:55:16.5398730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5398801Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5399055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5399123Z layer_outputs = layer_module( 2025-08-14T21:55:16.5399332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5399415Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5399659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5399743Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5399991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5400057Z return func(*args, **kwargs) 2025-08-14T21:55:16.5400310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5400395Z self_outputs = self.self( 2025-08-14T21:55:16.5400633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5400700Z return func(*args, **kwargs) 2025-08-14T21:55:16.5400954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:16.5401195Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:16.5401199Z 2025-08-14T21:55:16.5401312Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5401506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5401569Z return mod(**inputs) 2025-08-14T21:55:16.5401844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5401919Z outputs = self.roberta( 2025-08-14T21:55:16.5402173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5402244Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5402501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5402570Z layer_outputs = layer_module( 2025-08-14T21:55:16.5402789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5402868Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5403120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5403208Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5403442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5403511Z return func(*args, **kwargs) 2025-08-14T21:55:16.5403771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5403838Z self_outputs = self.self( 2025-08-14T21:55:16.5404080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5404148Z return func(*args, **kwargs) 2025-08-14T21:55:16.5404400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:16.5404479Z self.key(current_states) 2025-08-14T21:55:16.5404482Z 2025-08-14T21:55:16.5404581Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5404783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5404847Z return mod(**inputs) 2025-08-14T21:55:16.5405103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5405180Z outputs = self.roberta( 2025-08-14T21:55:16.5405508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5405585Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5405854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5405949Z layer_outputs = layer_module( 2025-08-14T21:55:16.5406189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5406267Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5406521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5406626Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5406865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5406929Z return func(*args, **kwargs) 2025-08-14T21:55:16.5407188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5407254Z self_outputs = self.self( 2025-08-14T21:55:16.5407504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5407572Z return func(*args, **kwargs) 2025-08-14T21:55:16.5407843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:16.5407927Z self.value(current_states) 2025-08-14T21:55:16.5407931Z 2025-08-14T21:55:16.5408008Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.5408113Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5408298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5408359Z return mod(**inputs) 2025-08-14T21:55:16.5408608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5408672Z outputs = self.roberta( 2025-08-14T21:55:16.5408913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5408992Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5409238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5409313Z layer_outputs = layer_module( 2025-08-14T21:55:16.5409521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5409593Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5409843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5409920Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5410142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5410215Z return func(*args, **kwargs) 2025-08-14T21:55:16.5410461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5410534Z self_outputs = self.self( 2025-08-14T21:55:16.5410761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5410827Z return func(*args, **kwargs) 2025-08-14T21:55:16.5411078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:16.5411204Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:16.5411207Z 2025-08-14T21:55:16.5411308Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5411493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5411557Z return mod(**inputs) 2025-08-14T21:55:16.5411827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5411891Z outputs = self.roberta( 2025-08-14T21:55:16.5412138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5412257Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5412502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5412576Z layer_outputs = layer_module( 2025-08-14T21:55:16.5412783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5412855Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5413107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5413204Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5413430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5413504Z return func(*args, **kwargs) 2025-08-14T21:55:16.5413776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:16.5413904Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:16.5414145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:16.5414222Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5414226Z 2025-08-14T21:55:16.5414329Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5414517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5414589Z return mod(**inputs) 2025-08-14T21:55:16.5414828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5414891Z outputs = self.roberta( 2025-08-14T21:55:16.5415140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5415207Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5415452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5415525Z layer_outputs = layer_module( 2025-08-14T21:55:16.5415730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5415810Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5416059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5416141Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5416398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5416472Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5416764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.5416878Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.5417130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:16.5417216Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5417220Z 2025-08-14T21:55:16.5417315Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5417505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5417597Z return mod(**inputs) 2025-08-14T21:55:16.5417856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5417929Z outputs = self.roberta( 2025-08-14T21:55:16.5418184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5418252Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5418496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5418561Z layer_outputs = layer_module( 2025-08-14T21:55:16.5418772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5418843Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5419105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5419192Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5419442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5419515Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5419797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.5419905Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.5420154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:16.5420260Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:16.5420454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:16.5420526Z return self.act(input) 2025-08-14T21:55:16.5420530Z 2025-08-14T21:55:16.5420625Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5420820Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5420882Z return mod(**inputs) 2025-08-14T21:55:16.5421121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5421191Z outputs = self.roberta( 2025-08-14T21:55:16.5421430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5421496Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5421744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5421813Z layer_outputs = layer_module( 2025-08-14T21:55:16.5422027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5422101Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5422343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5422427Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5422662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5422733Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5423010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:16.5423131Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:16.5423385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:16.5423478Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5423481Z 2025-08-14T21:55:16.5423574Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5423768Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5423843Z return mod(**inputs) 2025-08-14T21:55:16.5424095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5424159Z outputs = self.roberta( 2025-08-14T21:55:16.5424404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5424480Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5424742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5424811Z layer_outputs = layer_module( 2025-08-14T21:55:16.5425019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5425108Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5425361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5425437Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5425657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5425732Z return func(*args, **kwargs) 2025-08-14T21:55:16.5425968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5426043Z self_outputs = self.self( 2025-08-14T21:55:16.5426265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5426330Z return func(*args, **kwargs) 2025-08-14T21:55:16.5426577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:16.5426765Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:16.5426772Z 2025-08-14T21:55:16.5426872Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5427054Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5427117Z return mod(**inputs) 2025-08-14T21:55:16.5427364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5427428Z outputs = self.roberta( 2025-08-14T21:55:16.5427674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5427752Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5428002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5428081Z layer_outputs = layer_module( 2025-08-14T21:55:16.5428286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5428359Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5428607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5428682Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5428908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5429000Z return func(*args, **kwargs) 2025-08-14T21:55:16.5429249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5429325Z self_outputs = self.self( 2025-08-14T21:55:16.5429558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5429639Z return func(*args, **kwargs) 2025-08-14T21:55:16.5429900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:16.5429968Z self.key(current_states) 2025-08-14T21:55:16.5429971Z 2025-08-14T21:55:16.5430076Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5430266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5430330Z return mod(**inputs) 2025-08-14T21:55:16.5430617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5430684Z outputs = self.roberta( 2025-08-14T21:55:16.5430946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5431025Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5431268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5431343Z layer_outputs = layer_module( 2025-08-14T21:55:16.5431550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5431622Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5431871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5431950Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5432173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5432246Z return func(*args, **kwargs) 2025-08-14T21:55:16.5432492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5432569Z self_outputs = self.self( 2025-08-14T21:55:16.5432795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5432859Z return func(*args, **kwargs) 2025-08-14T21:55:16.5433112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:16.5433181Z self.value(current_states) 2025-08-14T21:55:16.5433184Z 2025-08-14T21:55:16.5433269Z cudagraph partition due to non gpu ops 2025-08-14T21:55:16.5433368Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5433558Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5433629Z return mod(**inputs) 2025-08-14T21:55:16.5433872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5433937Z outputs = self.roberta( 2025-08-14T21:55:16.5434187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5434257Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5434507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5434574Z layer_outputs = layer_module( 2025-08-14T21:55:16.5434778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5434882Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5435129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5435207Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5435456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5435522Z return func(*args, **kwargs) 2025-08-14T21:55:16.5435772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:16.5435837Z self_outputs = self.self( 2025-08-14T21:55:16.5436060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5436133Z return func(*args, **kwargs) 2025-08-14T21:55:16.5436395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:16.5436527Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:16.5436530Z 2025-08-14T21:55:16.5436641Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5436832Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5436901Z return mod(**inputs) 2025-08-14T21:55:16.5437144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5437208Z outputs = self.roberta( 2025-08-14T21:55:16.5437459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5437528Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5437922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5437995Z layer_outputs = layer_module( 2025-08-14T21:55:16.5438206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5438292Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5438586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:16.5438674Z self_attention_outputs = self.attention( 2025-08-14T21:55:16.5438944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:16.5439018Z return func(*args, **kwargs) 2025-08-14T21:55:16.5439306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:16.5439435Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:16.5439701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:16.5439795Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5439799Z 2025-08-14T21:55:16.5439906Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5440125Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5440195Z return mod(**inputs) 2025-08-14T21:55:16.5440469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5440559Z outputs = self.roberta( 2025-08-14T21:55:16.5440805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5440874Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5441178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5441245Z layer_outputs = layer_module( 2025-08-14T21:55:16.5441461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5441576Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5441821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5441909Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5442154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5442239Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5442517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.5442658Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.5442916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:16.5443024Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5443029Z 2025-08-14T21:55:16.5443147Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5443338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5443403Z return mod(**inputs) 2025-08-14T21:55:16.5443659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5443727Z outputs = self.roberta( 2025-08-14T21:55:16.5443975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5444056Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5444301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5444379Z layer_outputs = layer_module( 2025-08-14T21:55:16.5444589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5444667Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5444922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5445006Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5445300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5445395Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5445680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:16.5445803Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:16.5446054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:16.5446175Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:16.5446381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:16.5446446Z return self.act(input) 2025-08-14T21:55:16.5446449Z 2025-08-14T21:55:16.5446550Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5446782Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5446843Z return mod(**inputs) 2025-08-14T21:55:16.5447096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 999, in forward 2025-08-14T21:55:16.5447184Z outputs = self.roberta( 2025-08-14T21:55:16.5447433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:16.5447512Z encoder_outputs = self.encoder( 2025-08-14T21:55:16.5447777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:16.5447854Z layer_outputs = layer_module( 2025-08-14T21:55:16.5448061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:16.5448134Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:16.5448387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:16.5448464Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:16.5448735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:16.5448809Z return forward_fn(*input_tensors) 2025-08-14T21:55:16.5449100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:16.5449234Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:16.5449479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:16.5449555Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:16.5449566Z 2025-08-14T21:55:16.5449662Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5449849Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5449918Z return mod(**inputs) 2025-08-14T21:55:16.5450172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1016, in forward 2025-08-14T21:55:16.5450264Z prediction_scores = self.lm_head(sequence_output) 2025-08-14T21:55:16.5450522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1149, in forward 2025-08-14T21:55:16.5450591Z x = self.dense(features) 2025-08-14T21:55:16.5450594Z 2025-08-14T21:55:16.5450695Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5450882Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5450945Z return mod(**inputs) 2025-08-14T21:55:16.5451199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1016, in forward 2025-08-14T21:55:16.5451291Z prediction_scores = self.lm_head(sequence_output) 2025-08-14T21:55:16.5451543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1154, in forward 2025-08-14T21:55:16.5451615Z x = self.decoder(x) 2025-08-14T21:55:16.5451619Z 2025-08-14T21:55:16.5451712Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:16.5451905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:16.5451969Z return mod(**inputs) 2025-08-14T21:55:16.5452216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1022, in forward 2025-08-14T21:55:16.5452296Z lm_loss = self.loss_function( 2025-08-14T21:55:16.5452524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-08-14T21:55:16.5452704Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-08-14T21:55:16.5452936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-08-14T21:55:16.5453130Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-08-14T21:55:16.5453134Z 2025-08-14T21:55:25.4577275Z Compilation time (from dynamo_timed): 15.001251532 2025-08-14T21:55:25.4715015Z pass 2025-08-14T21:55:25.4715540Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:55:25.4718631Z TIMING: _recursive_pre_grad_passes:0.00746 _recursive_joint_graph_passes:0.65129 _recursive_post_grad_passes:0.09676 async_compile.wait:0.77157 code_gen:7.76572 inductor_compile:8.93494 backend_compile:12.02083 gc:0.00105 entire_frame_compile:15.00125 total_wall_time:15.00125 2025-08-14T21:55:25.4720215Z STATS: call_* op count: 303 | FakeTensorMode.__torch_dispatch__:12464 | FakeTensor.__torch_dispatch__:4759 | ProxyTorchDispatchMode.__torch_dispatch__:4539 2025-08-14T21:55:25.4722310Z Dynamo produced 1 graphs covering 303 ops with 0 graph breaks (0 unique) 2025-08-14T21:55:30.5862328Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:55:30.5863290Z from pkg_resources import resource_filename 2025-08-14T21:55:31.1718957Z 2025-08-14T21:55:32.2994001Z loading model: 0it [00:00, ?it/s]We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-08-14T21:55:32.2995028Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-08-14T21:55:32.2996016Z WARNING:transformers.modeling_utils:We strongly recommend passing in an `attention_mask` since your input_ids may be padded. See https://huggingface.co/docs/transformers/troubleshooting#incorrect-output-when-padding-tokens-arent-masked. 2025-08-14T21:55:32.2996937Z You may ignore this warning if your `pad_token_id` (0) is identical to the `bos_token_id` (0), `eos_token_id` (2), or the `sep_token_id` (None), and your input is not padded. 2025-08-14T21:55:32.4373978Z 2025-08-14T21:55:32.4376348Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:55:32.4390632Z cpu eval RobertaForQuestionAnswering 2025-08-14T21:55:32.8731601Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:55:33.0848189Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:55:33.2861805Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:55:40.7307280Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7307973Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7308357Z return mod(**inputs) 2025-08-14T21:55:40.7312690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7318063Z outputs = self.roberta( 2025-08-14T21:55:40.7319211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-08-14T21:55:40.7319710Z embedding_output = self.embeddings( 2025-08-14T21:55:40.7320160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-08-14T21:55:40.7320797Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-14T21:55:40.7321485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1576, in create_position_ids_from_input_ids 2025-08-14T21:55:40.7322408Z mask = input_ids.ne(padding_idx).int() 2025-08-14T21:55:40.7322574Z 2025-08-14T21:55:40.7322674Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7322982Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7323206Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7323452Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7323682Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7323897Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7324121Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7324344Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7324559Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7324785Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7325008Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7325443Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7325782Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7326321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7326698Z return mod(**inputs) 2025-08-14T21:55:40.7327165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7327612Z outputs = self.roberta( 2025-08-14T21:55:40.7328044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-08-14T21:55:40.7328490Z embedding_output = self.embeddings( 2025-08-14T21:55:40.7328921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-08-14T21:55:40.7329489Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-14T21:55:40.7330135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1577, in create_position_ids_from_input_ids 2025-08-14T21:55:40.7330781Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-14T21:55:40.7331052Z 2025-08-14T21:55:40.7331176Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7331595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7331967Z return mod(**inputs) 2025-08-14T21:55:40.7332381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7332803Z outputs = self.roberta( 2025-08-14T21:55:40.7333213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 826, in forward 2025-08-14T21:55:40.7333653Z embedding_output = self.embeddings( 2025-08-14T21:55:40.7334077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 89, in forward 2025-08-14T21:55:40.7334666Z position_ids = create_position_ids_from_input_ids(input_ids, self.padding_idx, past_key_values_length) 2025-08-14T21:55:40.7335318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1577, in create_position_ids_from_input_ids 2025-08-14T21:55:40.7335937Z incremental_indices = (torch.cumsum(mask, dim=1).type_as(mask) + past_key_values_length) * mask 2025-08-14T21:55:40.7336205Z 2025-08-14T21:55:40.7336323Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7336717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7337068Z return mod(**inputs) 2025-08-14T21:55:40.7337483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7338140Z outputs = self.roberta( 2025-08-14T21:55:40.7338569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7339014Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7339484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7339916Z layer_outputs = layer_module( 2025-08-14T21:55:40.7340302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7340704Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7341132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7341583Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7342093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7342504Z return func(*args, **kwargs) 2025-08-14T21:55:40.7342966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7343410Z self_outputs = self.self( 2025-08-14T21:55:40.7343816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7344214Z return func(*args, **kwargs) 2025-08-14T21:55:40.7344635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:40.7345234Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:40.7345523Z 2025-08-14T21:55:40.7345649Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7346040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7346396Z return mod(**inputs) 2025-08-14T21:55:40.7346812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7347246Z outputs = self.roberta( 2025-08-14T21:55:40.7347649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7348111Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7348533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7348960Z layer_outputs = layer_module( 2025-08-14T21:55:40.7349342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7349741Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7350167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7350625Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7351045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7351446Z return func(*args, **kwargs) 2025-08-14T21:55:40.7351868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7352301Z self_outputs = self.self( 2025-08-14T21:55:40.7352694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7353089Z return func(*args, **kwargs) 2025-08-14T21:55:40.7353507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:40.7353964Z self.key(current_states) 2025-08-14T21:55:40.7354091Z 2025-08-14T21:55:40.7354213Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7354606Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7354988Z return mod(**inputs) 2025-08-14T21:55:40.7355395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7355821Z outputs = self.roberta( 2025-08-14T21:55:40.7356228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7356658Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7357081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7357528Z layer_outputs = layer_module( 2025-08-14T21:55:40.7357910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7358307Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7358747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7359188Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7359596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7359996Z return func(*args, **kwargs) 2025-08-14T21:55:40.7360398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7360820Z self_outputs = self.self( 2025-08-14T21:55:40.7361211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7361611Z return func(*args, **kwargs) 2025-08-14T21:55:40.7362014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:40.7362440Z self.value(current_states) 2025-08-14T21:55:40.7362569Z 2025-08-14T21:55:40.7362665Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7362917Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7363305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7363673Z return mod(**inputs) 2025-08-14T21:55:40.7364080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7364502Z outputs = self.roberta( 2025-08-14T21:55:40.7364908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7365434Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7365862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7366303Z layer_outputs = layer_module( 2025-08-14T21:55:40.7366691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7367098Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7367540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7367995Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7368427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7368845Z return func(*args, **kwargs) 2025-08-14T21:55:40.7369279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7369705Z self_outputs = self.self( 2025-08-14T21:55:40.7370099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7370526Z return func(*args, **kwargs) 2025-08-14T21:55:40.7370940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:40.7371435Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:40.7371645Z 2025-08-14T21:55:40.7371764Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7372148Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7372498Z return mod(**inputs) 2025-08-14T21:55:40.7372917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7373330Z outputs = self.roberta( 2025-08-14T21:55:40.7373750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7374183Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7374603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7375021Z layer_outputs = layer_module( 2025-08-14T21:55:40.7375408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7375784Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7376194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7376624Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7377029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7377427Z return func(*args, **kwargs) 2025-08-14T21:55:40.7377822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:40.7378298Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:40.7378766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:40.7379196Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7379343Z 2025-08-14T21:55:40.7379456Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7379836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7380183Z return mod(**inputs) 2025-08-14T21:55:40.7380575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7380995Z outputs = self.roberta( 2025-08-14T21:55:40.7381396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7381816Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7382218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7382637Z layer_outputs = layer_module( 2025-08-14T21:55:40.7383008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7383390Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7383811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7384290Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7384742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7385171Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7385655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7386169Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7386653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:40.7387073Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7387229Z 2025-08-14T21:55:40.7387339Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7387749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7388115Z return mod(**inputs) 2025-08-14T21:55:40.7388497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7388933Z outputs = self.roberta( 2025-08-14T21:55:40.7389331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7389745Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7390164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7390592Z layer_outputs = layer_module( 2025-08-14T21:55:40.7390953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7391335Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7391772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7392202Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7392630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7393059Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7393512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7394023Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7394501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:40.7394975Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:40.7395385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:40.7395754Z return self.act(input) 2025-08-14T21:55:40.7395875Z 2025-08-14T21:55:40.7395986Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7396374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7396728Z return mod(**inputs) 2025-08-14T21:55:40.7397121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7397550Z outputs = self.roberta( 2025-08-14T21:55:40.7397974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7398409Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7398838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7399294Z layer_outputs = layer_module( 2025-08-14T21:55:40.7399677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7400066Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7400522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7401001Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7401448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7401877Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7402336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:40.7402866Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:40.7403388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:40.7403827Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7403985Z 2025-08-14T21:55:40.7404098Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7404510Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7404859Z return mod(**inputs) 2025-08-14T21:55:40.7405351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7405789Z outputs = self.roberta( 2025-08-14T21:55:40.7406198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7406623Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7407051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7407486Z layer_outputs = layer_module( 2025-08-14T21:55:40.7407864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7408247Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7408681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7409123Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7409532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7409936Z return func(*args, **kwargs) 2025-08-14T21:55:40.7410349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7410772Z self_outputs = self.self( 2025-08-14T21:55:40.7411158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7411563Z return func(*args, **kwargs) 2025-08-14T21:55:40.7411977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:40.7412551Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:40.7412836Z 2025-08-14T21:55:40.7412946Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7413324Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7413662Z return mod(**inputs) 2025-08-14T21:55:40.7414045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7414471Z outputs = self.roberta( 2025-08-14T21:55:40.7414915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7415349Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7415769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7416230Z layer_outputs = layer_module( 2025-08-14T21:55:40.7416597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7416974Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7417419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7417844Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7418246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7418657Z return func(*args, **kwargs) 2025-08-14T21:55:40.7419064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7419476Z self_outputs = self.self( 2025-08-14T21:55:40.7419882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7420285Z return func(*args, **kwargs) 2025-08-14T21:55:40.7420701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:40.7421134Z self.key(current_states) 2025-08-14T21:55:40.7421261Z 2025-08-14T21:55:40.7421378Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7421776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7422149Z return mod(**inputs) 2025-08-14T21:55:40.7422560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7423005Z outputs = self.roberta( 2025-08-14T21:55:40.7423418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7423858Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7424280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7424710Z layer_outputs = layer_module( 2025-08-14T21:55:40.7425083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7425475Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7425907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7426359Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7426791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7427207Z return func(*args, **kwargs) 2025-08-14T21:55:40.7427624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7428063Z self_outputs = self.self( 2025-08-14T21:55:40.7428462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7428863Z return func(*args, **kwargs) 2025-08-14T21:55:40.7429280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:40.7429715Z self.value(current_states) 2025-08-14T21:55:40.7429845Z 2025-08-14T21:55:40.7429946Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7430238Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7430642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7431007Z return mod(**inputs) 2025-08-14T21:55:40.7431407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7431874Z outputs = self.roberta( 2025-08-14T21:55:40.7432282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7449029Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7449556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7450004Z layer_outputs = layer_module( 2025-08-14T21:55:40.7450401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7450989Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7451454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7451961Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7452404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7452824Z return func(*args, **kwargs) 2025-08-14T21:55:40.7453253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7453686Z self_outputs = self.self( 2025-08-14T21:55:40.7454086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7454494Z return func(*args, **kwargs) 2025-08-14T21:55:40.7454911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:40.7455410Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:40.7455627Z 2025-08-14T21:55:40.7455753Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7456162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7456520Z return mod(**inputs) 2025-08-14T21:55:40.7456941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7457383Z outputs = self.roberta( 2025-08-14T21:55:40.7457790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7458227Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7458656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7459087Z layer_outputs = layer_module( 2025-08-14T21:55:40.7459462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7459859Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7460295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7460735Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7461154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7461671Z return func(*args, **kwargs) 2025-08-14T21:55:40.7462089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:40.7462579Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:40.7463104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:40.7463544Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7463698Z 2025-08-14T21:55:40.7463824Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7464250Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7464611Z return mod(**inputs) 2025-08-14T21:55:40.7465020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7465449Z outputs = self.roberta( 2025-08-14T21:55:40.7465851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7466279Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7466725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7467153Z layer_outputs = layer_module( 2025-08-14T21:55:40.7467575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7467975Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7468409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7468845Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7469286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7469720Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7470183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7470696Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7471178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:40.7471625Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7471781Z 2025-08-14T21:55:40.7471907Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7472295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7472649Z return mod(**inputs) 2025-08-14T21:55:40.7473057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7473484Z outputs = self.roberta( 2025-08-14T21:55:40.7473894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7474327Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7474751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7475169Z layer_outputs = layer_module( 2025-08-14T21:55:40.7475544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7475940Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7476367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7476807Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7477242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7477668Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7478124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7478668Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7479147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:40.7479640Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:40.7480051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:40.7480428Z return self.act(input) 2025-08-14T21:55:40.7480550Z 2025-08-14T21:55:40.7480671Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7481057Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7481414Z return mod(**inputs) 2025-08-14T21:55:40.7481837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7482274Z outputs = self.roberta( 2025-08-14T21:55:40.7482673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7483121Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7483543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7483970Z layer_outputs = layer_module( 2025-08-14T21:55:40.7484340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7484726Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7485156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7485772Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7486223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7486649Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7487111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:40.7487641Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:40.7488141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:40.7488589Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7488737Z 2025-08-14T21:55:40.7488859Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7489241Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7489598Z return mod(**inputs) 2025-08-14T21:55:40.7490007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7490429Z outputs = self.roberta( 2025-08-14T21:55:40.7490837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7491270Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7491687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7492112Z layer_outputs = layer_module( 2025-08-14T21:55:40.7492481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7492877Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7493311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7493797Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7494211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7494625Z return func(*args, **kwargs) 2025-08-14T21:55:40.7495034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7495481Z self_outputs = self.self( 2025-08-14T21:55:40.7495872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7496274Z return func(*args, **kwargs) 2025-08-14T21:55:40.7496688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:40.7497265Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:40.7497550Z 2025-08-14T21:55:40.7497690Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7498080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7498440Z return mod(**inputs) 2025-08-14T21:55:40.7498871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7499308Z outputs = self.roberta( 2025-08-14T21:55:40.7499709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7500139Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7500572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7501003Z layer_outputs = layer_module( 2025-08-14T21:55:40.7501380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7501772Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7502214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7502648Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7503060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7503464Z return func(*args, **kwargs) 2025-08-14T21:55:40.7503877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7504295Z self_outputs = self.self( 2025-08-14T21:55:40.7504684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7505084Z return func(*args, **kwargs) 2025-08-14T21:55:40.7505494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:40.7505921Z self.key(current_states) 2025-08-14T21:55:40.7506045Z 2025-08-14T21:55:40.7506166Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7506556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7506903Z return mod(**inputs) 2025-08-14T21:55:40.7507313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7507743Z outputs = self.roberta( 2025-08-14T21:55:40.7508140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7508572Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7508997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7509450Z layer_outputs = layer_module( 2025-08-14T21:55:40.7509814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7510204Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7510656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7511084Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7511501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7511905Z return func(*args, **kwargs) 2025-08-14T21:55:40.7512315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7512733Z self_outputs = self.self( 2025-08-14T21:55:40.7513135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7513537Z return func(*args, **kwargs) 2025-08-14T21:55:40.7513968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:40.7514393Z self.value(current_states) 2025-08-14T21:55:40.7514529Z 2025-08-14T21:55:40.7514619Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7514884Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7515269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7515621Z return mod(**inputs) 2025-08-14T21:55:40.7516030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7516466Z outputs = self.roberta( 2025-08-14T21:55:40.7516869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7517301Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7517726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7518147Z layer_outputs = layer_module( 2025-08-14T21:55:40.7518523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7518917Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7519363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7519807Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7520221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7520627Z return func(*args, **kwargs) 2025-08-14T21:55:40.7521034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7521458Z self_outputs = self.self( 2025-08-14T21:55:40.7521847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7522248Z return func(*args, **kwargs) 2025-08-14T21:55:40.7522657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:40.7523150Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:40.7523352Z 2025-08-14T21:55:40.7523474Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7523864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7524233Z return mod(**inputs) 2025-08-14T21:55:40.7524643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7525072Z outputs = self.roberta( 2025-08-14T21:55:40.7525579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7526046Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7526471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7526911Z layer_outputs = layer_module( 2025-08-14T21:55:40.7527294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7527681Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7528166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7528618Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7529037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7529463Z return func(*args, **kwargs) 2025-08-14T21:55:40.7529880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:40.7530363Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:40.7530844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:40.7531280Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7531437Z 2025-08-14T21:55:40.7531551Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7531942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7532291Z return mod(**inputs) 2025-08-14T21:55:40.7532702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7533128Z outputs = self.roberta( 2025-08-14T21:55:40.7533533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7533959Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7534382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7534810Z layer_outputs = layer_module( 2025-08-14T21:55:40.7535176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7535575Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7536014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7536468Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7536894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7537321Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7537929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7538457Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7538944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:40.7539389Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7539540Z 2025-08-14T21:55:40.7539660Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7540122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7540474Z return mod(**inputs) 2025-08-14T21:55:40.7540885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7541354Z outputs = self.roberta( 2025-08-14T21:55:40.7541757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7542190Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7542611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7543035Z layer_outputs = layer_module( 2025-08-14T21:55:40.7543408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7543795Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7544427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7544872Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7545335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7545769Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7546231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7546734Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7547211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:40.7547680Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:40.7548091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:40.7548461Z return self.act(input) 2025-08-14T21:55:40.7548587Z 2025-08-14T21:55:40.7548995Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7549386Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7549734Z return mod(**inputs) 2025-08-14T21:55:40.7550139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7550568Z outputs = self.roberta( 2025-08-14T21:55:40.7550976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7551401Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7551811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7552243Z layer_outputs = layer_module( 2025-08-14T21:55:40.7552609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7553003Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7553436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7553872Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7554297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7554722Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7555166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:40.7555667Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:40.7556167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:40.7556593Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7556736Z 2025-08-14T21:55:40.7556852Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7557238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7557576Z return mod(**inputs) 2025-08-14T21:55:40.7557967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7558378Z outputs = self.roberta( 2025-08-14T21:55:40.7558764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7559178Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7559603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7560024Z layer_outputs = layer_module( 2025-08-14T21:55:40.7560400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7560782Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7561218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7561647Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7562061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7562467Z return func(*args, **kwargs) 2025-08-14T21:55:40.7562880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7563296Z self_outputs = self.self( 2025-08-14T21:55:40.7563684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7564107Z return func(*args, **kwargs) 2025-08-14T21:55:40.7564523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:40.7565112Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:40.7565482Z 2025-08-14T21:55:40.7565599Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7565990Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7566330Z return mod(**inputs) 2025-08-14T21:55:40.7566735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7567154Z outputs = self.roberta( 2025-08-14T21:55:40.7567543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7567951Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7568362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7568775Z layer_outputs = layer_module( 2025-08-14T21:55:40.7569128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7569519Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7569946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7570377Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7570768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7571199Z return func(*args, **kwargs) 2025-08-14T21:55:40.7571598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7572000Z self_outputs = self.self( 2025-08-14T21:55:40.7572437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7572831Z return func(*args, **kwargs) 2025-08-14T21:55:40.7573229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:40.7573636Z self.key(current_states) 2025-08-14T21:55:40.7573763Z 2025-08-14T21:55:40.7573873Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7574248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7574624Z return mod(**inputs) 2025-08-14T21:55:40.7575012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7575428Z outputs = self.roberta( 2025-08-14T21:55:40.7575878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7576287Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7576693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7577104Z layer_outputs = layer_module( 2025-08-14T21:55:40.7577463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7577830Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7578248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7578671Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7579071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7579465Z return func(*args, **kwargs) 2025-08-14T21:55:40.7579864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7580274Z self_outputs = self.self( 2025-08-14T21:55:40.7580641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7581037Z return func(*args, **kwargs) 2025-08-14T21:55:40.7581435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:40.7581846Z self.value(current_states) 2025-08-14T21:55:40.7581972Z 2025-08-14T21:55:40.7582061Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7582312Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7582684Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7583018Z return mod(**inputs) 2025-08-14T21:55:40.7583412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7583835Z outputs = self.roberta( 2025-08-14T21:55:40.7584237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7584655Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7585074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7585498Z layer_outputs = layer_module( 2025-08-14T21:55:40.7585890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7586277Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7586726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7587202Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7587605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7588004Z return func(*args, **kwargs) 2025-08-14T21:55:40.7588417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7588827Z self_outputs = self.self( 2025-08-14T21:55:40.7589212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7589611Z return func(*args, **kwargs) 2025-08-14T21:55:40.7590037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:40.7590522Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:40.7590750Z 2025-08-14T21:55:40.7590867Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7591255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7591606Z return mod(**inputs) 2025-08-14T21:55:40.7592003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7592430Z outputs = self.roberta( 2025-08-14T21:55:40.7592833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7593264Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7593686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7594109Z layer_outputs = layer_module( 2025-08-14T21:55:40.7594482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7594866Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7595296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7595731Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7596137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7596535Z return func(*args, **kwargs) 2025-08-14T21:55:40.7596942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:40.7597428Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:40.7597903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:40.7598349Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7598508Z 2025-08-14T21:55:40.7598621Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7599007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7599351Z return mod(**inputs) 2025-08-14T21:55:40.7599754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7600184Z outputs = self.roberta( 2025-08-14T21:55:40.7600587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7601030Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7601443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7601854Z layer_outputs = layer_module( 2025-08-14T21:55:40.7602211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7602609Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7603025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7603475Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7603885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7604296Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7604785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7605384Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7605895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:40.7606355Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7606501Z 2025-08-14T21:55:40.7606620Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7606994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7607339Z return mod(**inputs) 2025-08-14T21:55:40.7607737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7608152Z outputs = self.roberta( 2025-08-14T21:55:40.7608557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7608987Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7609396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7609804Z layer_outputs = layer_module( 2025-08-14T21:55:40.7610171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7610546Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7610983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7611413Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7611827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7612238Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7612680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7613182Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7613652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:40.7614114Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:40.7614508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:40.7614869Z return self.act(input) 2025-08-14T21:55:40.7614985Z 2025-08-14T21:55:40.7615100Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7615474Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7615809Z return mod(**inputs) 2025-08-14T21:55:40.7616229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7616651Z outputs = self.roberta( 2025-08-14T21:55:40.7617041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7617484Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7617899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7618314Z layer_outputs = layer_module( 2025-08-14T21:55:40.7618668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7619045Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7619469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7619923Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7620349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7620760Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7621230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:40.7621744Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:40.7622244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:40.7622679Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7622823Z 2025-08-14T21:55:40.7622938Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7623378Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7623738Z return mod(**inputs) 2025-08-14T21:55:40.7624134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7624569Z outputs = self.roberta( 2025-08-14T21:55:40.7624974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7625405Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7625823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7626241Z layer_outputs = layer_module( 2025-08-14T21:55:40.7626613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7627003Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7627453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7627886Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7628301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7628706Z return func(*args, **kwargs) 2025-08-14T21:55:40.7629112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7629535Z self_outputs = self.self( 2025-08-14T21:55:40.7629925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7630325Z return func(*args, **kwargs) 2025-08-14T21:55:40.7630734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:40.7631308Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:40.7631635Z 2025-08-14T21:55:40.7631755Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7632144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7632511Z return mod(**inputs) 2025-08-14T21:55:40.7632913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7633337Z outputs = self.roberta( 2025-08-14T21:55:40.7633731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7634158Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7634577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7635003Z layer_outputs = layer_module( 2025-08-14T21:55:40.7635393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7635787Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7636233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7636663Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7637075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7637475Z return func(*args, **kwargs) 2025-08-14T21:55:40.7638016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7638442Z self_outputs = self.self( 2025-08-14T21:55:40.7638839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7639247Z return func(*args, **kwargs) 2025-08-14T21:55:40.7639658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:40.7640092Z self.key(current_states) 2025-08-14T21:55:40.7640229Z 2025-08-14T21:55:40.7640346Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7640740Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7641091Z return mod(**inputs) 2025-08-14T21:55:40.7641501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7641935Z outputs = self.roberta( 2025-08-14T21:55:40.7642342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7642764Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7643189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7643615Z layer_outputs = layer_module( 2025-08-14T21:55:40.7643983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7644381Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7644809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7645291Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7645705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7646108Z return func(*args, **kwargs) 2025-08-14T21:55:40.7646521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7647014Z self_outputs = self.self( 2025-08-14T21:55:40.7647404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7647808Z return func(*args, **kwargs) 2025-08-14T21:55:40.7648223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:40.7648696Z self.value(current_states) 2025-08-14T21:55:40.7648830Z 2025-08-14T21:55:40.7648919Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7649178Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7649555Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7649906Z return mod(**inputs) 2025-08-14T21:55:40.7650311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7650767Z outputs = self.roberta( 2025-08-14T21:55:40.7651168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7651595Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7652038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7652470Z layer_outputs = layer_module( 2025-08-14T21:55:40.7652837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7653222Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7653649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7654074Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7654495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7654904Z return func(*args, **kwargs) 2025-08-14T21:55:40.7655316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7655729Z self_outputs = self.self( 2025-08-14T21:55:40.7656130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7656540Z return func(*args, **kwargs) 2025-08-14T21:55:40.7656949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:40.7657436Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:40.7657641Z 2025-08-14T21:55:40.7657755Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7658140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7658485Z return mod(**inputs) 2025-08-14T21:55:40.7658889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7659318Z outputs = self.roberta( 2025-08-14T21:55:40.7659731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7660135Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7660539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7660948Z layer_outputs = layer_module( 2025-08-14T21:55:40.7661304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7661679Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7662123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7662544Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7662947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7663360Z return func(*args, **kwargs) 2025-08-14T21:55:40.7663759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:40.7664231Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:40.7664712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:40.7665150Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7665300Z 2025-08-14T21:55:40.7665421Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7665836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7666181Z return mod(**inputs) 2025-08-14T21:55:40.7666589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7667024Z outputs = self.roberta( 2025-08-14T21:55:40.7667424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7667854Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7668273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7668695Z layer_outputs = layer_module( 2025-08-14T21:55:40.7669060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7669438Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7669860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7670280Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7670702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7671112Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7671556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7672048Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7672506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:40.7672934Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7673079Z 2025-08-14T21:55:40.7673188Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7673561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7673899Z return mod(**inputs) 2025-08-14T21:55:40.7674294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7674707Z outputs = self.roberta( 2025-08-14T21:55:40.7675099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7675525Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7675952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7676369Z layer_outputs = layer_module( 2025-08-14T21:55:40.7676739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7677158Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7677586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7678029Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7678482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7678909Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7679359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7679871Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7680361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:40.7680837Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:40.7681248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:40.7681614Z return self.act(input) 2025-08-14T21:55:40.7681733Z 2025-08-14T21:55:40.7681869Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7682254Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7682606Z return mod(**inputs) 2025-08-14T21:55:40.7683009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7683434Z outputs = self.roberta( 2025-08-14T21:55:40.7683828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7684262Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7684684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7685104Z layer_outputs = layer_module( 2025-08-14T21:55:40.7685570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7685967Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7686401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7686845Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7687282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7687713Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7688165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:40.7688694Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:40.7689188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:40.7689639Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7689790Z 2025-08-14T21:55:40.7689905Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7690295Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7690644Z return mod(**inputs) 2025-08-14T21:55:40.7691049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7691474Z outputs = self.roberta( 2025-08-14T21:55:40.7691878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7692338Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7692753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7693180Z layer_outputs = layer_module( 2025-08-14T21:55:40.7693558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7695189Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7695623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7696084Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7696504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7696923Z return func(*args, **kwargs) 2025-08-14T21:55:40.7697362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7697798Z self_outputs = self.self( 2025-08-14T21:55:40.7698202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7698649Z return func(*args, **kwargs) 2025-08-14T21:55:40.7699062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:40.7699630Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:40.7699911Z 2025-08-14T21:55:40.7700028Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7700407Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7700755Z return mod(**inputs) 2025-08-14T21:55:40.7701159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7701578Z outputs = self.roberta( 2025-08-14T21:55:40.7701980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7702406Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7702834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7703250Z layer_outputs = layer_module( 2025-08-14T21:55:40.7703619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7704008Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7704452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7704892Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7705309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7705712Z return func(*args, **kwargs) 2025-08-14T21:55:40.7706115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7706538Z self_outputs = self.self( 2025-08-14T21:55:40.7706922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7707328Z return func(*args, **kwargs) 2025-08-14T21:55:40.7707731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:40.7708154Z self.key(current_states) 2025-08-14T21:55:40.7708275Z 2025-08-14T21:55:40.7708392Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7708769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7709144Z return mod(**inputs) 2025-08-14T21:55:40.7709563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7710000Z outputs = self.roberta( 2025-08-14T21:55:40.7710421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7710851Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7711266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7711686Z layer_outputs = layer_module( 2025-08-14T21:55:40.7712050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7712436Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7712891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7713307Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7713723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7714124Z return func(*args, **kwargs) 2025-08-14T21:55:40.7714538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7714972Z self_outputs = self.self( 2025-08-14T21:55:40.7715364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7715765Z return func(*args, **kwargs) 2025-08-14T21:55:40.7716169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:40.7716603Z self.value(current_states) 2025-08-14T21:55:40.7716738Z 2025-08-14T21:55:40.7716830Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7717086Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7717471Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7717826Z return mod(**inputs) 2025-08-14T21:55:40.7718233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7718652Z outputs = self.roberta( 2025-08-14T21:55:40.7719054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7719480Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7719896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7720316Z layer_outputs = layer_module( 2025-08-14T21:55:40.7720690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7721078Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7721510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7721941Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7722353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7722753Z return func(*args, **kwargs) 2025-08-14T21:55:40.7723158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7723581Z self_outputs = self.self( 2025-08-14T21:55:40.7723975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7724430Z return func(*args, **kwargs) 2025-08-14T21:55:40.7724845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:40.7725433Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:40.7725663Z 2025-08-14T21:55:40.7725786Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7726172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7726526Z return mod(**inputs) 2025-08-14T21:55:40.7726933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7727376Z outputs = self.roberta( 2025-08-14T21:55:40.7727775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7728234Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7728660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7729088Z layer_outputs = layer_module( 2025-08-14T21:55:40.7729468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7729862Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7730299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7730737Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7731147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7731542Z return func(*args, **kwargs) 2025-08-14T21:55:40.7731953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:40.7732432Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:40.7732916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:40.7733356Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7733504Z 2025-08-14T21:55:40.7733621Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7734001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7734351Z return mod(**inputs) 2025-08-14T21:55:40.7734752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7735181Z outputs = self.roberta( 2025-08-14T21:55:40.7735588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7736010Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7736427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7736846Z layer_outputs = layer_module( 2025-08-14T21:55:40.7737222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7737753Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7738191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7738637Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7739078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7739509Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7740030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7740542Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7741022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:40.7741496Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7741648Z 2025-08-14T21:55:40.7741760Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7742156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7742511Z return mod(**inputs) 2025-08-14T21:55:40.7742911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7743353Z outputs = self.roberta( 2025-08-14T21:55:40.7743778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7744203Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7744641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7745073Z layer_outputs = layer_module( 2025-08-14T21:55:40.7745450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7745841Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7746268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7746697Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7747120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7747537Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7747999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7748514Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7748995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:40.7749447Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:40.7749844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:40.7750199Z return self.act(input) 2025-08-14T21:55:40.7750314Z 2025-08-14T21:55:40.7750423Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7750802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7751157Z return mod(**inputs) 2025-08-14T21:55:40.7751560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7751979Z outputs = self.roberta( 2025-08-14T21:55:40.7752405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7752834Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7753266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7753693Z layer_outputs = layer_module( 2025-08-14T21:55:40.7754067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7754457Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7754898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7755362Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7755794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7756246Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7756681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:40.7757191Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:40.7757664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:40.7758090Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7758236Z 2025-08-14T21:55:40.7758344Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7758749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7758837Z return mod(**inputs) 2025-08-14T21:55:40.7759143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7759222Z outputs = self.roberta( 2025-08-14T21:55:40.7759497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7759583Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7759857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7759939Z layer_outputs = layer_module( 2025-08-14T21:55:40.7760170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7760254Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7760536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7760622Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7760886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7760963Z return func(*args, **kwargs) 2025-08-14T21:55:40.7761235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7761318Z self_outputs = self.self( 2025-08-14T21:55:40.7761580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7761657Z return func(*args, **kwargs) 2025-08-14T21:55:40.7761945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:40.7762176Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:40.7762180Z 2025-08-14T21:55:40.7762300Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7762516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7762591Z return mod(**inputs) 2025-08-14T21:55:40.7762891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7762967Z outputs = self.roberta( 2025-08-14T21:55:40.7763254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7763334Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7763615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7763728Z layer_outputs = layer_module( 2025-08-14T21:55:40.7763968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7764054Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7764363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7764452Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7764714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7764790Z return func(*args, **kwargs) 2025-08-14T21:55:40.7765067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7765151Z self_outputs = self.self( 2025-08-14T21:55:40.7765499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7765585Z return func(*args, **kwargs) 2025-08-14T21:55:40.7765894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:40.7765973Z self.key(current_states) 2025-08-14T21:55:40.7765978Z 2025-08-14T21:55:40.7766098Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7766315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7766388Z return mod(**inputs) 2025-08-14T21:55:40.7766683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7766759Z outputs = self.roberta( 2025-08-14T21:55:40.7767048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7767131Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7767410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7767498Z layer_outputs = layer_module( 2025-08-14T21:55:40.7767738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7767824Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7768113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7768201Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7768465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7768541Z return func(*args, **kwargs) 2025-08-14T21:55:40.7768825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7768912Z self_outputs = self.self( 2025-08-14T21:55:40.7769173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7769257Z return func(*args, **kwargs) 2025-08-14T21:55:40.7769538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:40.7769616Z self.value(current_states) 2025-08-14T21:55:40.7769620Z 2025-08-14T21:55:40.7769716Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7769845Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7770061Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7770139Z return mod(**inputs) 2025-08-14T21:55:40.7770423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7770527Z outputs = self.roberta( 2025-08-14T21:55:40.7770814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7770911Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7771198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7771275Z layer_outputs = layer_module( 2025-08-14T21:55:40.7771511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7771602Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7771883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7771977Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7772254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7772331Z return func(*args, **kwargs) 2025-08-14T21:55:40.7772640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7772719Z self_outputs = self.self( 2025-08-14T21:55:40.7772976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7773058Z return func(*args, **kwargs) 2025-08-14T21:55:40.7773340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:40.7773490Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:40.7773494Z 2025-08-14T21:55:40.7773605Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7773822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7773902Z return mod(**inputs) 2025-08-14T21:55:40.7774185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7774267Z outputs = self.roberta( 2025-08-14T21:55:40.7774546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7774624Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7774912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7774989Z layer_outputs = layer_module( 2025-08-14T21:55:40.7775225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7775320Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7775600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7775694Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7775951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7776025Z return func(*args, **kwargs) 2025-08-14T21:55:40.7776313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:40.7776450Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:40.7776734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:40.7776824Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7776848Z 2025-08-14T21:55:40.7776960Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7777182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7777253Z return mod(**inputs) 2025-08-14T21:55:40.7777542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7777642Z outputs = self.roberta( 2025-08-14T21:55:40.7777925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7778011Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7778288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7778366Z layer_outputs = layer_module( 2025-08-14T21:55:40.7778609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7778716Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7779015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7779124Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7779409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7779499Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7779819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7779948Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7780252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:40.7780341Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7780347Z 2025-08-14T21:55:40.7780462Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7780679Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7780750Z return mod(**inputs) 2025-08-14T21:55:40.7781051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7781123Z outputs = self.roberta( 2025-08-14T21:55:40.7781418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7781495Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7781785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7781868Z layer_outputs = layer_module( 2025-08-14T21:55:40.7782112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7782193Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7782501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7782591Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7782880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7782960Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7783284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7783421Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7783723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:40.7783871Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:40.7784097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:40.7784176Z return self.act(input) 2025-08-14T21:55:40.7784211Z 2025-08-14T21:55:40.7784331Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7784547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7784617Z return mod(**inputs) 2025-08-14T21:55:40.7784913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7784987Z outputs = self.roberta( 2025-08-14T21:55:40.7785272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7785353Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7785689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7785778Z layer_outputs = layer_module( 2025-08-14T21:55:40.7786031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7786119Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7786428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7786517Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7786807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7786889Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7787211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:40.7787363Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:40.7787652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:40.7787747Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7787751Z 2025-08-14T21:55:40.7787860Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7788075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7788153Z return mod(**inputs) 2025-08-14T21:55:40.7788440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7788514Z outputs = self.roberta( 2025-08-14T21:55:40.7788810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7788889Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7789184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7789262Z layer_outputs = layer_module( 2025-08-14T21:55:40.7789503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7789595Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7789894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7789989Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7790251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7790329Z return func(*args, **kwargs) 2025-08-14T21:55:40.7790642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7790719Z self_outputs = self.self( 2025-08-14T21:55:40.7790982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7791083Z return func(*args, **kwargs) 2025-08-14T21:55:40.7791365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:40.7791597Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:40.7791601Z 2025-08-14T21:55:40.7791712Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7791926Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7792005Z return mod(**inputs) 2025-08-14T21:55:40.7792310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7792393Z outputs = self.roberta( 2025-08-14T21:55:40.7792687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7792768Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7793055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7793132Z layer_outputs = layer_module( 2025-08-14T21:55:40.7793366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7793457Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7793734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7793832Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7794090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7794165Z return func(*args, **kwargs) 2025-08-14T21:55:40.7794452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7794529Z self_outputs = self.self( 2025-08-14T21:55:40.7794794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7794868Z return func(*args, **kwargs) 2025-08-14T21:55:40.7795148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:40.7795232Z self.key(current_states) 2025-08-14T21:55:40.7795236Z 2025-08-14T21:55:40.7795345Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7795562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7795642Z return mod(**inputs) 2025-08-14T21:55:40.7795924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7796006Z outputs = self.roberta( 2025-08-14T21:55:40.7796283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7796361Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7796651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7796729Z layer_outputs = layer_module( 2025-08-14T21:55:40.7796963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7797078Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7797363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7797458Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7797713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7797804Z return func(*args, **kwargs) 2025-08-14T21:55:40.7798096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7798172Z self_outputs = self.self( 2025-08-14T21:55:40.7798440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7798515Z return func(*args, **kwargs) 2025-08-14T21:55:40.7798815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:40.7798904Z self.value(current_states) 2025-08-14T21:55:40.7798908Z 2025-08-14T21:55:40.7798998Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7799126Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7799349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7799421Z return mod(**inputs) 2025-08-14T21:55:40.7799714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7799789Z outputs = self.roberta( 2025-08-14T21:55:40.7800067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7800152Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7800437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7800516Z layer_outputs = layer_module( 2025-08-14T21:55:40.7800760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7800845Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7801135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7801222Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7801476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7801560Z return func(*args, **kwargs) 2025-08-14T21:55:40.7801841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7801923Z self_outputs = self.self( 2025-08-14T21:55:40.7802183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7802260Z return func(*args, **kwargs) 2025-08-14T21:55:40.7802549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:40.7802697Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:40.7802700Z 2025-08-14T21:55:40.7802812Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7803035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7803106Z return mod(**inputs) 2025-08-14T21:55:40.7803396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7803472Z outputs = self.roberta( 2025-08-14T21:55:40.7803754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7803865Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7804150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7804244Z layer_outputs = layer_module( 2025-08-14T21:55:40.7804493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7804577Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7804868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7804957Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7805298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7805401Z return func(*args, **kwargs) 2025-08-14T21:55:40.7805713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:40.7805863Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:40.7806172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:40.7806264Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7806269Z 2025-08-14T21:55:40.7806394Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7806611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7806683Z return mod(**inputs) 2025-08-14T21:55:40.7806981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7807057Z outputs = self.roberta( 2025-08-14T21:55:40.7807352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7807434Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7807717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7807807Z layer_outputs = layer_module( 2025-08-14T21:55:40.7808045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7808135Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7808417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7808507Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7808792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7808878Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7809198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7809337Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7809628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:40.7809725Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7809729Z 2025-08-14T21:55:40.7809839Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7810053Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7810133Z return mod(**inputs) 2025-08-14T21:55:40.7810418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7810531Z outputs = self.roberta( 2025-08-14T21:55:40.7810815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7810895Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7811184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7811280Z layer_outputs = layer_module( 2025-08-14T21:55:40.7811518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7811610Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7811907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7812004Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7812297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7812384Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7812725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7812854Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7813143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:40.7813265Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:40.7813497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:40.7813580Z return self.act(input) 2025-08-14T21:55:40.7813584Z 2025-08-14T21:55:40.7813694Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7813911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7813989Z return mod(**inputs) 2025-08-14T21:55:40.7814274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7814357Z outputs = self.roberta( 2025-08-14T21:55:40.7814644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7814723Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7815014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7815090Z layer_outputs = layer_module( 2025-08-14T21:55:40.7815327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7815417Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7815703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7815798Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7816073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7816155Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7816478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:40.7816619Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:40.7816909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:40.7816995Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7816999Z 2025-08-14T21:55:40.7817131Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7817352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7817423Z return mod(**inputs) 2025-08-14T21:55:40.7817717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7817817Z outputs = self.roberta( 2025-08-14T21:55:40.7818102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7818189Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7818474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7818550Z layer_outputs = layer_module( 2025-08-14T21:55:40.7818799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7818900Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7819190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7819292Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7819556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7819640Z return func(*args, **kwargs) 2025-08-14T21:55:40.7819920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7819995Z self_outputs = self.self( 2025-08-14T21:55:40.7820262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7820337Z return func(*args, **kwargs) 2025-08-14T21:55:40.7820631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:40.7820856Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:40.7820860Z 2025-08-14T21:55:40.7820972Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7821197Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7821267Z return mod(**inputs) 2025-08-14T21:55:40.7821561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7821634Z outputs = self.roberta( 2025-08-14T21:55:40.7821917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7822007Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7822294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7822373Z layer_outputs = layer_module( 2025-08-14T21:55:40.7822623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7822710Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7823003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7823092Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7823355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7823438Z return func(*args, **kwargs) 2025-08-14T21:55:40.7823723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7823825Z self_outputs = self.self( 2025-08-14T21:55:40.7824085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7824161Z return func(*args, **kwargs) 2025-08-14T21:55:40.7824451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:40.7824545Z self.key(current_states) 2025-08-14T21:55:40.7824549Z 2025-08-14T21:55:40.7824660Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7824884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7824955Z return mod(**inputs) 2025-08-14T21:55:40.7825258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7825333Z outputs = self.roberta( 2025-08-14T21:55:40.7825630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7825722Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7826037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7826115Z layer_outputs = layer_module( 2025-08-14T21:55:40.7826361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7826444Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7826733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7826820Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7827078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7827165Z return func(*args, **kwargs) 2025-08-14T21:55:40.7827445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7827527Z self_outputs = self.self( 2025-08-14T21:55:40.7827788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7827863Z return func(*args, **kwargs) 2025-08-14T21:55:40.7828149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:40.7828228Z self.value(current_states) 2025-08-14T21:55:40.7828232Z 2025-08-14T21:55:40.7828317Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7828436Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7828648Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7828729Z return mod(**inputs) 2025-08-14T21:55:40.7829014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7829089Z outputs = self.roberta( 2025-08-14T21:55:40.7829380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7829459Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7829742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7829828Z layer_outputs = layer_module( 2025-08-14T21:55:40.7830063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7830153Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7830432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7830542Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7830816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7830892Z return func(*args, **kwargs) 2025-08-14T21:55:40.7831207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7831282Z self_outputs = self.self( 2025-08-14T21:55:40.7831544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7831626Z return func(*args, **kwargs) 2025-08-14T21:55:40.7831906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:40.7832049Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:40.7832055Z 2025-08-14T21:55:40.7832190Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7832402Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7832480Z return mod(**inputs) 2025-08-14T21:55:40.7832791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7832868Z outputs = self.roberta( 2025-08-14T21:55:40.7833157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7833236Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7833524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7833601Z layer_outputs = layer_module( 2025-08-14T21:55:40.7833840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7833933Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7834213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7834303Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7834573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7834647Z return func(*args, **kwargs) 2025-08-14T21:55:40.7834933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:40.7835069Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:40.7835352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:40.7835450Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7835455Z 2025-08-14T21:55:40.7835566Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7835786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7835858Z return mod(**inputs) 2025-08-14T21:55:40.7836141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7836225Z outputs = self.roberta( 2025-08-14T21:55:40.7836504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7836582Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7836871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7836973Z layer_outputs = layer_module( 2025-08-14T21:55:40.7837248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7837331Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7837763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7837942Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7838224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7838308Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7838647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7838774Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7839089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:40.7839182Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7839187Z 2025-08-14T21:55:40.7839298Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7839573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7839649Z return mod(**inputs) 2025-08-14T21:55:40.7839941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7840017Z outputs = self.roberta( 2025-08-14T21:55:40.7840302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7840390Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7840678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7840760Z layer_outputs = layer_module( 2025-08-14T21:55:40.7841014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7841097Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7841404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7841494Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7841777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7841867Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7842193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7842327Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7842630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:40.7842752Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:40.7842994Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:40.7843071Z return self.act(input) 2025-08-14T21:55:40.7843075Z 2025-08-14T21:55:40.7843185Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7843418Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7843490Z return mod(**inputs) 2025-08-14T21:55:40.7843787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7843862Z outputs = self.roberta( 2025-08-14T21:55:40.7844151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7844270Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7844558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7844646Z layer_outputs = layer_module( 2025-08-14T21:55:40.7844903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7844989Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7845477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7845572Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7845851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7845944Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7846286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:40.7846439Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:40.7846753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:40.7846845Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7846851Z 2025-08-14T21:55:40.7846972Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7847198Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7847277Z return mod(**inputs) 2025-08-14T21:55:40.7847562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7847637Z outputs = self.roberta( 2025-08-14T21:55:40.7847931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7848009Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7848290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7848378Z layer_outputs = layer_module( 2025-08-14T21:55:40.7848615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7848705Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7849006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7849092Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7849358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7849438Z return func(*args, **kwargs) 2025-08-14T21:55:40.7849717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7849799Z self_outputs = self.self( 2025-08-14T21:55:40.7850056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7850139Z return func(*args, **kwargs) 2025-08-14T21:55:40.7850419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:40.7850640Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:40.7850645Z 2025-08-14T21:55:40.7850762Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7850974Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7851073Z return mod(**inputs) 2025-08-14T21:55:40.7851361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7851435Z outputs = self.roberta( 2025-08-14T21:55:40.7851724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7851821Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7852106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7852191Z layer_outputs = layer_module( 2025-08-14T21:55:40.7852433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7852524Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7852820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7852912Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7853179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7853272Z return func(*args, **kwargs) 2025-08-14T21:55:40.7853568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7853644Z self_outputs = self.self( 2025-08-14T21:55:40.7853907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7853992Z return func(*args, **kwargs) 2025-08-14T21:55:40.7854278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:40.7854358Z self.key(current_states) 2025-08-14T21:55:40.7854363Z 2025-08-14T21:55:40.7854485Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7854700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7854782Z return mod(**inputs) 2025-08-14T21:55:40.7855075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7855153Z outputs = self.roberta( 2025-08-14T21:55:40.7855444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7855527Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7855811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7855898Z layer_outputs = layer_module( 2025-08-14T21:55:40.7856139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7856236Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7856519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7856611Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7856881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7856958Z return func(*args, **kwargs) 2025-08-14T21:55:40.7857247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7857326Z self_outputs = self.self( 2025-08-14T21:55:40.7857588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7857673Z return func(*args, **kwargs) 2025-08-14T21:55:40.7857981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:40.7858060Z self.value(current_states) 2025-08-14T21:55:40.7858063Z 2025-08-14T21:55:40.7858161Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7858271Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7858516Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7858588Z return mod(**inputs) 2025-08-14T21:55:40.7858875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7858957Z outputs = self.roberta( 2025-08-14T21:55:40.7859238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7859316Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7859656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7859732Z layer_outputs = layer_module( 2025-08-14T21:55:40.7859993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7860080Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7860361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7860457Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7860723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7860804Z return func(*args, **kwargs) 2025-08-14T21:55:40.7861085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7861163Z self_outputs = self.self( 2025-08-14T21:55:40.7861429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7861502Z return func(*args, **kwargs) 2025-08-14T21:55:40.7861783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:40.7861933Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:40.7861937Z 2025-08-14T21:55:40.7862047Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7862269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7862340Z return mod(**inputs) 2025-08-14T21:55:40.7862622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7862708Z outputs = self.roberta( 2025-08-14T21:55:40.7862989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7863075Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7863354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7863431Z layer_outputs = layer_module( 2025-08-14T21:55:40.7863674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7863756Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7864037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7864130Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7864399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7864502Z return func(*args, **kwargs) 2025-08-14T21:55:40.7864787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:40.7864925Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:40.7865232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:40.7865320Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7865324Z 2025-08-14T21:55:40.7865439Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7865650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7865720Z return mod(**inputs) 2025-08-14T21:55:40.7866013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7866104Z outputs = self.roberta( 2025-08-14T21:55:40.7866389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7866475Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7866769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7866856Z layer_outputs = layer_module( 2025-08-14T21:55:40.7867091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7867174Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7867463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7867552Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7867831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7867922Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7868237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7868373Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7868663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:40.7868750Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7868754Z 2025-08-14T21:55:40.7868871Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7869082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7869161Z return mod(**inputs) 2025-08-14T21:55:40.7869445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7869520Z outputs = self.roberta( 2025-08-14T21:55:40.7869807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7869887Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7870164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7870248Z layer_outputs = layer_module( 2025-08-14T21:55:40.7870489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7870578Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7870849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7870935Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7871234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7871315Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7871630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7871777Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7872050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:40.7872175Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:40.7872397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:40.7872472Z return self.act(input) 2025-08-14T21:55:40.7872483Z 2025-08-14T21:55:40.7872610Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7872821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7872899Z return mod(**inputs) 2025-08-14T21:55:40.7873191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7873268Z outputs = self.roberta( 2025-08-14T21:55:40.7873547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7873623Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7873902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7873976Z layer_outputs = layer_module( 2025-08-14T21:55:40.7874207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7874300Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7874571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7874659Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7874936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7875014Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7875328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:40.7875466Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:40.7875761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:40.7875853Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7875860Z 2025-08-14T21:55:40.7875967Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7876181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7876251Z return mod(**inputs) 2025-08-14T21:55:40.7876527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7876606Z outputs = self.roberta( 2025-08-14T21:55:40.7876879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7876955Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7877233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7877306Z layer_outputs = layer_module( 2025-08-14T21:55:40.7877544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7877644Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7877915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7878026Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7878277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7878350Z return func(*args, **kwargs) 2025-08-14T21:55:40.7878634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7878709Z self_outputs = self.self( 2025-08-14T21:55:40.7878972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7879049Z return func(*args, **kwargs) 2025-08-14T21:55:40.7879342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:40.7879592Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:40.7879598Z 2025-08-14T21:55:40.7879711Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7879932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7880003Z return mod(**inputs) 2025-08-14T21:55:40.7880292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7880374Z outputs = self.roberta( 2025-08-14T21:55:40.7880656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7880736Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7881027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7881104Z layer_outputs = layer_module( 2025-08-14T21:55:40.7881348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7881432Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7881715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7881810Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7882068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7882150Z return func(*args, **kwargs) 2025-08-14T21:55:40.7882432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7882508Z self_outputs = self.self( 2025-08-14T21:55:40.7882775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7882850Z return func(*args, **kwargs) 2025-08-14T21:55:40.7883136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:40.7883220Z self.key(current_states) 2025-08-14T21:55:40.7883224Z 2025-08-14T21:55:40.7883332Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7883553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7883624Z return mod(**inputs) 2025-08-14T21:55:40.7883913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7884012Z outputs = self.roberta( 2025-08-14T21:55:40.7884305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7884383Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7884682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7884786Z layer_outputs = layer_module( 2025-08-14T21:55:40.7885030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7885113Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7885492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7885594Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7885874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7885959Z return func(*args, **kwargs) 2025-08-14T21:55:40.7886247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7886343Z self_outputs = self.self( 2025-08-14T21:55:40.7886615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7886690Z return func(*args, **kwargs) 2025-08-14T21:55:40.7886978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:40.7887063Z self.value(current_states) 2025-08-14T21:55:40.7887067Z 2025-08-14T21:55:40.7887155Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7887270Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7887485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7887560Z return mod(**inputs) 2025-08-14T21:55:40.7887855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7887932Z outputs = self.roberta( 2025-08-14T21:55:40.7888215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7888306Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7888587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7888671Z layer_outputs = layer_module( 2025-08-14T21:55:40.7888907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7888990Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7889278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7889365Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7889631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7889707Z return func(*args, **kwargs) 2025-08-14T21:55:40.7889991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7890073Z self_outputs = self.self( 2025-08-14T21:55:40.7890334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7890408Z return func(*args, **kwargs) 2025-08-14T21:55:40.7890702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:40.7890868Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:40.7890872Z 2025-08-14T21:55:40.7890991Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7891205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7891278Z return mod(**inputs) 2025-08-14T21:55:40.7891657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7891730Z outputs = self.roberta( 2025-08-14T21:55:40.7892017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7892096Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7892376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7892460Z layer_outputs = layer_module( 2025-08-14T21:55:40.7892721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7892810Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7893134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7893222Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7893490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7893565Z return func(*args, **kwargs) 2025-08-14T21:55:40.7893849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:40.7893994Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:40.7894280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:40.7894381Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7894388Z 2025-08-14T21:55:40.7894499Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7894712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7894791Z return mod(**inputs) 2025-08-14T21:55:40.7895076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7895147Z outputs = self.roberta( 2025-08-14T21:55:40.7895434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7895511Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7895798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7895878Z layer_outputs = layer_module( 2025-08-14T21:55:40.7896116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7896207Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7896508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7896598Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7896886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7896967Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7897294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7897421Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7897721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:40.7897817Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7897821Z 2025-08-14T21:55:40.7897931Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7898150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7898237Z return mod(**inputs) 2025-08-14T21:55:40.7898527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7898609Z outputs = self.roberta( 2025-08-14T21:55:40.7898897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7898974Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7899287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7899365Z layer_outputs = layer_module( 2025-08-14T21:55:40.7899609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7899709Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7899991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7900087Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7900363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7900453Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7900768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7900897Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7901185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:40.7901307Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:40.7901537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:40.7901621Z return self.act(input) 2025-08-14T21:55:40.7901625Z 2025-08-14T21:55:40.7901735Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7901961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7902034Z return mod(**inputs) 2025-08-14T21:55:40.7902318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7902399Z outputs = self.roberta( 2025-08-14T21:55:40.7902682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7902767Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7903048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7903126Z layer_outputs = layer_module( 2025-08-14T21:55:40.7903369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7903452Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7903731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7903830Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7904106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7904221Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7904535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:40.7904677Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:40.7904986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:40.7905074Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7905078Z 2025-08-14T21:55:40.7905197Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7905421Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7905492Z return mod(**inputs) 2025-08-14T21:55:40.7905781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7905872Z outputs = self.roberta( 2025-08-14T21:55:40.7906154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7906241Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7906536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7906622Z layer_outputs = layer_module( 2025-08-14T21:55:40.7906859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7906942Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7907229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7907318Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7907591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7907667Z return func(*args, **kwargs) 2025-08-14T21:55:40.7907946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7908030Z self_outputs = self.self( 2025-08-14T21:55:40.7908290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7908365Z return func(*args, **kwargs) 2025-08-14T21:55:40.7908650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 324, in forward 2025-08-14T21:55:40.7908871Z self.query(hidden_states).view(bsz, -1, self.num_attention_heads, self.attention_head_size).transpose(1, 2) 2025-08-14T21:55:40.7908875Z 2025-08-14T21:55:40.7908991Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7909208Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7909278Z return mod(**inputs) 2025-08-14T21:55:40.7909568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7909645Z outputs = self.roberta( 2025-08-14T21:55:40.7909925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7910011Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7910288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7910371Z layer_outputs = layer_module( 2025-08-14T21:55:40.7910608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7910691Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7911017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7911104Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7911373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7911468Z return func(*args, **kwargs) 2025-08-14T21:55:40.7911756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7911839Z self_outputs = self.self( 2025-08-14T21:55:40.7912101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7912176Z return func(*args, **kwargs) 2025-08-14T21:55:40.7912465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 352, in forward 2025-08-14T21:55:40.7912563Z self.key(current_states) 2025-08-14T21:55:40.7912567Z 2025-08-14T21:55:40.7912685Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7912897Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7912986Z return mod(**inputs) 2025-08-14T21:55:40.7913284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7913357Z outputs = self.roberta( 2025-08-14T21:55:40.7913649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7913727Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7914006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7914089Z layer_outputs = layer_module( 2025-08-14T21:55:40.7914327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7914411Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7914702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7914789Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7915056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7915130Z return func(*args, **kwargs) 2025-08-14T21:55:40.7915409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7915492Z self_outputs = self.self( 2025-08-14T21:55:40.7915750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7915826Z return func(*args, **kwargs) 2025-08-14T21:55:40.7916115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 357, in forward 2025-08-14T21:55:40.7916191Z self.value(current_states) 2025-08-14T21:55:40.7916198Z 2025-08-14T21:55:40.7916295Z cudagraph partition due to non gpu ops 2025-08-14T21:55:40.7916405Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7916619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7916698Z return mod(**inputs) 2025-08-14T21:55:40.7916981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7917062Z outputs = self.roberta( 2025-08-14T21:55:40.7917344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7917449Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7917739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7917816Z layer_outputs = layer_module( 2025-08-14T21:55:40.7918054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7918165Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7918449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7918543Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7918800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7918874Z return func(*args, **kwargs) 2025-08-14T21:55:40.7919179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 466, in forward 2025-08-14T21:55:40.7919256Z self_outputs = self.self( 2025-08-14T21:55:40.7919514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7919612Z return func(*args, **kwargs) 2025-08-14T21:55:40.7919895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 388, in forward 2025-08-14T21:55:40.7920047Z attn_output = torch.nn.functional.scaled_dot_product_attention( 2025-08-14T21:55:40.7920051Z 2025-08-14T21:55:40.7920162Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7920376Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7920454Z return mod(**inputs) 2025-08-14T21:55:40.7920739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7920822Z outputs = self.roberta( 2025-08-14T21:55:40.7921103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7921184Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7921474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7921551Z layer_outputs = layer_module( 2025-08-14T21:55:40.7921787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7921881Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7922163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 539, in forward 2025-08-14T21:55:40.7922258Z self_attention_outputs = self.attention( 2025-08-14T21:55:40.7922522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/utils/deprecation.py", line 172, in wrapped_func 2025-08-14T21:55:40.7922597Z return func(*args, **kwargs) 2025-08-14T21:55:40.7922888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 476, in forward 2025-08-14T21:55:40.7923028Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:55:40.7923330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 412, in forward 2025-08-14T21:55:40.7923419Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7923423Z 2025-08-14T21:55:40.7923532Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7923760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7923830Z return mod(**inputs) 2025-08-14T21:55:40.7924117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7924223Z outputs = self.roberta( 2025-08-14T21:55:40.7924505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7924589Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7924888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7924964Z layer_outputs = layer_module( 2025-08-14T21:55:40.7925291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7925384Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7925681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7925780Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7926091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7926185Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7926524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7926654Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7926953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 492, in forward 2025-08-14T21:55:40.7927041Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7927046Z 2025-08-14T21:55:40.7927162Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7927393Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7927466Z return mod(**inputs) 2025-08-14T21:55:40.7927757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7927830Z outputs = self.roberta( 2025-08-14T21:55:40.7928111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7928200Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7928480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7928566Z layer_outputs = layer_module( 2025-08-14T21:55:40.7928801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7928884Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7929181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7929272Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7929558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7929642Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7929956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 577, in feed_forward_chunk 2025-08-14T21:55:40.7930090Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:55:40.7930381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 493, in forward 2025-08-14T21:55:40.7930502Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:55:40.7930734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:55:40.7930810Z return self.act(input) 2025-08-14T21:55:40.7930834Z 2025-08-14T21:55:40.7930953Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7931168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7931238Z return mod(**inputs) 2025-08-14T21:55:40.7931533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1516, in forward 2025-08-14T21:55:40.7931626Z outputs = self.roberta( 2025-08-14T21:55:40.7931911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 890, in forward 2025-08-14T21:55:40.7931989Z encoder_outputs = self.encoder( 2025-08-14T21:55:40.7932267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 631, in forward 2025-08-14T21:55:40.7932351Z layer_outputs = layer_module( 2025-08-14T21:55:40.7932603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:55:40.7932690Z return super().__call__(*args, **kwargs) 2025-08-14T21:55:40.7933001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 569, in forward 2025-08-14T21:55:40.7933094Z layer_output = apply_chunking_to_forward( 2025-08-14T21:55:40.7933376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:55:40.7933458Z return forward_fn(*input_tensors) 2025-08-14T21:55:40.7933773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 578, in feed_forward_chunk 2025-08-14T21:55:40.7933919Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:55:40.7934216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 506, in forward 2025-08-14T21:55:40.7934315Z hidden_states = self.dense(hidden_states) 2025-08-14T21:55:40.7934319Z 2025-08-14T21:55:40.7934427Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7934643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7934723Z return mod(**inputs) 2025-08-14T21:55:40.7935007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1530, in forward 2025-08-14T21:55:40.7935096Z logits = self.qa_outputs(sequence_output) 2025-08-14T21:55:40.7935108Z 2025-08-14T21:55:40.7935218Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7935428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7935507Z return mod(**inputs) 2025-08-14T21:55:40.7935791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1548, in forward 2025-08-14T21:55:40.7935907Z start_loss = loss_fct(start_logits, start_positions) 2025-08-14T21:55:40.7935911Z 2025-08-14T21:55:40.7936026Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:55:40.7936239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:55:40.7936320Z return mod(**inputs) 2025-08-14T21:55:40.7936604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/roberta/modeling_roberta.py", line 1549, in forward 2025-08-14T21:55:40.7936704Z end_loss = loss_fct(end_logits, end_positions) 2025-08-14T21:55:40.7936708Z 2025-08-14T21:55:48.4713338Z Compilation time (from dynamo_timed): 13.884143929 2025-08-14T21:55:48.4713672Z pass 2025-08-14T21:55:48.4713976Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:55:48.4714792Z TIMING: _recursive_pre_grad_passes:0.00728 _recursive_joint_graph_passes:0.65646 _recursive_post_grad_passes:0.08766 async_compile.wait:0.00268 code_gen:6.62271 inductor_compile:7.79849 backend_compile:10.87517 gc:0.00141 entire_frame_compile:13.88414 total_wall_time:13.88414 2025-08-14T21:55:48.4715983Z STATS: call_* op count: 303 | FakeTensorMode.__torch_dispatch__:12465 | FakeTensor.__torch_dispatch__:4777 | ProxyTorchDispatchMode.__torch_dispatch__:4566 2025-08-14T21:55:48.4716551Z Dynamo produced 1 graphs covering 303 ops with 0 graph breaks (0 unique) 2025-08-14T21:55:53.7351907Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:55:53.7353444Z from pkg_resources import resource_filename 2025-08-14T21:55:54.2960064Z 2025-08-14T21:55:55.4505422Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:55:55.4506504Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:55:55.4514781Z cpu eval T5ForConditionalGeneration 2025-08-14T21:55:56.8124602Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:55:57.2161377Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:55:57.6363461Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:56:06.9580031Z cudagraph partition due to non gpu ops 2025-08-14T21:56:06.9584218Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9584757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9585129Z return mod(**inputs) 2025-08-14T21:56:06.9585554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:06.9586022Z decoder_outputs = self.decoder( 2025-08-14T21:56:06.9586425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9586859Z layer_outputs = layer_module( 2025-08-14T21:56:06.9587229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9587607Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9587982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9588382Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9588805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9589274Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9589689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 546, in forward 2025-08-14T21:56:06.9590068Z position_bias = position_bias + causal_mask 2025-08-14T21:56:06.9590222Z 2025-08-14T21:56:06.9590330Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9590694Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9591018Z return mod(**inputs) 2025-08-14T21:56:06.9591352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:06.9591717Z decoder_outputs = self.decoder( 2025-08-14T21:56:06.9592118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9592513Z layer_outputs = layer_module( 2025-08-14T21:56:06.9592862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9593596Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9593989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9594367Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9594751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-14T21:56:06.9595227Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:56:06.9595618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:06.9595995Z return self.weight * hidden_states 2025-08-14T21:56:06.9596136Z 2025-08-14T21:56:06.9596244Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9596613Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9596943Z return mod(**inputs) 2025-08-14T21:56:06.9597344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:06.9597715Z decoder_outputs = self.decoder( 2025-08-14T21:56:06.9598117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9598520Z layer_outputs = layer_module( 2025-08-14T21:56:06.9598901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9599267Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9599641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9600011Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9600395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9600767Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9601126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:56:06.9601504Z query_states = self.q(hidden_states) 2025-08-14T21:56:06.9601645Z 2025-08-14T21:56:06.9601750Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9602130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9602449Z return mod(**inputs) 2025-08-14T21:56:06.9602798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:06.9603181Z decoder_outputs = self.decoder( 2025-08-14T21:56:06.9603549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9603924Z layer_outputs = layer_module( 2025-08-14T21:56:06.9604276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9604640Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9605007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9605605Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9606020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9606423Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9606826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:56:06.9607215Z key_states = self.k(current_states) 2025-08-14T21:56:06.9607347Z 2025-08-14T21:56:06.9607461Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9607810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9608176Z return mod(**inputs) 2025-08-14T21:56:06.9608557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:06.9608977Z decoder_outputs = self.decoder( 2025-08-14T21:56:06.9609364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9609737Z layer_outputs = layer_module( 2025-08-14T21:56:06.9610083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9610443Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9610798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9611161Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9611545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9611907Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9612284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:56:06.9612707Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:56:06.9612884Z 2025-08-14T21:56:06.9612992Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9613335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9613665Z return mod(**inputs) 2025-08-14T21:56:06.9614009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:06.9614362Z decoder_outputs = self.decoder( 2025-08-14T21:56:06.9614719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9615077Z layer_outputs = layer_module( 2025-08-14T21:56:06.9615413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9615759Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9616119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9616482Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9616833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9617208Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9617576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:06.9618022Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:06.9618232Z 2025-08-14T21:56:06.9618335Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9618727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9619048Z return mod(**inputs) 2025-08-14T21:56:06.9619380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:06.9619735Z decoder_outputs = self.decoder( 2025-08-14T21:56:06.9620093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9620458Z layer_outputs = layer_module( 2025-08-14T21:56:06.9620800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9621159Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9621526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9621925Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9622287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9622664Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9623061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:56:06.9623422Z value_states = self.v(current_states) 2025-08-14T21:56:06.9623567Z 2025-08-14T21:56:06.9623671Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9624028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9624354Z return mod(**inputs) 2025-08-14T21:56:06.9624691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:06.9625085Z decoder_outputs = self.decoder( 2025-08-14T21:56:06.9625455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9625821Z layer_outputs = layer_module( 2025-08-14T21:56:06.9626172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9626537Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9626905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9627278Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9627671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9628065Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9628471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:06.9628909Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:06.9629078Z 2025-08-14T21:56:06.9629182Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9629542Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9629862Z return mod(**inputs) 2025-08-14T21:56:06.9630209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:06.9630590Z decoder_outputs = self.decoder( 2025-08-14T21:56:06.9630963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9631326Z layer_outputs = layer_module( 2025-08-14T21:56:06.9631673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9632043Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9632404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9632778Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9633151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9633520Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9633887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:06.9634296Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:06.9634461Z 2025-08-14T21:56:06.9634561Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9634911Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9635243Z return mod(**inputs) 2025-08-14T21:56:06.9635590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:06.9635963Z decoder_outputs = self.decoder( 2025-08-14T21:56:06.9636318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9636716Z layer_outputs = layer_module( 2025-08-14T21:56:06.9637068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9637434Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9638201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9638592Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9638966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9639434Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9639798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:56:06.9640235Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:06.9640402Z 2025-08-14T21:56:06.9640514Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9640879Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9641212Z return mod(**inputs) 2025-08-14T21:56:06.9641601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:06.9642004Z decoder_outputs = self.decoder( 2025-08-14T21:56:06.9642386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9642780Z layer_outputs = layer_module( 2025-08-14T21:56:06.9643146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9643528Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9643930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9644331Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9644728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9645120Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9645847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:56:06.9646535Z attn_output = self.o(attn_output) 2025-08-14T21:56:06.9646757Z 2025-08-14T21:56:06.9646910Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9647468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9647793Z return mod(**inputs) 2025-08-14T21:56:06.9648132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:06.9648489Z decoder_outputs = self.decoder( 2025-08-14T21:56:06.9649227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9649588Z layer_outputs = layer_module( 2025-08-14T21:56:06.9649922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9650265Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9650626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:06.9650998Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:06.9651444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:06.9651816Z attention_output = self.EncDecAttention( 2025-08-14T21:56:06.9652195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:56:06.9652589Z query_states = self.q(hidden_states) 2025-08-14T21:56:06.9652722Z 2025-08-14T21:56:06.9652826Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9653176Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9653495Z return mod(**inputs) 2025-08-14T21:56:06.9653836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9654203Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9654660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9655032Z layer_outputs = layer_module( 2025-08-14T21:56:06.9655372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9655752Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9656125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9656519Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9656870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9657246Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9657620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:56:06.9657978Z query_states = self.q(hidden_states) 2025-08-14T21:56:06.9658122Z 2025-08-14T21:56:06.9658225Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9658582Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9658907Z return mod(**inputs) 2025-08-14T21:56:06.9659244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9659624Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9659983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9660354Z layer_outputs = layer_module( 2025-08-14T21:56:06.9660690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9661057Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9661430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9661801Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9662172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9662549Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9662921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:56:06.9663284Z key_states = self.k(current_states) 2025-08-14T21:56:06.9663422Z 2025-08-14T21:56:06.9663529Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9663887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9664201Z return mod(**inputs) 2025-08-14T21:56:06.9664539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9664940Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9665309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9665677Z layer_outputs = layer_module( 2025-08-14T21:56:06.9666025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9666405Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9666770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9667143Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9667517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9667896Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9668280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:56:06.9668703Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:56:06.9668889Z 2025-08-14T21:56:06.9669005Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9669390Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9669725Z return mod(**inputs) 2025-08-14T21:56:06.9670076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9670449Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9670805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9671178Z layer_outputs = layer_module( 2025-08-14T21:56:06.9671694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9672068Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9672481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9672863Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9673244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9673625Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9674005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:06.9674527Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:06.9674737Z 2025-08-14T21:56:06.9674860Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9675200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9675516Z return mod(**inputs) 2025-08-14T21:56:06.9675853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9676215Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9676560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9676931Z layer_outputs = layer_module( 2025-08-14T21:56:06.9677268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9677608Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9677965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9678329Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9678686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9679071Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9679434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:06.9679867Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:06.9680086Z 2025-08-14T21:56:06.9680194Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9680532Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9680846Z return mod(**inputs) 2025-08-14T21:56:06.9681204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9681561Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9681915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9682303Z layer_outputs = layer_module( 2025-08-14T21:56:06.9682646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9683001Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9683386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9683769Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9684132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9684505Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9684873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:06.9685442Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:06.9685809Z 2025-08-14T21:56:06.9685975Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9686571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9687074Z return mod(**inputs) 2025-08-14T21:56:06.9687504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9687863Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9688214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9688568Z layer_outputs = layer_module( 2025-08-14T21:56:06.9688894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9689242Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9689601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9689971Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9690319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9690687Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9691051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:56:06.9691411Z value_states = self.v(current_states) 2025-08-14T21:56:06.9691545Z 2025-08-14T21:56:06.9691645Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9691980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9692286Z return mod(**inputs) 2025-08-14T21:56:06.9692604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9693001Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9693346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9693693Z layer_outputs = layer_module( 2025-08-14T21:56:06.9694013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9694397Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9694754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9695106Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9695458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9695815Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9696162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:06.9696560Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:06.9696721Z 2025-08-14T21:56:06.9696818Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9697175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9697478Z return mod(**inputs) 2025-08-14T21:56:06.9697809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9698175Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9698532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9698877Z layer_outputs = layer_module( 2025-08-14T21:56:06.9699202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9699549Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9699889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9700252Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9700611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9700982Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9701324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:06.9701705Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:06.9701855Z 2025-08-14T21:56:06.9701962Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9702299Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9702595Z return mod(**inputs) 2025-08-14T21:56:06.9702928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9703279Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9703615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9703985Z layer_outputs = layer_module( 2025-08-14T21:56:06.9704315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9704653Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9704991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9705346Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9705699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9706066Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9706420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:56:06.9706797Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:06.9706951Z 2025-08-14T21:56:06.9707060Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9707413Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9707771Z return mod(**inputs) 2025-08-14T21:56:06.9708113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9708490Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9708901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9709247Z layer_outputs = layer_module( 2025-08-14T21:56:06.9709590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9709934Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9710338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9710701Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9711063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9711427Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9711796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:56:06.9712160Z attn_output = self.o(attn_output) 2025-08-14T21:56:06.9712285Z 2025-08-14T21:56:06.9712366Z cudagraph partition due to non gpu ops 2025-08-14T21:56:06.9712599Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9712959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9713275Z return mod(**inputs) 2025-08-14T21:56:06.9713605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9713967Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9714369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9714720Z layer_outputs = layer_module( 2025-08-14T21:56:06.9715056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9715409Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9715772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:06.9716145Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:06.9716524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:56:06.9716906Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:56:06.9717285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:06.9717642Z return self.weight * hidden_states 2025-08-14T21:56:06.9717780Z 2025-08-14T21:56:06.9717881Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9718229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9718535Z return mod(**inputs) 2025-08-14T21:56:06.9718871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9719240Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9719613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9719965Z layer_outputs = layer_module( 2025-08-14T21:56:06.9720303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9720673Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9721024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:06.9721402Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:06.9721774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:06.9722176Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:06.9722563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:56:06.9722948Z hidden_states = self.wi(hidden_states) 2025-08-14T21:56:06.9723080Z 2025-08-14T21:56:06.9723187Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9723529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9723855Z return mod(**inputs) 2025-08-14T21:56:06.9724192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9724560Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9724895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9725370Z layer_outputs = layer_module( 2025-08-14T21:56:06.9725710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9726053Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9726410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:06.9726789Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:06.9727172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:06.9727579Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:06.9727993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:56:06.9728347Z hidden_states = self.act(hidden_states) 2025-08-14T21:56:06.9728474Z 2025-08-14T21:56:06.9728581Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9728913Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9729217Z return mod(**inputs) 2025-08-14T21:56:06.9729548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9729907Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9730255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9730625Z layer_outputs = layer_module( 2025-08-14T21:56:06.9730956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9731297Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9731653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:06.9732021Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:06.9732386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:06.9732775Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:06.9733196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:56:06.9733554Z hidden_states = self.wo(hidden_states) 2025-08-14T21:56:06.9733681Z 2025-08-14T21:56:06.9733760Z cudagraph partition due to non gpu ops 2025-08-14T21:56:06.9734006Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9734345Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9734654Z return mod(**inputs) 2025-08-14T21:56:06.9734978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9735328Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9735670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9736012Z layer_outputs = layer_module( 2025-08-14T21:56:06.9736360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9736705Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9737068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9737419Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9737915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-14T21:56:06.9738305Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:56:06.9738679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:06.9739022Z return self.weight * hidden_states 2025-08-14T21:56:06.9739156Z 2025-08-14T21:56:06.9739255Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9739598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9739896Z return mod(**inputs) 2025-08-14T21:56:06.9740225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9740577Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9740921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9741263Z layer_outputs = layer_module( 2025-08-14T21:56:06.9741587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9741929Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9742269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9742623Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9742977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9743339Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9743693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:56:06.9744054Z query_states = self.q(hidden_states) 2025-08-14T21:56:06.9744188Z 2025-08-14T21:56:06.9744291Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9744617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9744914Z return mod(**inputs) 2025-08-14T21:56:06.9745237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9745583Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9745920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9746350Z layer_outputs = layer_module( 2025-08-14T21:56:06.9746670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9747052Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9747428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9747786Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9748137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9748483Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9748845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:56:06.9749186Z key_states = self.k(current_states) 2025-08-14T21:56:06.9749309Z 2025-08-14T21:56:06.9749440Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9749769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9750067Z return mod(**inputs) 2025-08-14T21:56:06.9750409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9750745Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9751080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9751424Z layer_outputs = layer_module( 2025-08-14T21:56:06.9751742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9752066Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9752407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9752751Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9753084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9753432Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9753775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:56:06.9754169Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:56:06.9754335Z 2025-08-14T21:56:06.9754430Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9754759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9755059Z return mod(**inputs) 2025-08-14T21:56:06.9755386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9755731Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9756076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9756425Z layer_outputs = layer_module( 2025-08-14T21:56:06.9756744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9757106Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9757473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9757846Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9758226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9758620Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9758989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:06.9759437Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:06.9759651Z 2025-08-14T21:56:06.9759755Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9760140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9760451Z return mod(**inputs) 2025-08-14T21:56:06.9760777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9761138Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9761485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9761840Z layer_outputs = layer_module( 2025-08-14T21:56:06.9762164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9762529Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9762888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9763261Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9763624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9763992Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9764351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:06.9764782Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:06.9764989Z 2025-08-14T21:56:06.9765089Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9765562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9765895Z return mod(**inputs) 2025-08-14T21:56:06.9766233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9766605Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9766974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9767345Z layer_outputs = layer_module( 2025-08-14T21:56:06.9767678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9768035Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9768395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9768753Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9769124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9769492Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9769844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:06.9770274Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:06.9770480Z 2025-08-14T21:56:06.9770582Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9770930Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9771242Z return mod(**inputs) 2025-08-14T21:56:06.9771579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9771942Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9772289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9772682Z layer_outputs = layer_module( 2025-08-14T21:56:06.9773014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9773371Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9773742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9774109Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9774468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9774831Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9775179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:56:06.9775537Z value_states = self.v(current_states) 2025-08-14T21:56:06.9775669Z 2025-08-14T21:56:06.9775793Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9776140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9776456Z return mod(**inputs) 2025-08-14T21:56:06.9776808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9777171Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9777512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9777924Z layer_outputs = layer_module( 2025-08-14T21:56:06.9778260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9778599Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9778956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9779321Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9779680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9780037Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9780394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:06.9780789Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:06.9780938Z 2025-08-14T21:56:06.9781042Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9781374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9781683Z return mod(**inputs) 2025-08-14T21:56:06.9782008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9782353Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9782699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9783051Z layer_outputs = layer_module( 2025-08-14T21:56:06.9783383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9783719Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9784066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9784418Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9784756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9785113Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9785466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:06.9785871Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:06.9786022Z 2025-08-14T21:56:06.9786118Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9786457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9786784Z return mod(**inputs) 2025-08-14T21:56:06.9787110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9787455Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9787795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9788143Z layer_outputs = layer_module( 2025-08-14T21:56:06.9788459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9788804Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9789172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9789529Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9789907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9790267Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9790612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:56:06.9790981Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:06.9791141Z 2025-08-14T21:56:06.9791238Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9791575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9791879Z return mod(**inputs) 2025-08-14T21:56:06.9792199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9792548Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9792885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9793235Z layer_outputs = layer_module( 2025-08-14T21:56:06.9793555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9793898Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9794244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9794587Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9794936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9795295Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9795642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:56:06.9795992Z attn_output = self.o(attn_output) 2025-08-14T21:56:06.9796120Z 2025-08-14T21:56:06.9796220Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9796562Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9796905Z return mod(**inputs) 2025-08-14T21:56:06.9797234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9797587Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9797936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9798278Z layer_outputs = layer_module( 2025-08-14T21:56:06.9798633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9798978Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9799321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:06.9799716Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:06.9800083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:56:06.9800458Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:56:06.9800814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:06.9801165Z return self.weight * hidden_states 2025-08-14T21:56:06.9801291Z 2025-08-14T21:56:06.9801395Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9801750Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9802051Z return mod(**inputs) 2025-08-14T21:56:06.9802376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9802753Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9803104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9803467Z layer_outputs = layer_module( 2025-08-14T21:56:06.9803805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9804156Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9804510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:06.9804887Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:06.9805408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:06.9806068Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:06.9806726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:56:06.9807336Z hidden_states = self.wi(hidden_states) 2025-08-14T21:56:06.9807538Z 2025-08-14T21:56:06.9807653Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9807993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9808304Z return mod(**inputs) 2025-08-14T21:56:06.9808630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9808980Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9809322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9809672Z layer_outputs = layer_module( 2025-08-14T21:56:06.9810007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9810337Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9810677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:06.9811033Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:06.9811385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:06.9811759Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:06.9812136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:56:06.9812485Z hidden_states = self.act(hidden_states) 2025-08-14T21:56:06.9812669Z 2025-08-14T21:56:06.9812768Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9813100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9813398Z return mod(**inputs) 2025-08-14T21:56:06.9813722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9814092Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9814452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9814814Z layer_outputs = layer_module( 2025-08-14T21:56:06.9815138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9815486Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9815877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:06.9816245Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:06.9816602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:06.9817029Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:06.9817419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:56:06.9817776Z hidden_states = self.wo(hidden_states) 2025-08-14T21:56:06.9817902Z 2025-08-14T21:56:06.9817981Z cudagraph partition due to non gpu ops 2025-08-14T21:56:06.9818209Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9818547Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9818851Z return mod(**inputs) 2025-08-14T21:56:06.9819183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9819542Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9819893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9820245Z layer_outputs = layer_module( 2025-08-14T21:56:06.9820575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9820921Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9821263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9821620Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9821971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-14T21:56:06.9822348Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:56:06.9822719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:06.9823074Z return self.weight * hidden_states 2025-08-14T21:56:06.9823200Z 2025-08-14T21:56:06.9823306Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9823647Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9823945Z return mod(**inputs) 2025-08-14T21:56:06.9824269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9824623Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9824959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9825310Z layer_outputs = layer_module( 2025-08-14T21:56:06.9825635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9826002Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9826342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9826715Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9827064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9827414Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9827779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:56:06.9828142Z query_states = self.q(hidden_states) 2025-08-14T21:56:06.9828270Z 2025-08-14T21:56:06.9828377Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9828748Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9829058Z return mod(**inputs) 2025-08-14T21:56:06.9829385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9829754Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9830099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9830449Z layer_outputs = layer_module( 2025-08-14T21:56:06.9830773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9831102Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9831451Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9831803Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9832156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9832521Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9832872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:56:06.9833225Z key_states = self.k(current_states) 2025-08-14T21:56:06.9833350Z 2025-08-14T21:56:06.9833448Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9833783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9834087Z return mod(**inputs) 2025-08-14T21:56:06.9834421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9834763Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9835105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9835454Z layer_outputs = layer_module( 2025-08-14T21:56:06.9835769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9836110Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9836463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9836811Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9837152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9837504Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9838033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:56:06.9838449Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:56:06.9838628Z 2025-08-14T21:56:06.9838791Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9839141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9839456Z return mod(**inputs) 2025-08-14T21:56:06.9839788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9840186Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9840540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9840898Z layer_outputs = layer_module( 2025-08-14T21:56:06.9841223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9841588Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9841983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9842416Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9842812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9843249Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9843627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:06.9844045Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:06.9844249Z 2025-08-14T21:56:06.9844350Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9844703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9845025Z return mod(**inputs) 2025-08-14T21:56:06.9845567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9846184Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9846758Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9847249Z layer_outputs = layer_module( 2025-08-14T21:56:06.9847592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9847952Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9848324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9848694Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9849066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9849442Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9849812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:06.9850275Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:06.9850492Z 2025-08-14T21:56:06.9850605Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9850953Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9851264Z return mod(**inputs) 2025-08-14T21:56:06.9851602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9851976Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9852314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9852668Z layer_outputs = layer_module( 2025-08-14T21:56:06.9852998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9853404Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9853749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9854108Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9854462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9854844Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9855196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:06.9855614Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:06.9855808Z 2025-08-14T21:56:06.9855917Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9856261Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9856604Z return mod(**inputs) 2025-08-14T21:56:06.9856944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9857304Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9857671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9858044Z layer_outputs = layer_module( 2025-08-14T21:56:06.9858391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9858751Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9859115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9859497Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9859870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9860218Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9860576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:56:06.9860930Z value_states = self.v(current_states) 2025-08-14T21:56:06.9861060Z 2025-08-14T21:56:06.9861165Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9861498Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9861808Z return mod(**inputs) 2025-08-14T21:56:06.9862134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9862483Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9862830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9863181Z layer_outputs = layer_module( 2025-08-14T21:56:06.9863513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9863852Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9864201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9864566Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9864917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9865284Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9865651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:06.9866045Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:06.9866211Z 2025-08-14T21:56:06.9866310Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9866681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9866991Z return mod(**inputs) 2025-08-14T21:56:06.9867331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9867707Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9868068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9868431Z layer_outputs = layer_module( 2025-08-14T21:56:06.9868778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9869145Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9869518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9869908Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9870287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9870664Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9871063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:06.9871462Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:06.9871630Z 2025-08-14T21:56:06.9871733Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9872092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9872433Z return mod(**inputs) 2025-08-14T21:56:06.9872773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9873148Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9873514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9873880Z layer_outputs = layer_module( 2025-08-14T21:56:06.9874219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9874582Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9874953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9875321Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9875690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9876061Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9876429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:56:06.9876826Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:06.9876994Z 2025-08-14T21:56:06.9877095Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9877457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9877777Z return mod(**inputs) 2025-08-14T21:56:06.9878128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9878500Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9878861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9879223Z layer_outputs = layer_module( 2025-08-14T21:56:06.9879568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9879928Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9880316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9880696Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9881045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9881421Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9881764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:56:06.9882118Z attn_output = self.o(attn_output) 2025-08-14T21:56:06.9882242Z 2025-08-14T21:56:06.9882332Z cudagraph partition due to non gpu ops 2025-08-14T21:56:06.9882564Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9882905Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9883214Z return mod(**inputs) 2025-08-14T21:56:06.9883565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9883921Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9884293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9884653Z layer_outputs = layer_module( 2025-08-14T21:56:06.9884987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9885507Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9886125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:06.9886786Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:06.9887250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:56:06.9887654Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:56:06.9888040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:06.9888405Z return self.weight * hidden_states 2025-08-14T21:56:06.9888533Z 2025-08-14T21:56:06.9888635Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9888983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9889301Z return mod(**inputs) 2025-08-14T21:56:06.9889635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9889992Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9890337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9890688Z layer_outputs = layer_module( 2025-08-14T21:56:06.9891009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9891353Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9891712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:06.9892088Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:06.9892447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:06.9892845Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:06.9893242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:56:06.9893602Z hidden_states = self.wi(hidden_states) 2025-08-14T21:56:06.9893735Z 2025-08-14T21:56:06.9893832Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9894225Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9894528Z return mod(**inputs) 2025-08-14T21:56:06.9894846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9895201Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9895585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9895927Z layer_outputs = layer_module( 2025-08-14T21:56:06.9896257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9896600Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9896951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:06.9897314Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:06.9897709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:06.9898115Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:06.9898535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:56:06.9898901Z hidden_states = self.act(hidden_states) 2025-08-14T21:56:06.9899049Z 2025-08-14T21:56:06.9899147Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9899485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9899785Z return mod(**inputs) 2025-08-14T21:56:06.9900114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9900478Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9900832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9901183Z layer_outputs = layer_module( 2025-08-14T21:56:06.9901516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9901866Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9902235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:06.9902679Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:06.9903045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:06.9903441Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:06.9903825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:56:06.9904189Z hidden_states = self.wo(hidden_states) 2025-08-14T21:56:06.9904320Z 2025-08-14T21:56:06.9904408Z cudagraph partition due to non gpu ops 2025-08-14T21:56:06.9904643Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9904982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9905293Z return mod(**inputs) 2025-08-14T21:56:06.9905624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9905976Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9906327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9906679Z layer_outputs = layer_module( 2025-08-14T21:56:06.9907010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9907351Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9907742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9908104Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9908454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-14T21:56:06.9908862Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:56:06.9909246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:06.9909603Z return self.weight * hidden_states 2025-08-14T21:56:06.9909731Z 2025-08-14T21:56:06.9909829Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9910178Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9910492Z return mod(**inputs) 2025-08-14T21:56:06.9910836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9911196Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9911566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9911923Z layer_outputs = layer_module( 2025-08-14T21:56:06.9912247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9912596Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9912938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9913285Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9913617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9913962Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9914304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:56:06.9914639Z query_states = self.q(hidden_states) 2025-08-14T21:56:06.9914773Z 2025-08-14T21:56:06.9914874Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9915217Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9915527Z return mod(**inputs) 2025-08-14T21:56:06.9915848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9916198Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9916548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9916879Z layer_outputs = layer_module( 2025-08-14T21:56:06.9917207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9917553Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9917902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9918252Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9918602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9918955Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9919299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:56:06.9919650Z key_states = self.k(current_states) 2025-08-14T21:56:06.9919779Z 2025-08-14T21:56:06.9919877Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9920213Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9920526Z return mod(**inputs) 2025-08-14T21:56:06.9920857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9921207Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9921548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9921913Z layer_outputs = layer_module( 2025-08-14T21:56:06.9922240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9922589Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9922936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9923294Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9923714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9924095Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9924434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:56:06.9924842Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:56:06.9925017Z 2025-08-14T21:56:06.9925122Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9925598Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9925925Z return mod(**inputs) 2025-08-14T21:56:06.9926292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9926683Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9927040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9927396Z layer_outputs = layer_module( 2025-08-14T21:56:06.9927725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9928070Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9928416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9928770Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9929122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9929474Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9929828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:06.9930252Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:06.9930447Z 2025-08-14T21:56:06.9930553Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9930887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9931193Z return mod(**inputs) 2025-08-14T21:56:06.9931526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9931875Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9932218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9932573Z layer_outputs = layer_module( 2025-08-14T21:56:06.9932903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9933236Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9933584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9933975Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9934322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9934672Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9935039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:06.9935453Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:06.9935647Z 2025-08-14T21:56:06.9935744Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9936082Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9936385Z return mod(**inputs) 2025-08-14T21:56:06.9936711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9937074Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9937415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9937881Z layer_outputs = layer_module( 2025-08-14T21:56:06.9938235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9938593Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9938955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9939323Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9939677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9940051Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9940411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:06.9940834Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:06.9941029Z 2025-08-14T21:56:06.9941128Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9941465Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9941767Z return mod(**inputs) 2025-08-14T21:56:06.9942086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9942440Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9942782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9943131Z layer_outputs = layer_module( 2025-08-14T21:56:06.9943446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9943797Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9944154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9944513Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9944876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9945241Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9945605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:56:06.9945951Z value_states = self.v(current_states) 2025-08-14T21:56:06.9946089Z 2025-08-14T21:56:06.9946192Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9946543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9946922Z return mod(**inputs) 2025-08-14T21:56:06.9947254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9947616Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9947970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9948351Z layer_outputs = layer_module( 2025-08-14T21:56:06.9948691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9949042Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9949402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9949764Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9950126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9950521Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9950876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:06.9951016Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:06.9951022Z 2025-08-14T21:56:06.9951123Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9951326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9951390Z return mod(**inputs) 2025-08-14T21:56:06.9951617Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9951695Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9951923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9952003Z layer_outputs = layer_module( 2025-08-14T21:56:06.9952215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9952290Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9952521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9952601Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9952823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9952910Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9953130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:06.9953240Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:06.9953243Z 2025-08-14T21:56:06.9953342Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9953540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9953616Z return mod(**inputs) 2025-08-14T21:56:06.9953843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9953915Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9954146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9954214Z layer_outputs = layer_module( 2025-08-14T21:56:06.9954430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9954505Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9954726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9954833Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9955060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9955148Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9955372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:56:06.9955495Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:06.9955499Z 2025-08-14T21:56:06.9955607Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9955798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9955860Z return mod(**inputs) 2025-08-14T21:56:06.9956095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9956165Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9956416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9956487Z layer_outputs = layer_module( 2025-08-14T21:56:06.9956715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9956801Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9957034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9957109Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9957332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9957406Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9957627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:56:06.9957702Z attn_output = self.o(attn_output) 2025-08-14T21:56:06.9957706Z 2025-08-14T21:56:06.9957802Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9957992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9958055Z return mod(**inputs) 2025-08-14T21:56:06.9958284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9958353Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9958569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9958643Z layer_outputs = layer_module( 2025-08-14T21:56:06.9958847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9958920Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9959148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9959223Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9959445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward 2025-08-14T21:56:06.9959569Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-14T21:56:06.9959572Z 2025-08-14T21:56:06.9959646Z cudagraph partition due to non gpu ops 2025-08-14T21:56:06.9959749Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9959932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9959994Z return mod(**inputs) 2025-08-14T21:56:06.9960228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9960294Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9960533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9960600Z layer_outputs = layer_module( 2025-08-14T21:56:06.9960800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9960896Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9961109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:06.9961201Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:06.9961418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:56:06.9961510Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:56:06.9961734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:06.9961829Z return self.weight * hidden_states 2025-08-14T21:56:06.9961833Z 2025-08-14T21:56:06.9961930Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9962139Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9962204Z return mod(**inputs) 2025-08-14T21:56:06.9962432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9962510Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9962730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9962807Z layer_outputs = layer_module( 2025-08-14T21:56:06.9963016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9963091Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9963322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:06.9963410Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:06.9963635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:06.9963747Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:06.9963963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:56:06.9964047Z hidden_states = self.wi(hidden_states) 2025-08-14T21:56:06.9964050Z 2025-08-14T21:56:06.9964144Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9964335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9964408Z return mod(**inputs) 2025-08-14T21:56:06.9964631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9964708Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9964928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9964998Z layer_outputs = layer_module( 2025-08-14T21:56:06.9965302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9965390Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9965621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:06.9965706Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:06.9965924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:06.9966043Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:06.9966297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:56:06.9966381Z hidden_states = self.act(hidden_states) 2025-08-14T21:56:06.9966386Z 2025-08-14T21:56:06.9966483Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9966709Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9966780Z return mod(**inputs) 2025-08-14T21:56:06.9967006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9967079Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9967317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9967388Z layer_outputs = layer_module( 2025-08-14T21:56:06.9967640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9967725Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9967998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:06.9968101Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:06.9968347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:06.9968473Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:06.9968715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:56:06.9968805Z hidden_states = self.wo(hidden_states) 2025-08-14T21:56:06.9968809Z 2025-08-14T21:56:06.9968895Z cudagraph partition due to non gpu ops 2025-08-14T21:56:06.9968992Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9969185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9969258Z return mod(**inputs) 2025-08-14T21:56:06.9969482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9969560Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9969783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9969852Z layer_outputs = layer_module( 2025-08-14T21:56:06.9970070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9970143Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9970366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9970454Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9970678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-14T21:56:06.9970785Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:56:06.9971008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:06.9971083Z return self.weight * hidden_states 2025-08-14T21:56:06.9971087Z 2025-08-14T21:56:06.9971191Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9971379Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9971446Z return mod(**inputs) 2025-08-14T21:56:06.9987731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9987980Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9988427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9988515Z layer_outputs = layer_module( 2025-08-14T21:56:06.9988749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9988870Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9989116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9989198Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9989430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9989524Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9989747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:56:06.9989861Z query_states = self.q(hidden_states) 2025-08-14T21:56:06.9989868Z 2025-08-14T21:56:06.9989978Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9990181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9990288Z return mod(**inputs) 2025-08-14T21:56:06.9990523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9990606Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9990831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9990904Z layer_outputs = layer_module( 2025-08-14T21:56:06.9991123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9991204Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9991429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9991517Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9991740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9991831Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9992054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:56:06.9992128Z key_states = self.k(current_states) 2025-08-14T21:56:06.9992132Z 2025-08-14T21:56:06.9992240Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9992435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9992500Z return mod(**inputs) 2025-08-14T21:56:06.9992735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9992812Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9993050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9993122Z layer_outputs = layer_module( 2025-08-14T21:56:06.9993337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9993421Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9993644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9993727Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9993949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9994028Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9994255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:56:06.9994402Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:56:06.9994406Z 2025-08-14T21:56:06.9994510Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9994711Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9994793Z return mod(**inputs) 2025-08-14T21:56:06.9995026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9995097Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9995317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9995391Z layer_outputs = layer_module( 2025-08-14T21:56:06.9995598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9995700Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9995924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9996016Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9996243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9996320Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9996535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:06.9996688Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:06.9996692Z 2025-08-14T21:56:06.9996791Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9996980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9997049Z return mod(**inputs) 2025-08-14T21:56:06.9997268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9997347Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9997567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9997643Z layer_outputs = layer_module( 2025-08-14T21:56:06.9997848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:06.9997922Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:06.9998148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:06.9998223Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:06.9998444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:06.9998523Z attention_output = self.SelfAttention( 2025-08-14T21:56:06.9998742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:06.9998894Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:06.9998899Z 2025-08-14T21:56:06.9998997Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:06.9999182Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:06.9999253Z return mod(**inputs) 2025-08-14T21:56:06.9999474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:06.9999551Z encoder_outputs = self.encoder( 2025-08-14T21:56:06.9999772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:06.9999859Z layer_outputs = layer_module( 2025-08-14T21:56:07.0000076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0000151Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0000387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0000471Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0000686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0000769Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0000988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:07.0001128Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:07.0001133Z 2025-08-14T21:56:07.0001255Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0001447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0001519Z return mod(**inputs) 2025-08-14T21:56:07.0001761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0001834Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0002060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0002129Z layer_outputs = layer_module( 2025-08-14T21:56:07.0002334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0002415Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0002636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0002720Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0002938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0003015Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0003239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:56:07.0003313Z value_states = self.v(current_states) 2025-08-14T21:56:07.0003316Z 2025-08-14T21:56:07.0003422Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0003611Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0003675Z return mod(**inputs) 2025-08-14T21:56:07.0003902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0003977Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0004200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0004277Z layer_outputs = layer_module( 2025-08-14T21:56:07.0004488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0004571Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0004794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0004872Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0005106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0005297Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0005560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0005714Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0005719Z 2025-08-14T21:56:07.0005828Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0006048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0006147Z return mod(**inputs) 2025-08-14T21:56:07.0006383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0006466Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0006737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0006814Z layer_outputs = layer_module( 2025-08-14T21:56:07.0007029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0007137Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0007384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0007461Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0007697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0007786Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0008002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0008110Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0008113Z 2025-08-14T21:56:07.0008209Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0008394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0008465Z return mod(**inputs) 2025-08-14T21:56:07.0008689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0008758Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0008987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0009055Z layer_outputs = layer_module( 2025-08-14T21:56:07.0009265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0009339Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0009554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0009636Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0009850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0009938Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0010153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:56:07.0010254Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:07.0010257Z 2025-08-14T21:56:07.0010361Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0010546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0010608Z return mod(**inputs) 2025-08-14T21:56:07.0010834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0010901Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0011126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0011193Z layer_outputs = layer_module( 2025-08-14T21:56:07.0011436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0011515Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0011732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0011830Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0012045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0012121Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0012340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:56:07.0012414Z attn_output = self.o(attn_output) 2025-08-14T21:56:07.0012418Z 2025-08-14T21:56:07.0012494Z cudagraph partition due to non gpu ops 2025-08-14T21:56:07.0012596Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0012798Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0012867Z return mod(**inputs) 2025-08-14T21:56:07.0013111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0013183Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0013409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0013475Z layer_outputs = layer_module( 2025-08-14T21:56:07.0013679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0013759Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0013974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0014068Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0014286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:56:07.0014377Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:56:07.0014605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:07.0014679Z return self.weight * hidden_states 2025-08-14T21:56:07.0014683Z 2025-08-14T21:56:07.0014785Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0014975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0015036Z return mod(**inputs) 2025-08-14T21:56:07.0015265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0015334Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0015555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0015630Z layer_outputs = layer_module( 2025-08-14T21:56:07.0015837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0015921Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0016138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0016225Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0016447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0016561Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0016779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:56:07.0016886Z hidden_states = self.wi(hidden_states) 2025-08-14T21:56:07.0016889Z 2025-08-14T21:56:07.0016985Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0017179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0017261Z return mod(**inputs) 2025-08-14T21:56:07.0017489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0017568Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0017794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0017871Z layer_outputs = layer_module( 2025-08-14T21:56:07.0018084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0018158Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0018406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0018494Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0018740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0018860Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0019075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:56:07.0019156Z hidden_states = self.act(hidden_states) 2025-08-14T21:56:07.0019160Z 2025-08-14T21:56:07.0019254Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0019437Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0019505Z return mod(**inputs) 2025-08-14T21:56:07.0019727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0019794Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0020020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0020089Z layer_outputs = layer_module( 2025-08-14T21:56:07.0020298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0020370Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0020583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0020675Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0020892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0021012Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0021234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:56:07.0021309Z hidden_states = self.wo(hidden_states) 2025-08-14T21:56:07.0021313Z 2025-08-14T21:56:07.0021400Z cudagraph partition due to non gpu ops 2025-08-14T21:56:07.0021500Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0021686Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0021760Z return mod(**inputs) 2025-08-14T21:56:07.0021984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0022062Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0022290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0022375Z layer_outputs = layer_module( 2025-08-14T21:56:07.0022590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0022661Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0022880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0022980Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0023202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-14T21:56:07.0023308Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:56:07.0023529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:07.0023602Z return self.weight * hidden_states 2025-08-14T21:56:07.0023605Z 2025-08-14T21:56:07.0023713Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0023914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0023985Z return mod(**inputs) 2025-08-14T21:56:07.0025131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0025228Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0025458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0025524Z layer_outputs = layer_module( 2025-08-14T21:56:07.0025729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0025810Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0026026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0026112Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0026328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0026407Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0026635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:56:07.0026711Z query_states = self.q(hidden_states) 2025-08-14T21:56:07.0026714Z 2025-08-14T21:56:07.0026817Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0027003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0027065Z return mod(**inputs) 2025-08-14T21:56:07.0027290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0027361Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0027589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0027663Z layer_outputs = layer_module( 2025-08-14T21:56:07.0027876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0027960Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0028184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0028262Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0028498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0028575Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0028791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:56:07.0028891Z key_states = self.k(current_states) 2025-08-14T21:56:07.0028897Z 2025-08-14T21:56:07.0028993Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0029189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0029253Z return mod(**inputs) 2025-08-14T21:56:07.0029503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0029580Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0029811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0029887Z layer_outputs = layer_module( 2025-08-14T21:56:07.0030092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0030166Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0030407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0030486Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0030718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0030806Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0031023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:56:07.0031155Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:56:07.0031159Z 2025-08-14T21:56:07.0031257Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0031445Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0031515Z return mod(**inputs) 2025-08-14T21:56:07.0031738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0031809Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0032043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0032110Z layer_outputs = layer_module( 2025-08-14T21:56:07.0032318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0032389Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0032601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0032682Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0032894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0032976Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0033190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:07.0033330Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:07.0033334Z 2025-08-14T21:56:07.0033438Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0033622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0033682Z return mod(**inputs) 2025-08-14T21:56:07.0033906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0033973Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0034196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0034261Z layer_outputs = layer_module( 2025-08-14T21:56:07.0034461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0037194Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0037441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0037531Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0038000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0038143Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0038391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:07.0038559Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:07.0038565Z 2025-08-14T21:56:07.0038673Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0038942Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0039016Z return mod(**inputs) 2025-08-14T21:56:07.0039300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0039422Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0039676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0039752Z layer_outputs = layer_module( 2025-08-14T21:56:07.0039995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0040075Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0040337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0040422Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0040678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0040772Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0041026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:07.0041185Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:07.0041189Z 2025-08-14T21:56:07.0041306Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0041517Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0041594Z return mod(**inputs) 2025-08-14T21:56:07.0041851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0041927Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0042186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0042260Z layer_outputs = layer_module( 2025-08-14T21:56:07.0042491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0042580Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0042824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0042915Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0043169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0043253Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0043520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:56:07.0043601Z value_states = self.v(current_states) 2025-08-14T21:56:07.0043604Z 2025-08-14T21:56:07.0043718Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0044010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0044077Z return mod(**inputs) 2025-08-14T21:56:07.0044336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0044439Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0044694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0044776Z layer_outputs = layer_module( 2025-08-14T21:56:07.0045005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0045095Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0045512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0045607Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0045861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0045963Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0046210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0046334Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0046338Z 2025-08-14T21:56:07.0046444Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0046665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0046731Z return mod(**inputs) 2025-08-14T21:56:07.0046955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0047036Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0047257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0047336Z layer_outputs = layer_module( 2025-08-14T21:56:07.0047546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0047645Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0047872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0047957Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0048216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0048300Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0048555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0048665Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0048671Z 2025-08-14T21:56:07.0048775Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0048991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0049061Z return mod(**inputs) 2025-08-14T21:56:07.0049312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0049390Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0049631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0049721Z layer_outputs = layer_module( 2025-08-14T21:56:07.0049930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0050007Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0050238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0050386Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0050616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0050713Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0050933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:56:07.0051042Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:07.0051046Z 2025-08-14T21:56:07.0051142Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0051339Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0051403Z return mod(**inputs) 2025-08-14T21:56:07.0051646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0051728Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0051983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0052053Z layer_outputs = layer_module( 2025-08-14T21:56:07.0052272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0052347Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0052575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0052652Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0052873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0052959Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0053181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:56:07.0053256Z attn_output = self.o(attn_output) 2025-08-14T21:56:07.0053268Z 2025-08-14T21:56:07.0053365Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0053553Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0053624Z return mod(**inputs) 2025-08-14T21:56:07.0053845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0053914Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0054144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0054211Z layer_outputs = layer_module( 2025-08-14T21:56:07.0054429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0054504Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0054724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0054810Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0055029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward 2025-08-14T21:56:07.0055156Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-14T21:56:07.0055160Z 2025-08-14T21:56:07.0055243Z cudagraph partition due to non gpu ops 2025-08-14T21:56:07.0055341Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0055537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0055599Z return mod(**inputs) 2025-08-14T21:56:07.0055823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0055928Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0056157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0056242Z layer_outputs = layer_module( 2025-08-14T21:56:07.0056458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0056533Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0056760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0056848Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0057068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:56:07.0057183Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:56:07.0057413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:07.0057516Z return self.weight * hidden_states 2025-08-14T21:56:07.0057523Z 2025-08-14T21:56:07.0057629Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0057834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0057912Z return mod(**inputs) 2025-08-14T21:56:07.0058157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0058236Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0058494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0058569Z layer_outputs = layer_module( 2025-08-14T21:56:07.0058816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0058902Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0059151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0059259Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0059561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0059685Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0059943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:56:07.0060026Z hidden_states = self.wi(hidden_states) 2025-08-14T21:56:07.0060030Z 2025-08-14T21:56:07.0060139Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0060347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0060419Z return mod(**inputs) 2025-08-14T21:56:07.0060683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0060763Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0061023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0061097Z layer_outputs = layer_module( 2025-08-14T21:56:07.0061323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0061413Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0061671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0061765Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0062024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0062170Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0062411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:56:07.0062506Z hidden_states = self.act(hidden_states) 2025-08-14T21:56:07.0062510Z 2025-08-14T21:56:07.0062619Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0062818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0062882Z return mod(**inputs) 2025-08-14T21:56:07.0063116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0063189Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0063438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0063519Z layer_outputs = layer_module( 2025-08-14T21:56:07.0063751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0063832Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0064072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0064159Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0064397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0064511Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0064750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:56:07.0064836Z hidden_states = self.wo(hidden_states) 2025-08-14T21:56:07.0064841Z 2025-08-14T21:56:07.0064918Z cudagraph partition due to non gpu ops 2025-08-14T21:56:07.0065017Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0065219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0065283Z return mod(**inputs) 2025-08-14T21:56:07.0065517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1725, in forward 2025-08-14T21:56:07.0065588Z encoder_outputs = self.encoder( 2025-08-14T21:56:07.0065817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1128, in forward 2025-08-14T21:56:07.0065929Z hidden_states = self.final_layer_norm(hidden_states) 2025-08-14T21:56:07.0066155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:07.0066237Z return self.weight * hidden_states 2025-08-14T21:56:07.0066242Z 2025-08-14T21:56:07.0066340Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0066533Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0066605Z return mod(**inputs) 2025-08-14T21:56:07.0066836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0066906Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0067141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0067208Z layer_outputs = layer_module( 2025-08-14T21:56:07.0067427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0067502Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0067729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0067832Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0068061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0068159Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0068389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:56:07.0068463Z key_states = self.k(current_states) 2025-08-14T21:56:07.0068467Z 2025-08-14T21:56:07.0068568Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0068765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0068827Z return mod(**inputs) 2025-08-14T21:56:07.0069077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0069151Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0069394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0069476Z layer_outputs = layer_module( 2025-08-14T21:56:07.0069686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0069763Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0069983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0070059Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0070286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0070365Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0070592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:56:07.0070725Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:56:07.0070728Z 2025-08-14T21:56:07.0070823Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0071014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0071074Z return mod(**inputs) 2025-08-14T21:56:07.0071298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0071365Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0071580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0071655Z layer_outputs = layer_module( 2025-08-14T21:56:07.0071858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0071932Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0072156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0072230Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0072454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0072531Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0072746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:07.0072892Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:07.0072896Z 2025-08-14T21:56:07.0072989Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0073172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0073260Z return mod(**inputs) 2025-08-14T21:56:07.0073479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0073557Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0073796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0073862Z layer_outputs = layer_module( 2025-08-14T21:56:07.0074076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0074148Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0074374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0074449Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0074691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0074779Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0075013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:56:07.0075089Z value_states = self.v(current_states) 2025-08-14T21:56:07.0075093Z 2025-08-14T21:56:07.0075196Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0075382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0075451Z return mod(**inputs) 2025-08-14T21:56:07.0075675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0075746Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0075989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0076057Z layer_outputs = layer_module( 2025-08-14T21:56:07.0076262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0076342Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0076559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0076642Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0076857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0076933Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0077152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0077251Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0077255Z 2025-08-14T21:56:07.0077358Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0077543Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0077604Z return mod(**inputs) 2025-08-14T21:56:07.0077831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0077901Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0078120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0078196Z layer_outputs = layer_module( 2025-08-14T21:56:07.0078400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0078478Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0078695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0078768Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0079014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0079094Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0079336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0079447Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0079450Z 2025-08-14T21:56:07.0079549Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0079747Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0079809Z return mod(**inputs) 2025-08-14T21:56:07.0080036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0080130Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0080355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0080435Z layer_outputs = layer_module( 2025-08-14T21:56:07.0080663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0080749Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0080974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0081049Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0081263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0081346Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0081558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:56:07.0081664Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:07.0081667Z 2025-08-14T21:56:07.0081761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0081944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0082015Z return mod(**inputs) 2025-08-14T21:56:07.0082233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0082308Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0082525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0082592Z layer_outputs = layer_module( 2025-08-14T21:56:07.0082801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0082873Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0083089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0083170Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0083385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0083470Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0083685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:56:07.0083758Z attn_output = self.o(attn_output) 2025-08-14T21:56:07.0083761Z 2025-08-14T21:56:07.0083844Z cudagraph partition due to non gpu ops 2025-08-14T21:56:07.0083940Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0084122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0084194Z return mod(**inputs) 2025-08-14T21:56:07.0084436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0084512Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0084732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0084819Z layer_outputs = layer_module( 2025-08-14T21:56:07.0085037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0085111Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0085454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0085548Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0085791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:56:07.0085890Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:56:07.0086112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:07.0086208Z return self.weight * hidden_states 2025-08-14T21:56:07.0086215Z 2025-08-14T21:56:07.0086331Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0086540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0086619Z return mod(**inputs) 2025-08-14T21:56:07.0086868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0086946Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0087201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0087277Z layer_outputs = layer_module( 2025-08-14T21:56:07.0087508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0087596Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0087840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0087939Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0088172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0088297Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0088525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:56:07.0088600Z hidden_states = self.wi(hidden_states) 2025-08-14T21:56:07.0088604Z 2025-08-14T21:56:07.0088713Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0088904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0088969Z return mod(**inputs) 2025-08-14T21:56:07.0089204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0089275Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0089512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0089586Z layer_outputs = layer_module( 2025-08-14T21:56:07.0089794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0089872Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0090091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0090176Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0090421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0090530Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0090772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:56:07.0090847Z hidden_states = self.act(hidden_states) 2025-08-14T21:56:07.0090851Z 2025-08-14T21:56:07.0090946Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0091141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0091202Z return mod(**inputs) 2025-08-14T21:56:07.0091423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0091501Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0091736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0091815Z layer_outputs = layer_module( 2025-08-14T21:56:07.0092035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0092110Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0092334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0092418Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0092632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0092745Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0092963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:56:07.0093046Z hidden_states = self.wo(hidden_states) 2025-08-14T21:56:07.0093051Z 2025-08-14T21:56:07.0093146Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0093335Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0093406Z return mod(**inputs) 2025-08-14T21:56:07.0093630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0093704Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0093920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0093987Z layer_outputs = layer_module( 2025-08-14T21:56:07.0094201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0094275Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0094491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0094578Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0094794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-14T21:56:07.0094901Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:56:07.0095115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:07.0095186Z return self.weight * hidden_states 2025-08-14T21:56:07.0095189Z 2025-08-14T21:56:07.0095291Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0095476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0095536Z return mod(**inputs) 2025-08-14T21:56:07.0095765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0095851Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0096082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0096173Z layer_outputs = layer_module( 2025-08-14T21:56:07.0096379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0096459Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0096679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0096760Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0096982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0097074Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0097304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:56:07.0097381Z query_states = self.q(hidden_states) 2025-08-14T21:56:07.0097384Z 2025-08-14T21:56:07.0097500Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0097700Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0097761Z return mod(**inputs) 2025-08-14T21:56:07.0097992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0098062Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0098285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0098363Z layer_outputs = layer_module( 2025-08-14T21:56:07.0098578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0098655Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0098896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0098975Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0099218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0099296Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0099518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:56:07.0099598Z key_states = self.k(current_states) 2025-08-14T21:56:07.0099602Z 2025-08-14T21:56:07.0099700Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0099901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0099965Z return mod(**inputs) 2025-08-14T21:56:07.0100204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0100282Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0100500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0100567Z layer_outputs = layer_module( 2025-08-14T21:56:07.0100780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0100854Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0101077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0101153Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0101372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0101477Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0101703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:56:07.0101853Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:56:07.0101857Z 2025-08-14T21:56:07.0101956Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0102156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0102225Z return mod(**inputs) 2025-08-14T21:56:07.0102447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0102515Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0102761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0102828Z layer_outputs = layer_module( 2025-08-14T21:56:07.0103057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0103152Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0103374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0103454Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0103674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0103750Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0103976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:07.0104120Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:07.0104125Z 2025-08-14T21:56:07.0104228Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0104416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0104479Z return mod(**inputs) 2025-08-14T21:56:07.0104708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0104778Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0105005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0105072Z layer_outputs = layer_module( 2025-08-14T21:56:07.0105275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0105356Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0105576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0105652Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0105877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0105955Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0106182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:56:07.0106256Z value_states = self.v(current_states) 2025-08-14T21:56:07.0106259Z 2025-08-14T21:56:07.0106354Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0106548Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0106610Z return mod(**inputs) 2025-08-14T21:56:07.0106840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0106909Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0107149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0107224Z layer_outputs = layer_module( 2025-08-14T21:56:07.0107434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0107523Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0107749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0107824Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0108045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0108120Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0108351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0108463Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0108467Z 2025-08-14T21:56:07.0108562Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0108767Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0108838Z return mod(**inputs) 2025-08-14T21:56:07.0109064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0109138Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0109362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0109427Z layer_outputs = layer_module( 2025-08-14T21:56:07.0109642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0109716Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0109942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0110018Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0110238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0110321Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0110538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0110636Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0110639Z 2025-08-14T21:56:07.0110743Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0110928Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0110996Z return mod(**inputs) 2025-08-14T21:56:07.0111219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0111289Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0111517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0111586Z layer_outputs = layer_module( 2025-08-14T21:56:07.0111789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0111871Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0112088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0112170Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0112386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0112463Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0112713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:56:07.0112814Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:07.0112833Z 2025-08-14T21:56:07.0112936Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0113122Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0113184Z return mod(**inputs) 2025-08-14T21:56:07.0113409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0113478Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0113695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0113769Z layer_outputs = layer_module( 2025-08-14T21:56:07.0113992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0114076Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0114312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0114392Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0114618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0114705Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0114919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:56:07.0115000Z attn_output = self.o(attn_output) 2025-08-14T21:56:07.0115004Z 2025-08-14T21:56:07.0115080Z cudagraph partition due to non gpu ops 2025-08-14T21:56:07.0115183Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0115367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0115429Z return mod(**inputs) 2025-08-14T21:56:07.0115653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0115724Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0115948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0116013Z layer_outputs = layer_module( 2025-08-14T21:56:07.0116216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0116297Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0116510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0116587Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0116809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward 2025-08-14T21:56:07.0116909Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:56:07.0117135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:07.0117208Z return self.weight * hidden_states 2025-08-14T21:56:07.0117212Z 2025-08-14T21:56:07.0117306Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0117500Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0117561Z return mod(**inputs) 2025-08-14T21:56:07.0117783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0117859Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0118081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0118185Z layer_outputs = layer_module( 2025-08-14T21:56:07.0118392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0118489Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0118716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0118792Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0119017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0119096Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0119318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:56:07.0119424Z query_states = self.q(hidden_states) 2025-08-14T21:56:07.0119429Z 2025-08-14T21:56:07.0119525Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0119730Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0119802Z return mod(**inputs) 2025-08-14T21:56:07.0120024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0120099Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0120318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0120385Z layer_outputs = layer_module( 2025-08-14T21:56:07.0120597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0120670Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0120888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0120972Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0121191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0121277Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0121496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:56:07.0121567Z key_states = self.k(current_states) 2025-08-14T21:56:07.0121571Z 2025-08-14T21:56:07.0121671Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0121859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0121929Z return mod(**inputs) 2025-08-14T21:56:07.0122153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0122222Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0122448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0122516Z layer_outputs = layer_module( 2025-08-14T21:56:07.0122724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0122804Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0123025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0123106Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0123322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0123400Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0123625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:56:07.0123768Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:56:07.0123772Z 2025-08-14T21:56:07.0123873Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0124077Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0124138Z return mod(**inputs) 2025-08-14T21:56:07.0124366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0124434Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0124656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0124731Z layer_outputs = layer_module( 2025-08-14T21:56:07.0124950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0125032Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0125412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0125509Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0125774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0125864Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0126123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:07.0126295Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:07.0126301Z 2025-08-14T21:56:07.0126412Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0126637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0126719Z return mod(**inputs) 2025-08-14T21:56:07.0126975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0127062Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0127311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0127393Z layer_outputs = layer_module( 2025-08-14T21:56:07.0127624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0127707Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0127963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0128048Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0128303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0128402Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0128685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:56:07.0128776Z value_states = self.v(current_states) 2025-08-14T21:56:07.0128780Z 2025-08-14T21:56:07.0128883Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0129091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0129168Z return mod(**inputs) 2025-08-14T21:56:07.0129424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0129499Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0129764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0129838Z layer_outputs = layer_module( 2025-08-14T21:56:07.0130102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0130183Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0130443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0130536Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0130788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0130879Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0131130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0131239Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0131243Z 2025-08-14T21:56:07.0131375Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0131588Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0131657Z return mod(**inputs) 2025-08-14T21:56:07.0131938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0132017Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0132272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0132345Z layer_outputs = layer_module( 2025-08-14T21:56:07.0132573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0132659Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0132903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0132993Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0133236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0133323Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0133572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0133681Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0133685Z 2025-08-14T21:56:07.0133790Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0134003Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0134073Z return mod(**inputs) 2025-08-14T21:56:07.0134327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0134404Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0134649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0134733Z layer_outputs = layer_module( 2025-08-14T21:56:07.0134965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0135047Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0135298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0135381Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0135631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0135715Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0135959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:56:07.0136128Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:07.0136132Z 2025-08-14T21:56:07.0136236Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0136452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0136538Z return mod(**inputs) 2025-08-14T21:56:07.0136785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0136868Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0137113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0137185Z layer_outputs = layer_module( 2025-08-14T21:56:07.0137422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0137518Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0137909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0137998Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0138346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0138444Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0138688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:56:07.0138769Z attn_output = self.o(attn_output) 2025-08-14T21:56:07.0138783Z 2025-08-14T21:56:07.0138867Z cudagraph partition due to non gpu ops 2025-08-14T21:56:07.0138974Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0139189Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0139260Z return mod(**inputs) 2025-08-14T21:56:07.0139508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0139594Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0139843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0139927Z layer_outputs = layer_module( 2025-08-14T21:56:07.0140161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0140242Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0140492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0140588Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0140830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:56:07.0140942Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:56:07.0141186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:07.0141276Z return self.weight * hidden_states 2025-08-14T21:56:07.0141281Z 2025-08-14T21:56:07.0141387Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0141592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0141670Z return mod(**inputs) 2025-08-14T21:56:07.0141966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0142044Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0142308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0142384Z layer_outputs = layer_module( 2025-08-14T21:56:07.0142657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0142737Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0142990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0143122Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0143368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0143497Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0143749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:56:07.0143833Z hidden_states = self.wi(hidden_states) 2025-08-14T21:56:07.0143837Z 2025-08-14T21:56:07.0143978Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0144185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0144255Z return mod(**inputs) 2025-08-14T21:56:07.0144516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0144591Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0144827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0144895Z layer_outputs = layer_module( 2025-08-14T21:56:07.0145105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0145188Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0145412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0145506Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0145733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0145847Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0146077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:56:07.0146156Z hidden_states = self.act(hidden_states) 2025-08-14T21:56:07.0146159Z 2025-08-14T21:56:07.0146258Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0146455Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0146518Z return mod(**inputs) 2025-08-14T21:56:07.0146753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0146825Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0147051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0147128Z layer_outputs = layer_module( 2025-08-14T21:56:07.0147340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0147418Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0147648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0147733Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0147964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0148075Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0148299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:56:07.0148382Z hidden_states = self.wo(hidden_states) 2025-08-14T21:56:07.0148405Z 2025-08-14T21:56:07.0148484Z cudagraph partition due to non gpu ops 2025-08-14T21:56:07.0148593Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0148800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0148863Z return mod(**inputs) 2025-08-14T21:56:07.0149099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0149169Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0149396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0149472Z layer_outputs = layer_module( 2025-08-14T21:56:07.0149701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0149783Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0150008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0150100Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0150332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-14T21:56:07.0150434Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:56:07.0150658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:07.0150738Z return self.weight * hidden_states 2025-08-14T21:56:07.0150741Z 2025-08-14T21:56:07.0150840Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0151036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0151100Z return mod(**inputs) 2025-08-14T21:56:07.0151325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0151403Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0151633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0151707Z layer_outputs = layer_module( 2025-08-14T21:56:07.0151931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0152003Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0152234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0152312Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0152532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0152619Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0152839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:56:07.0152921Z query_states = self.q(hidden_states) 2025-08-14T21:56:07.0152926Z 2025-08-14T21:56:07.0153024Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0153214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0153284Z return mod(**inputs) 2025-08-14T21:56:07.0153507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0153575Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0153805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0153872Z layer_outputs = layer_module( 2025-08-14T21:56:07.0154087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0154195Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0154407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0154505Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0154715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0154798Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0155012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:56:07.0155083Z key_states = self.k(current_states) 2025-08-14T21:56:07.0155086Z 2025-08-14T21:56:07.0155184Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0155381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0155445Z return mod(**inputs) 2025-08-14T21:56:07.0155691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0155764Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0155997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0156064Z layer_outputs = layer_module( 2025-08-14T21:56:07.0156271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0156351Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0156569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0156645Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0156876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0156958Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0157188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:56:07.0157315Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:56:07.0157319Z 2025-08-14T21:56:07.0157428Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0157621Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0157683Z return mod(**inputs) 2025-08-14T21:56:07.0157912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0157981Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0158203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0158278Z layer_outputs = layer_module( 2025-08-14T21:56:07.0158487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0158562Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0158789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0158865Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0159088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0159163Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0159387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:07.0159539Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:07.0159561Z 2025-08-14T21:56:07.0159657Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0159851Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0159929Z return mod(**inputs) 2025-08-14T21:56:07.0160154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0160230Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0160453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0160519Z layer_outputs = layer_module( 2025-08-14T21:56:07.0160737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0160811Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0161062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0161140Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0161371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0161458Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0161676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:56:07.0161747Z value_states = self.v(current_states) 2025-08-14T21:56:07.0161759Z 2025-08-14T21:56:07.0161853Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0162036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0162105Z return mod(**inputs) 2025-08-14T21:56:07.0162325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0162395Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0162620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0162688Z layer_outputs = layer_module( 2025-08-14T21:56:07.0162900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0162971Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0163189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0163270Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0163485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0163559Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0163784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0163884Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0163887Z 2025-08-14T21:56:07.0163991Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0164179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0164241Z return mod(**inputs) 2025-08-14T21:56:07.0164466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0164533Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0164756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0164822Z layer_outputs = layer_module( 2025-08-14T21:56:07.0165047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0165154Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0165572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0165668Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0165947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0166030Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0166276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0166390Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0166394Z 2025-08-14T21:56:07.0166491Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0166703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0166766Z return mod(**inputs) 2025-08-14T21:56:07.0166990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0167085Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0167307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0167380Z layer_outputs = layer_module( 2025-08-14T21:56:07.0167595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0167671Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0167908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0167988Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0168223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0168305Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0168532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:56:07.0168644Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:07.0168648Z 2025-08-14T21:56:07.0168747Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0168941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0169015Z return mod(**inputs) 2025-08-14T21:56:07.0169249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0169328Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0169554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0169622Z layer_outputs = layer_module( 2025-08-14T21:56:07.0169831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0169903Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0170114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0170195Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0170407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0170490Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0170701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:56:07.0170771Z attn_output = self.o(attn_output) 2025-08-14T21:56:07.0170774Z 2025-08-14T21:56:07.0170878Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0171063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0171153Z return mod(**inputs) 2025-08-14T21:56:07.0171377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0171463Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0171692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0171758Z layer_outputs = layer_module( 2025-08-14T21:56:07.0171964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0172043Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0172257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0172356Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0172578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 609, in forward 2025-08-14T21:56:07.0172719Z hidden_states = hidden_states + self.dropout(attention_output[0]) 2025-08-14T21:56:07.0172726Z 2025-08-14T21:56:07.0172811Z cudagraph partition due to non gpu ops 2025-08-14T21:56:07.0172905Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0173097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0173158Z return mod(**inputs) 2025-08-14T21:56:07.0173378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0173453Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0173671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0173739Z layer_outputs = layer_module( 2025-08-14T21:56:07.0173954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0174029Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0174256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0174332Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0174545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward 2025-08-14T21:56:07.0174650Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:56:07.0174865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:07.0174937Z return self.weight * hidden_states 2025-08-14T21:56:07.0174941Z 2025-08-14T21:56:07.0175042Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0175229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0175297Z return mod(**inputs) 2025-08-14T21:56:07.0175517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0175586Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0175811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0175878Z layer_outputs = layer_module( 2025-08-14T21:56:07.0176083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0176164Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0176380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0176464Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0176701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0176783Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0177035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:56:07.0177112Z query_states = self.q(hidden_states) 2025-08-14T21:56:07.0177115Z 2025-08-14T21:56:07.0177217Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0177416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0177480Z return mod(**inputs) 2025-08-14T21:56:07.0177716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0177788Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0178036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0178117Z layer_outputs = layer_module( 2025-08-14T21:56:07.0178353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0178439Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0178677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0178753Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0178982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0179061Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0179295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:56:07.0179372Z key_states = self.k(current_states) 2025-08-14T21:56:07.0179377Z 2025-08-14T21:56:07.0179476Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0179680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0179745Z return mod(**inputs) 2025-08-14T21:56:07.0179974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0180052Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0180282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0180358Z layer_outputs = layer_module( 2025-08-14T21:56:07.0180575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0180649Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0180884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0180967Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0181203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0181289Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0181512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:56:07.0181647Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:56:07.0181650Z 2025-08-14T21:56:07.0182012Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0182204Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0182277Z return mod(**inputs) 2025-08-14T21:56:07.0182507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0182609Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0182838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0182938Z layer_outputs = layer_module( 2025-08-14T21:56:07.0183162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0183237Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0183463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0183559Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0183776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0183878Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0184097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:07.0184244Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:07.0184276Z 2025-08-14T21:56:07.0184382Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0184571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0184642Z return mod(**inputs) 2025-08-14T21:56:07.0184861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0184930Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0185155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0185221Z layer_outputs = layer_module( 2025-08-14T21:56:07.0185428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0185511Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0185728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0185810Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0186025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0186103Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0186324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:56:07.0186397Z value_states = self.v(current_states) 2025-08-14T21:56:07.0186400Z 2025-08-14T21:56:07.0186503Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0186688Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0186751Z return mod(**inputs) 2025-08-14T21:56:07.0186978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0187048Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0187269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0187350Z layer_outputs = layer_module( 2025-08-14T21:56:07.0187555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0187634Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0187852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0187927Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0188148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0188245Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0188464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0188589Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0188593Z 2025-08-14T21:56:07.0188686Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0188877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0188940Z return mod(**inputs) 2025-08-14T21:56:07.0189158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0189233Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0189468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0189546Z layer_outputs = layer_module( 2025-08-14T21:56:07.0189751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0189840Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0190069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0190143Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0190359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0190444Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0190659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0190766Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0190770Z 2025-08-14T21:56:07.0190867Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0191052Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0191124Z return mod(**inputs) 2025-08-14T21:56:07.0191347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0191415Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0191643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0191709Z layer_outputs = layer_module( 2025-08-14T21:56:07.0191919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0191991Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0192207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0192291Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0192508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0192593Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0192808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:56:07.0192904Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:07.0192908Z 2025-08-14T21:56:07.0193009Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0193195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0193257Z return mod(**inputs) 2025-08-14T21:56:07.0193485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0193554Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0193799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0193868Z layer_outputs = layer_module( 2025-08-14T21:56:07.0194096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0194179Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0194399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0194477Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0194704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0194785Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0195027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:56:07.0195102Z attn_output = self.o(attn_output) 2025-08-14T21:56:07.0195106Z 2025-08-14T21:56:07.0195181Z cudagraph partition due to non gpu ops 2025-08-14T21:56:07.0195300Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0195490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0195559Z return mod(**inputs) 2025-08-14T21:56:07.0195781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0195850Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0196077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0196144Z layer_outputs = layer_module( 2025-08-14T21:56:07.0196354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0196437Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0196656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0196750Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0196972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:56:07.0197062Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:56:07.0197289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:07.0197361Z return self.weight * hidden_states 2025-08-14T21:56:07.0197365Z 2025-08-14T21:56:07.0197468Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0197661Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0197722Z return mod(**inputs) 2025-08-14T21:56:07.0197954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0198026Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0198249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0198323Z layer_outputs = layer_module( 2025-08-14T21:56:07.0198532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0198613Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0198832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0198916Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0199144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0199276Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0199494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:56:07.0199594Z hidden_states = self.wi(hidden_states) 2025-08-14T21:56:07.0199597Z 2025-08-14T21:56:07.0199692Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0199884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0199947Z return mod(**inputs) 2025-08-14T21:56:07.0200165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0200242Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0200474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0200552Z layer_outputs = layer_module( 2025-08-14T21:56:07.0200760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0200848Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0201074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0201158Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0201376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0201492Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0201708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:56:07.0201790Z hidden_states = self.act(hidden_states) 2025-08-14T21:56:07.0201794Z 2025-08-14T21:56:07.0201890Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0202080Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0202150Z return mod(**inputs) 2025-08-14T21:56:07.0202370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0202440Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0202672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0202741Z layer_outputs = layer_module( 2025-08-14T21:56:07.0202959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0203035Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0203259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0203354Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0203578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0203697Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0203921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:56:07.0203997Z hidden_states = self.wo(hidden_states) 2025-08-14T21:56:07.0204000Z 2025-08-14T21:56:07.0204086Z cudagraph partition due to non gpu ops 2025-08-14T21:56:07.0204183Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0204374Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0204448Z return mod(**inputs) 2025-08-14T21:56:07.0204676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0204785Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0205023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0205095Z layer_outputs = layer_module( 2025-08-14T21:56:07.0205533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0205627Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0205897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0205993Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0206299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-14T21:56:07.0206412Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:56:07.0206670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:07.0206750Z return self.weight * hidden_states 2025-08-14T21:56:07.0206754Z 2025-08-14T21:56:07.0206884Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0207087Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0207159Z return mod(**inputs) 2025-08-14T21:56:07.0207391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0207463Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0207702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0207772Z layer_outputs = layer_module( 2025-08-14T21:56:07.0207989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0208073Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0208301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0208389Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0208619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0208701Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0208941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:56:07.0209015Z query_states = self.q(hidden_states) 2025-08-14T21:56:07.0209018Z 2025-08-14T21:56:07.0209121Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0209310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0209374Z return mod(**inputs) 2025-08-14T21:56:07.0209607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0209678Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0209956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0210031Z layer_outputs = layer_module( 2025-08-14T21:56:07.0210235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0210314Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0210531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0210607Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0210833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0210909Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0211147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:56:07.0211229Z key_states = self.k(current_states) 2025-08-14T21:56:07.0211253Z 2025-08-14T21:56:07.0211350Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0211545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0211607Z return mod(**inputs) 2025-08-14T21:56:07.0211829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0211909Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0212130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0212225Z layer_outputs = layer_module( 2025-08-14T21:56:07.0212442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0212517Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0212765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0212843Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0213064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0213147Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0213367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:56:07.0213494Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:56:07.0213498Z 2025-08-14T21:56:07.0213596Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0213783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0213854Z return mod(**inputs) 2025-08-14T21:56:07.0214089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0214158Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0214391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0214458Z layer_outputs = layer_module( 2025-08-14T21:56:07.0214673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0214747Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0214965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0215050Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0215273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0215356Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0215580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:07.0215739Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:07.0215743Z 2025-08-14T21:56:07.0215846Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0216031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0216091Z return mod(**inputs) 2025-08-14T21:56:07.0216322Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0216408Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0216680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0216770Z layer_outputs = layer_module( 2025-08-14T21:56:07.0216977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0217079Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0217302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0217378Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0217608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0217685Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0217915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:56:07.0218008Z value_states = self.v(current_states) 2025-08-14T21:56:07.0218014Z 2025-08-14T21:56:07.0218114Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0218325Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0218393Z return mod(**inputs) 2025-08-14T21:56:07.0218627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0218697Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0218931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0219006Z layer_outputs = layer_module( 2025-08-14T21:56:07.0219213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0219287Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0219522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0219601Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0219832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0219911Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0220140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0220248Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0220253Z 2025-08-14T21:56:07.0220346Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0220540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0220603Z return mod(**inputs) 2025-08-14T21:56:07.0220831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0220910Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0221138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0221207Z layer_outputs = layer_module( 2025-08-14T21:56:07.0221426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0221499Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0221732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0221811Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0222043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0222129Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0222372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0222494Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0222505Z 2025-08-14T21:56:07.0222604Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0222812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0222885Z return mod(**inputs) 2025-08-14T21:56:07.0223111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0223181Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0223416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0223483Z layer_outputs = layer_module( 2025-08-14T21:56:07.0223717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0223795Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0224042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0224132Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0224352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0224430Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0224659Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:56:07.0224770Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:07.0224775Z 2025-08-14T21:56:07.0224878Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0225068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0225132Z return mod(**inputs) 2025-08-14T21:56:07.0225361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0225431Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0225652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0225726Z layer_outputs = layer_module( 2025-08-14T21:56:07.0225931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0226010Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0226226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0226303Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0226531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0226610Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0226840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:56:07.0226916Z attn_output = self.o(attn_output) 2025-08-14T21:56:07.0226920Z 2025-08-14T21:56:07.0226999Z cudagraph partition due to non gpu ops 2025-08-14T21:56:07.0227106Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0227297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0227360Z return mod(**inputs) 2025-08-14T21:56:07.0227592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0227663Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0227904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0228002Z layer_outputs = layer_module( 2025-08-14T21:56:07.0228212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0228309Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0228529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0228607Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0228837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward 2025-08-14T21:56:07.0228938Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:56:07.0229170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:07.0229289Z return self.weight * hidden_states 2025-08-14T21:56:07.0229293Z 2025-08-14T21:56:07.0229397Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0229599Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0229693Z return mod(**inputs) 2025-08-14T21:56:07.0229929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0229999Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0230224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0230298Z layer_outputs = layer_module( 2025-08-14T21:56:07.0230508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0230581Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0230814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0230902Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0231128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0231208Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0231427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:56:07.0231510Z query_states = self.q(hidden_states) 2025-08-14T21:56:07.0231514Z 2025-08-14T21:56:07.0231615Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0231815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0231879Z return mod(**inputs) 2025-08-14T21:56:07.0232108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0232188Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0232417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0232486Z layer_outputs = layer_module( 2025-08-14T21:56:07.0232708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0232781Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0233012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0233089Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0233312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0233398Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0233621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:56:07.0233715Z key_states = self.k(current_states) 2025-08-14T21:56:07.0233719Z 2025-08-14T21:56:07.0233824Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0234016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0234103Z return mod(**inputs) 2025-08-14T21:56:07.0234330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0234400Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0234633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0234700Z layer_outputs = layer_module( 2025-08-14T21:56:07.0234911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0235011Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0235241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0235342Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0235575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0235656Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0235892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:56:07.0236018Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:56:07.0236022Z 2025-08-14T21:56:07.0236127Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0236322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0236388Z return mod(**inputs) 2025-08-14T21:56:07.0236627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0236702Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0236934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0237026Z layer_outputs = layer_module( 2025-08-14T21:56:07.0237236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0237315Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0237538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0237749Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0237995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0238076Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0238311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:07.0238460Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:07.0238466Z 2025-08-14T21:56:07.0238563Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0238766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0238832Z return mod(**inputs) 2025-08-14T21:56:07.0239063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0239144Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0239373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0239457Z layer_outputs = layer_module( 2025-08-14T21:56:07.0239699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0239778Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0240017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0240143Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0240372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0240462Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0240694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:56:07.0240778Z value_states = self.v(current_states) 2025-08-14T21:56:07.0240781Z 2025-08-14T21:56:07.0240921Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0241114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0241187Z return mod(**inputs) 2025-08-14T21:56:07.0241439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0241523Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0241755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0241823Z layer_outputs = layer_module( 2025-08-14T21:56:07.0242047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0242120Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0242342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0242427Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0242651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0242738Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0242965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0243070Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0243073Z 2025-08-14T21:56:07.0243178Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0243373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0243443Z return mod(**inputs) 2025-08-14T21:56:07.0243681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0243755Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0244002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0244075Z layer_outputs = layer_module( 2025-08-14T21:56:07.0244296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0244383Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0244618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0244706Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0244941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0245021Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0245346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0245464Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0245503Z 2025-08-14T21:56:07.0245607Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0245815Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0245898Z return mod(**inputs) 2025-08-14T21:56:07.0246140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0246214Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0246450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0246531Z layer_outputs = layer_module( 2025-08-14T21:56:07.0246750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0246834Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0247089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0247170Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0247431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0247513Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0247744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:56:07.0247852Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:07.0247855Z 2025-08-14T21:56:07.0247952Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0248146Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0248210Z return mod(**inputs) 2025-08-14T21:56:07.0248439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0248519Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0248756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0248828Z layer_outputs = layer_module( 2025-08-14T21:56:07.0249052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0249128Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0249368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0249445Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0249674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0249764Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0249996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:56:07.0250081Z attn_output = self.o(attn_output) 2025-08-14T21:56:07.0250085Z 2025-08-14T21:56:07.0250186Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0250382Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0250454Z return mod(**inputs) 2025-08-14T21:56:07.0250689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0250761Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0251002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0251073Z layer_outputs = layer_module( 2025-08-14T21:56:07.0251297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0251397Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0251629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0252602Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0252839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 647, in forward 2025-08-14T21:56:07.0252968Z layer_output = hidden_states + self.dropout(attention_output[0]) 2025-08-14T21:56:07.0252979Z 2025-08-14T21:56:07.0253060Z cudagraph partition due to non gpu ops 2025-08-14T21:56:07.0253161Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0253365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0253430Z return mod(**inputs) 2025-08-14T21:56:07.0253687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0253773Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0254031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0254111Z layer_outputs = layer_module( 2025-08-14T21:56:07.0254331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0254408Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0254649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0254741Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0254976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:56:07.0255085Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:56:07.0255316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:07.0255403Z return self.weight * hidden_states 2025-08-14T21:56:07.0255407Z 2025-08-14T21:56:07.0255508Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0255706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0255781Z return mod(**inputs) 2025-08-14T21:56:07.0256018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0256097Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0256334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0256405Z layer_outputs = layer_module( 2025-08-14T21:56:07.0256633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0256714Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0256947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0257047Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0257281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0257402Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0257635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:56:07.0257714Z hidden_states = self.wi(hidden_states) 2025-08-14T21:56:07.0257718Z 2025-08-14T21:56:07.0257826Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0258024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0258109Z return mod(**inputs) 2025-08-14T21:56:07.0258355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0258430Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0258727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0258798Z layer_outputs = layer_module( 2025-08-14T21:56:07.0259013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0259098Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0259329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0259427Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0259674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0259792Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0260050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:56:07.0260134Z hidden_states = self.act(hidden_states) 2025-08-14T21:56:07.0260138Z 2025-08-14T21:56:07.0260239Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0260441Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0260516Z return mod(**inputs) 2025-08-14T21:56:07.0260745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0260815Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0261039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0261116Z layer_outputs = layer_module( 2025-08-14T21:56:07.0261320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0261393Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0261621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0261703Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0261930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0262036Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0262254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:56:07.0262336Z hidden_states = self.wo(hidden_states) 2025-08-14T21:56:07.0262341Z 2025-08-14T21:56:07.0262415Z cudagraph partition due to non gpu ops 2025-08-14T21:56:07.0262518Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0262701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0262763Z return mod(**inputs) 2025-08-14T21:56:07.0262993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0263062Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0263281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0263358Z layer_outputs = layer_module( 2025-08-14T21:56:07.0263563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0263641Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0263861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0263958Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0264182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-14T21:56:07.0264298Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:56:07.0264515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:07.0264595Z return self.weight * hidden_states 2025-08-14T21:56:07.0264599Z 2025-08-14T21:56:07.0264692Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0264884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0264945Z return mod(**inputs) 2025-08-14T21:56:07.0265181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0265260Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0265494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0265570Z layer_outputs = layer_module( 2025-08-14T21:56:07.0265774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0265845Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0266069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0266145Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0266362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0266446Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0266663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:56:07.0266744Z query_states = self.q(hidden_states) 2025-08-14T21:56:07.0266748Z 2025-08-14T21:56:07.0266844Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0267039Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0267111Z return mod(**inputs) 2025-08-14T21:56:07.0267343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0267423Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0267654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0267725Z layer_outputs = layer_module( 2025-08-14T21:56:07.0267947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0268026Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0268255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0268345Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0268571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0268659Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0268892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:56:07.0268965Z key_states = self.k(current_states) 2025-08-14T21:56:07.0268968Z 2025-08-14T21:56:07.0269071Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0269257Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0269320Z return mod(**inputs) 2025-08-14T21:56:07.0269575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0269643Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0269874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0269968Z layer_outputs = layer_module( 2025-08-14T21:56:07.0270175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0270256Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0270474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0270555Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0270798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0270877Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0271101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:56:07.0271240Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:56:07.0271245Z 2025-08-14T21:56:07.0271343Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0271537Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0271598Z return mod(**inputs) 2025-08-14T21:56:07.0271829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0271900Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0272127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0272204Z layer_outputs = layer_module( 2025-08-14T21:56:07.0272417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0272492Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0272724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0272802Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0273031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0273110Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0273333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:07.0273491Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:07.0273495Z 2025-08-14T21:56:07.0273605Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0273800Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0273862Z return mod(**inputs) 2025-08-14T21:56:07.0274085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0274161Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0274383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0274450Z layer_outputs = layer_module( 2025-08-14T21:56:07.0274664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0274736Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0274965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0275040Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0275278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0275364Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0275599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:56:07.0275678Z value_states = self.v(current_states) 2025-08-14T21:56:07.0275681Z 2025-08-14T21:56:07.0275776Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0275958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0276026Z return mod(**inputs) 2025-08-14T21:56:07.0276243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0276327Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0276554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0276623Z layer_outputs = layer_module( 2025-08-14T21:56:07.0276855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0276930Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0277155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0277238Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0277460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0277537Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0277768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0277871Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0277876Z 2025-08-14T21:56:07.0277979Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0278172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0278235Z return mod(**inputs) 2025-08-14T21:56:07.0278478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0278547Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0278773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0278839Z layer_outputs = layer_module( 2025-08-14T21:56:07.0279044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0279127Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0279345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0279421Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0279650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0279729Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0279957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0280058Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0280062Z 2025-08-14T21:56:07.0280160Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0280357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0280421Z return mod(**inputs) 2025-08-14T21:56:07.0280646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0280738Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0280971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0281064Z layer_outputs = layer_module( 2025-08-14T21:56:07.0281275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0281350Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0281584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0281661Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0281895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0281990Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0282222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:56:07.0282335Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:07.0282358Z 2025-08-14T21:56:07.0282462Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0282671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0282742Z return mod(**inputs) 2025-08-14T21:56:07.0282971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0283049Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0283275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0283344Z layer_outputs = layer_module( 2025-08-14T21:56:07.0283567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0283644Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0283879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0283959Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0284186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0284272Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0284500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:56:07.0284576Z attn_output = self.o(attn_output) 2025-08-14T21:56:07.0284580Z 2025-08-14T21:56:07.0284668Z cudagraph partition due to non gpu ops 2025-08-14T21:56:07.0284768Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0284977Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0285047Z return mod(**inputs) 2025-08-14T21:56:07.0285424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0285526Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0285772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0285847Z layer_outputs = layer_module( 2025-08-14T21:56:07.0286087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0286167Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0286418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0286504Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0286746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward 2025-08-14T21:56:07.0286898Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:56:07.0287142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:07.0287236Z return self.weight * hidden_states 2025-08-14T21:56:07.0287248Z 2025-08-14T21:56:07.0287346Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0287536Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0287606Z return mod(**inputs) 2025-08-14T21:56:07.0287834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0287904Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0288160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0288231Z layer_outputs = layer_module( 2025-08-14T21:56:07.0288462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0288540Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0288762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0288849Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0289069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0289150Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0289378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:56:07.0289453Z query_states = self.q(hidden_states) 2025-08-14T21:56:07.0289457Z 2025-08-14T21:56:07.0289561Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0289749Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0289814Z return mod(**inputs) 2025-08-14T21:56:07.0290046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0290116Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0290347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0290415Z layer_outputs = layer_module( 2025-08-14T21:56:07.0290622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0290704Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0290927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0291004Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0291232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0291311Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0291536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:56:07.0291609Z key_states = self.k(current_states) 2025-08-14T21:56:07.0291612Z 2025-08-14T21:56:07.0291709Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0291902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0291965Z return mod(**inputs) 2025-08-14T21:56:07.0292191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0292268Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0292523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0292601Z layer_outputs = layer_module( 2025-08-14T21:56:07.0292828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0292901Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0293133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0293209Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0293438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0293518Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0293755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:56:07.0293890Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:56:07.0293894Z 2025-08-14T21:56:07.0294006Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0294199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0294270Z return mod(**inputs) 2025-08-14T21:56:07.0294494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0294582Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0294799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0294866Z layer_outputs = layer_module( 2025-08-14T21:56:07.0295081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0295156Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0295387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0295474Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0295705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0295794Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0296019Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:07.0296169Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:07.0296173Z 2025-08-14T21:56:07.0296281Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0296478Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0296552Z return mod(**inputs) 2025-08-14T21:56:07.0296785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0296859Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0297107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0297177Z layer_outputs = layer_module( 2025-08-14T21:56:07.0297385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0297465Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0297689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0297773Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0297996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0298095Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0298326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:56:07.0298418Z value_states = self.v(current_states) 2025-08-14T21:56:07.0298421Z 2025-08-14T21:56:07.0298527Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0298716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0298790Z return mod(**inputs) 2025-08-14T21:56:07.0299012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0299080Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0299296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0299383Z layer_outputs = layer_module( 2025-08-14T21:56:07.0299591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0299684Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0299902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0299980Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0300206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0300285Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0300507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0300616Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0300620Z 2025-08-14T21:56:07.0300717Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0300916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0300979Z return mod(**inputs) 2025-08-14T21:56:07.0301202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0301282Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0301504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0301581Z layer_outputs = layer_module( 2025-08-14T21:56:07.0301790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0301864Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0302091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0302169Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0302389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0302477Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0302699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0302806Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0302810Z 2025-08-14T21:56:07.0302906Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0303095Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0303167Z return mod(**inputs) 2025-08-14T21:56:07.0303391Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0303462Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0303693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0303782Z layer_outputs = layer_module( 2025-08-14T21:56:07.0304005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0304111Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0304329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0304413Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0304630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0304716Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0304950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:56:07.0305052Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:07.0305056Z 2025-08-14T21:56:07.0305158Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0305359Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0305422Z return mod(**inputs) 2025-08-14T21:56:07.0305653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0305723Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0305946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0306012Z layer_outputs = layer_module( 2025-08-14T21:56:07.0306217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0306299Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0306516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0306592Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0306817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0306894Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0307116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:56:07.0307189Z attn_output = self.o(attn_output) 2025-08-14T21:56:07.0307192Z 2025-08-14T21:56:07.0307268Z cudagraph partition due to non gpu ops 2025-08-14T21:56:07.0307371Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0307556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0307627Z return mod(**inputs) 2025-08-14T21:56:07.0307848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0307920Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0308149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0308219Z layer_outputs = layer_module( 2025-08-14T21:56:07.0308429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0308510Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0308732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0308837Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0309058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:56:07.0309150Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:56:07.0309394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:07.0309471Z return self.weight * hidden_states 2025-08-14T21:56:07.0309490Z 2025-08-14T21:56:07.0309595Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0309784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0309846Z return mod(**inputs) 2025-08-14T21:56:07.0310080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0310152Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0310384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0310478Z layer_outputs = layer_module( 2025-08-14T21:56:07.0310698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0310783Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0311035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0311127Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0311366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0311481Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0311707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:56:07.0311793Z hidden_states = self.wi(hidden_states) 2025-08-14T21:56:07.0311796Z 2025-08-14T21:56:07.0311903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0312091Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0312157Z return mod(**inputs) 2025-08-14T21:56:07.0312379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0312457Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0312683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0312757Z layer_outputs = layer_module( 2025-08-14T21:56:07.0312969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0313042Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0313273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0313361Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0313584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0313706Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0313928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:56:07.0314014Z hidden_states = self.act(hidden_states) 2025-08-14T21:56:07.0314017Z 2025-08-14T21:56:07.0314114Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0314305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0314377Z return mod(**inputs) 2025-08-14T21:56:07.0314602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0314671Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0314908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0314994Z layer_outputs = layer_module( 2025-08-14T21:56:07.0315221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0315312Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0315549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0315639Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0315861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0315975Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0316200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:56:07.0316291Z hidden_states = self.wo(hidden_states) 2025-08-14T21:56:07.0316297Z 2025-08-14T21:56:07.0316404Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0316610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0316677Z return mod(**inputs) 2025-08-14T21:56:07.0316913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0316982Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0317215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0317285Z layer_outputs = layer_module( 2025-08-14T21:56:07.0317500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0317585Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0317816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0317911Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0318142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 343, in forward 2025-08-14T21:56:07.0318270Z hidden_states = hidden_states + self.dropout(forwarded_states) 2025-08-14T21:56:07.0318274Z 2025-08-14T21:56:07.0318361Z cudagraph partition due to non gpu ops 2025-08-14T21:56:07.0318461Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0318657Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0318728Z return mod(**inputs) 2025-08-14T21:56:07.0318957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0319037Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0319276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0319346Z layer_outputs = layer_module( 2025-08-14T21:56:07.0319566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0319641Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0319863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0319949Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0320168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 598, in forward 2025-08-14T21:56:07.0320274Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:56:07.0320499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:07.0320577Z return self.weight * hidden_states 2025-08-14T21:56:07.0320606Z 2025-08-14T21:56:07.0320715Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0320916Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0321043Z return mod(**inputs) 2025-08-14T21:56:07.0321279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0321350Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0321585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0321656Z layer_outputs = layer_module( 2025-08-14T21:56:07.0321874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0321971Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0322199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0322286Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0322527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0322611Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0322851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:56:07.0322932Z query_states = self.q(hidden_states) 2025-08-14T21:56:07.0322936Z 2025-08-14T21:56:07.0323042Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0323258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0323328Z return mod(**inputs) 2025-08-14T21:56:07.0323583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0323661Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0323908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0323993Z layer_outputs = layer_module( 2025-08-14T21:56:07.0324225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0324323Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0324554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0324634Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0324872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0324952Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0325273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:56:07.0325380Z key_states = self.k(current_states) 2025-08-14T21:56:07.0325384Z 2025-08-14T21:56:07.0325490Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0325713Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0325785Z return mod(**inputs) 2025-08-14T21:56:07.0326037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0326137Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0326388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0326463Z layer_outputs = layer_module( 2025-08-14T21:56:07.0326710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0326821Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0327058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0327154Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0327385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0327473Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0327699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:56:07.0327833Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:56:07.0327837Z 2025-08-14T21:56:07.0327937Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0328149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0328226Z return mod(**inputs) 2025-08-14T21:56:07.0328463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0328551Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0328798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0328867Z layer_outputs = layer_module( 2025-08-14T21:56:07.0329093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0329169Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0329404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0329491Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0329728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0329810Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0330056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:07.0330208Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:07.0330212Z 2025-08-14T21:56:07.0330324Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0330521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0330586Z return mod(**inputs) 2025-08-14T21:56:07.0330830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0330903Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0331147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0331218Z layer_outputs = layer_module( 2025-08-14T21:56:07.0331437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0331521Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0331754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0331835Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0332076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0332157Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0332400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:56:07.0332476Z value_states = self.v(current_states) 2025-08-14T21:56:07.0332479Z 2025-08-14T21:56:07.0332582Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0332805Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0332869Z return mod(**inputs) 2025-08-14T21:56:07.0333113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0333202Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0333440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0333517Z layer_outputs = layer_module( 2025-08-14T21:56:07.0333734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0333811Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0334082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0334163Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0334401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0334497Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0334729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0334844Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0334847Z 2025-08-14T21:56:07.0334946Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0335149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0335214Z return mod(**inputs) 2025-08-14T21:56:07.0335447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0335528Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0335763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0335837Z layer_outputs = layer_module( 2025-08-14T21:56:07.0336076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0336159Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0336420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0336505Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0336756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0336848Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0337098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0337209Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0337213Z 2025-08-14T21:56:07.0337326Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0337535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0337819Z return mod(**inputs) 2025-08-14T21:56:07.0338092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0338173Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0338430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0338504Z layer_outputs = layer_module( 2025-08-14T21:56:07.0338735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0338828Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0339133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0339225Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0339463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0339572Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0339807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:56:07.0339912Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:07.0339916Z 2025-08-14T21:56:07.0340024Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0340219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0340285Z return mod(**inputs) 2025-08-14T21:56:07.0340551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0340628Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0340884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0340966Z layer_outputs = layer_module( 2025-08-14T21:56:07.0341183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0341266Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0341495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 681, in forward 2025-08-14T21:56:07.0341573Z self_attention_outputs = self.layer[0]( 2025-08-14T21:56:07.0341811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 599, in forward 2025-08-14T21:56:07.0341892Z attention_output = self.SelfAttention( 2025-08-14T21:56:07.0342128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:56:07.0342204Z attn_output = self.o(attn_output) 2025-08-14T21:56:07.0342209Z 2025-08-14T21:56:07.0342289Z cudagraph partition due to non gpu ops 2025-08-14T21:56:07.0342396Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0342593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0342658Z return mod(**inputs) 2025-08-14T21:56:07.0342900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0342974Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0343213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0343287Z layer_outputs = layer_module( 2025-08-14T21:56:07.0343506Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0343590Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0343822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0343903Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0344141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 634, in forward 2025-08-14T21:56:07.0344246Z normed_hidden_states = self.layer_norm(hidden_states) 2025-08-14T21:56:07.0344488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:07.0344564Z return self.weight * hidden_states 2025-08-14T21:56:07.0344568Z 2025-08-14T21:56:07.0344670Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0344874Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0344953Z return mod(**inputs) 2025-08-14T21:56:07.0345174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0345259Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0345473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0345546Z layer_outputs = layer_module( 2025-08-14T21:56:07.0345746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0345817Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0346035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0346125Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0346345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0346421Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0346647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 490, in forward 2025-08-14T21:56:07.0346730Z query_states = self.q(hidden_states) 2025-08-14T21:56:07.0346733Z 2025-08-14T21:56:07.0346825Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0347007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0347075Z return mod(**inputs) 2025-08-14T21:56:07.0347293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0347367Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0347587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0347654Z layer_outputs = layer_module( 2025-08-14T21:56:07.0347869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0347940Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0348161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0348235Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0348455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0348541Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0348761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 510, in forward 2025-08-14T21:56:07.0348836Z key_states = self.k(current_states) 2025-08-14T21:56:07.0348840Z 2025-08-14T21:56:07.0348953Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0349140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0349209Z return mod(**inputs) 2025-08-14T21:56:07.0349431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0349499Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0349725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0349791Z layer_outputs = layer_module( 2025-08-14T21:56:07.0349996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0350075Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0350293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0350394Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0350614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0350711Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0350944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 526, in forward 2025-08-14T21:56:07.0351069Z scores = torch.matmul(query_states, key_states.transpose(3, 2)) 2025-08-14T21:56:07.0351073Z 2025-08-14T21:56:07.0351176Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0351365Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0351429Z return mod(**inputs) 2025-08-14T21:56:07.0351678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0351751Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0352004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0352081Z layer_outputs = layer_module( 2025-08-14T21:56:07.0352289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0352367Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0352586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0352661Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0352884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0352961Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0353180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 558, in forward 2025-08-14T21:56:07.0353334Z attn_weights = nn.functional.softmax(scores.float(), dim=-1).type_as(scores) 2025-08-14T21:56:07.0353339Z 2025-08-14T21:56:07.0353435Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0353633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0353695Z return mod(**inputs) 2025-08-14T21:56:07.0353917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0353994Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0354216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0354291Z layer_outputs = layer_module( 2025-08-14T21:56:07.0354500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0354573Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0354800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0354877Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0355091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0355175Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0355390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 511, in forward 2025-08-14T21:56:07.0355469Z value_states = self.v(current_states) 2025-08-14T21:56:07.0355472Z 2025-08-14T21:56:07.0355566Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0355757Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0355844Z return mod(**inputs) 2025-08-14T21:56:07.0356069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0356146Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0356409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0356476Z layer_outputs = layer_module( 2025-08-14T21:56:07.0356687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0356759Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0356978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0357062Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0357294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0357381Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0357614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0357717Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0357720Z 2025-08-14T21:56:07.0357825Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0358019Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0358083Z return mod(**inputs) 2025-08-14T21:56:07.0358315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0358385Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0358622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0358691Z layer_outputs = layer_module( 2025-08-14T21:56:07.0358906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0358991Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0359218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0359303Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0359525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0359606Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0359841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 565, in forward 2025-08-14T21:56:07.0359944Z attn_output = torch.matmul(attn_weights, value_states) 2025-08-14T21:56:07.0359947Z 2025-08-14T21:56:07.0360048Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0360253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0360318Z return mod(**inputs) 2025-08-14T21:56:07.0360559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0360632Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0360862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0360941Z layer_outputs = layer_module( 2025-08-14T21:56:07.0361156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0361232Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0361471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0361549Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0361810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0361893Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0362139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 567, in forward 2025-08-14T21:56:07.0362250Z attn_output = attn_output.transpose(1, 2).contiguous() 2025-08-14T21:56:07.0362254Z 2025-08-14T21:56:07.0362353Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0362556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0362620Z return mod(**inputs) 2025-08-14T21:56:07.0362852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0362949Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0363184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0363254Z layer_outputs = layer_module( 2025-08-14T21:56:07.0363494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0363572Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0363812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 705, in forward 2025-08-14T21:56:07.0363889Z cross_attention_outputs = self.layer[1]( 2025-08-14T21:56:07.0364118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 635, in forward 2025-08-14T21:56:07.0364206Z attention_output = self.EncDecAttention( 2025-08-14T21:56:07.0364439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 569, in forward 2025-08-14T21:56:07.0364526Z attn_output = self.o(attn_output) 2025-08-14T21:56:07.0364530Z 2025-08-14T21:56:07.0364608Z cudagraph partition due to non gpu ops 2025-08-14T21:56:07.0364709Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0364915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0364981Z return mod(**inputs) 2025-08-14T21:56:07.0365297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0365399Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0365634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0365732Z layer_outputs = layer_module( 2025-08-14T21:56:07.0365954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0366031Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0366272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0366365Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0366599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 341, in forward 2025-08-14T21:56:07.0366705Z forwarded_states = self.layer_norm(hidden_states) 2025-08-14T21:56:07.0366935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 261, in forward 2025-08-14T21:56:07.0367020Z return self.weight * hidden_states 2025-08-14T21:56:07.0367024Z 2025-08-14T21:56:07.0367124Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0367323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0367400Z return mod(**inputs) 2025-08-14T21:56:07.0367661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0367744Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0367978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0368069Z layer_outputs = layer_module( 2025-08-14T21:56:07.0368294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0368372Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0368599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0368696Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0368938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0369062Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0369305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 287, in forward 2025-08-14T21:56:07.0369385Z hidden_states = self.wi(hidden_states) 2025-08-14T21:56:07.0369389Z 2025-08-14T21:56:07.0369498Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0369691Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0369756Z return mod(**inputs) 2025-08-14T21:56:07.0369996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0370067Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0370306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0370378Z layer_outputs = layer_module( 2025-08-14T21:56:07.0370596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0370680Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0370910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0371007Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0371235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0371347Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0371584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 288, in forward 2025-08-14T21:56:07.0371662Z hidden_states = self.act(hidden_states) 2025-08-14T21:56:07.0371666Z 2025-08-14T21:56:07.0371767Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0371971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0372036Z return mod(**inputs) 2025-08-14T21:56:07.0372277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1762, in forward 2025-08-14T21:56:07.0372349Z decoder_outputs = self.decoder( 2025-08-14T21:56:07.0372580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1092, in forward 2025-08-14T21:56:07.0372658Z layer_outputs = layer_module( 2025-08-14T21:56:07.0372873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:07.0372949Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:07.0373185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 731, in forward 2025-08-14T21:56:07.0373280Z hidden_states = self.layer[-1](hidden_states) 2025-08-14T21:56:07.0373517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 342, in forward 2025-08-14T21:56:07.0373620Z forwarded_states = self.DenseReluDense(forwarded_states) 2025-08-14T21:56:07.0373850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 296, in forward 2025-08-14T21:56:07.0373934Z hidden_states = self.wo(hidden_states) 2025-08-14T21:56:07.0373937Z 2025-08-14T21:56:07.0374012Z cudagraph partition due to non gpu ops 2025-08-14T21:56:07.0374113Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0374297Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0374357Z return mod(**inputs) 2025-08-14T21:56:07.0374603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1789, in forward 2025-08-14T21:56:07.0374722Z sequence_output = sequence_output * (self.model_dim**-0.5) 2025-08-14T21:56:07.0374727Z 2025-08-14T21:56:07.0374823Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0375038Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0375103Z return mod(**inputs) 2025-08-14T21:56:07.0375342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1791, in forward 2025-08-14T21:56:07.0375425Z lm_logits = self.lm_head(sequence_output) 2025-08-14T21:56:07.0375428Z 2025-08-14T21:56:07.0375524Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0375731Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0375792Z return mod(**inputs) 2025-08-14T21:56:07.0376025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1798, in forward 2025-08-14T21:56:07.0376158Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-08-14T21:56:07.0376162Z 2025-08-14T21:56:07.0376258Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0376457Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0376519Z return mod(**inputs) 2025-08-14T21:56:07.0376736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1798, in forward 2025-08-14T21:56:07.0376865Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-08-14T21:56:07.0376869Z 2025-08-14T21:56:07.0376959Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:07.0377147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:07.0377206Z return mod(**inputs) 2025-08-14T21:56:07.0377429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/t5/modeling_t5.py", line 1798, in forward 2025-08-14T21:56:07.0377563Z loss = loss_fct(lm_logits.view(-1, lm_logits.size(-1)), labels.view(-1)) 2025-08-14T21:56:07.0377567Z 2025-08-14T21:56:16.5423938Z Compilation time (from dynamo_timed): 17.297398391 2025-08-14T21:56:16.5580922Z pass 2025-08-14T21:56:16.5584855Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:56:16.5585661Z TIMING: _recursive_pre_grad_passes:0.01148 _recursive_joint_graph_passes:0.54499 _recursive_post_grad_passes:0.18666 async_compile.wait:0.74269 code_gen:9.19308 inductor_compile:10.71661 backend_compile:14.58534 gc:0.00036 entire_frame_compile:17.2974 total_wall_time:17.2974 2025-08-14T21:56:16.5589140Z STATS: call_* op count: 810 | FakeTensorMode.__torch_dispatch__:20429 | FakeTensor.__torch_dispatch__:5656 | ProxyTorchDispatchMode.__torch_dispatch__:7292 2025-08-14T21:56:16.5589766Z Dynamo produced 1 graphs covering 810 ops with 0 graph breaks (0 unique) 2025-08-14T21:56:22.0270686Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:56:22.0271889Z from pkg_resources import resource_filename 2025-08-14T21:56:22.6050323Z 2025-08-14T21:56:23.7587635Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:56:23.7589894Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:56:23.7600033Z cpu eval T5Small 2025-08-14T21:56:25.2157644Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:56:25.6217391Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:56:26.0319081Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:56:36.6540884Z Compilation time (from dynamo_timed): 9.052042704 2025-08-14T21:56:36.6713752Z pass 2025-08-14T21:56:36.6718512Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:56:36.6720594Z TIMING: _recursive_pre_grad_passes:0.01087 async_compile.wait:0.00588 backend_compile:6.36997 gc:0.00225 entire_frame_compile:9.05204 total_wall_time:9.05204 2025-08-14T21:56:36.6721123Z STATS: call_* op count: 810 | FakeTensorMode.__torch_dispatch__:2289 | FakeTensor.__torch_dispatch__:17 2025-08-14T21:56:36.6721489Z Dynamo produced 1 graphs covering 810 ops with 0 graph breaks (0 unique) 2025-08-14T21:56:41.1623758Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:56:41.1624666Z from pkg_resources import resource_filename 2025-08-14T21:56:41.7194572Z 2025-08-14T21:56:44.1533843Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:56:44.1538139Z loading model: 0it [00:02, ?it/s] 2025-08-14T21:56:44.1555623Z cpu eval TrOCRForCausalLM 2025-08-14T21:56:44.3079203Z WARNING:common:fp64 golden ref were not generated for TrOCRForCausalLM. Setting accuracy check to cosine 2025-08-14T21:56:44.3359808Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:56:44.6179458Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:56:44.8923958Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:56:52.1847656Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1848036Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1848295Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1848588Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1848803Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1849075Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1854140Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1859007Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1864118Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1866164Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1866534Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1872830Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1877790Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1882682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1883077Z return mod(**inputs) 2025-08-14T21:56:52.1883487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1884197Z outputs = self.model.decoder( 2025-08-14T21:56:52.1884577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1885014Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1885473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1885845Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1886243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.1886690Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.1887114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:56:52.1887594Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:56:52.1887765Z 2025-08-14T21:56:52.1887871Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1888269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1888591Z return mod(**inputs) 2025-08-14T21:56:52.1888937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1889322Z outputs = self.model.decoder( 2025-08-14T21:56:52.1889700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1890071Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1890399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1890743Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1891112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.1891501Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.1891894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:56:52.1892265Z key_states = self.k_proj(current_states) 2025-08-14T21:56:52.1892392Z 2025-08-14T21:56:52.1892498Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1892833Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1893146Z return mod(**inputs) 2025-08-14T21:56:52.1893489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1893867Z outputs = self.model.decoder( 2025-08-14T21:56:52.1894229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1894600Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1894940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1895276Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1895644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.1896035Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.1896423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:56:52.1896796Z value_states = self.v_proj(current_states) 2025-08-14T21:56:52.1896936Z 2025-08-14T21:56:52.1897016Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1897224Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1897416Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1897675Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1898023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1898324Z return mod(**inputs) 2025-08-14T21:56:52.1898718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1899082Z outputs = self.model.decoder( 2025-08-14T21:56:52.1899631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1899995Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1900318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1900656Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1901044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.1901420Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.1901820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:56:52.1902192Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:52.1902318Z 2025-08-14T21:56:52.1902420Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1902745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1903048Z return mod(**inputs) 2025-08-14T21:56:52.1903392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1903765Z outputs = self.model.decoder( 2025-08-14T21:56:52.1904105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1904463Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1904781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1905104Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1905468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.1905868Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.1906032Z 2025-08-14T21:56:52.1906133Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1906458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1906757Z return mod(**inputs) 2025-08-14T21:56:52.1907130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1907494Z outputs = self.model.decoder( 2025-08-14T21:56:52.1907845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1908200Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1908524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1908856Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1909207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.1909631Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.1909990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:52.1910310Z return self.act(input) 2025-08-14T21:56:52.1910413Z 2025-08-14T21:56:52.1910511Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1910861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1911160Z return mod(**inputs) 2025-08-14T21:56:52.1911485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1911864Z outputs = self.model.decoder( 2025-08-14T21:56:52.1912214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1912574Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1912883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1913217Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1913578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:56:52.1913958Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:52.1914089Z 2025-08-14T21:56:52.1914185Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1914528Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1914827Z return mod(**inputs) 2025-08-14T21:56:52.1915152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1915508Z outputs = self.model.decoder( 2025-08-14T21:56:52.1915869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1916237Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1916560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1916901Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1917270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.1917659Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.1918045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:56:52.1918454Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:56:52.1918610Z 2025-08-14T21:56:52.1918714Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1919048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1919355Z return mod(**inputs) 2025-08-14T21:56:52.1919704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1920075Z outputs = self.model.decoder( 2025-08-14T21:56:52.1920428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1920797Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1921127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1921468Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1921843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.1922237Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.1922627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:56:52.1922999Z key_states = self.k_proj(current_states) 2025-08-14T21:56:52.1923136Z 2025-08-14T21:56:52.1923234Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1923577Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1923900Z return mod(**inputs) 2025-08-14T21:56:52.1924245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1924626Z outputs = self.model.decoder( 2025-08-14T21:56:52.1924982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1925436Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1925798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1926159Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1926550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.1926990Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.1927378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:56:52.1927759Z value_states = self.v_proj(current_states) 2025-08-14T21:56:52.1927910Z 2025-08-14T21:56:52.1927990Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1928197Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1928397Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1928618Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1928951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1929264Z return mod(**inputs) 2025-08-14T21:56:52.1929609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1929972Z outputs = self.model.decoder( 2025-08-14T21:56:52.1930335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1930702Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1931030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1931366Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1931733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.1932127Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.1932507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:56:52.1932955Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:52.1933092Z 2025-08-14T21:56:52.1933188Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1933529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1933831Z return mod(**inputs) 2025-08-14T21:56:52.1934218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1934574Z outputs = self.model.decoder( 2025-08-14T21:56:52.1934919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1935276Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1935596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1935929Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1936289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.1936700Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.1936871Z 2025-08-14T21:56:52.1936969Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1937330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1937809Z return mod(**inputs) 2025-08-14T21:56:52.1938207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1938571Z outputs = self.model.decoder( 2025-08-14T21:56:52.1938917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1939281Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1939607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1939944Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1941324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.1941737Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.1942128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:52.1942451Z return self.act(input) 2025-08-14T21:56:52.1942553Z 2025-08-14T21:56:52.1942650Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1942984Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1943282Z return mod(**inputs) 2025-08-14T21:56:52.1943605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1943961Z outputs = self.model.decoder( 2025-08-14T21:56:52.1944309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1944665Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1944981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1945317Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1945677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:56:52.1946036Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:52.1946170Z 2025-08-14T21:56:52.1946266Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1946605Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1946917Z return mod(**inputs) 2025-08-14T21:56:52.1947239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1947595Z outputs = self.model.decoder( 2025-08-14T21:56:52.1947941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1948291Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1948609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1948940Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1949297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.1949667Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.1950038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:56:52.1950429Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:56:52.1950579Z 2025-08-14T21:56:52.1950679Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1951002Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1951404Z return mod(**inputs) 2025-08-14T21:56:52.1951737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1952098Z outputs = self.model.decoder( 2025-08-14T21:56:52.1952441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1952796Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1953116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1953443Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1953812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.1954224Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.1954618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:56:52.1954991Z key_states = self.k_proj(current_states) 2025-08-14T21:56:52.1955126Z 2025-08-14T21:56:52.1955221Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1955546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1955836Z return mod(**inputs) 2025-08-14T21:56:52.1956167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1956525Z outputs = self.model.decoder( 2025-08-14T21:56:52.1956878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1957277Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1957596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1957930Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1958277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.1958652Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.1959027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:56:52.1959404Z value_states = self.v_proj(current_states) 2025-08-14T21:56:52.1959536Z 2025-08-14T21:56:52.1959614Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1959814Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1960009Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1960223Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1960556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1960859Z return mod(**inputs) 2025-08-14T21:56:52.1961198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1961553Z outputs = self.model.decoder( 2025-08-14T21:56:52.1961906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1962270Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1962586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1962920Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1963280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.1963663Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.1964037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:56:52.1964436Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:52.1964564Z 2025-08-14T21:56:52.1964669Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1965022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1965320Z return mod(**inputs) 2025-08-14T21:56:52.1965739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1966112Z outputs = self.model.decoder( 2025-08-14T21:56:52.1966505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1966911Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1967269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1967629Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1968000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.1968412Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.1968577Z 2025-08-14T21:56:52.1968685Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1969024Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1969322Z return mod(**inputs) 2025-08-14T21:56:52.1969663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1970030Z outputs = self.model.decoder( 2025-08-14T21:56:52.1970380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1970746Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1971076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1971418Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1971783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.1972206Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.1972571Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:52.1972888Z return self.act(input) 2025-08-14T21:56:52.1973075Z 2025-08-14T21:56:52.1973180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1973523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1973832Z return mod(**inputs) 2025-08-14T21:56:52.1974171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1974539Z outputs = self.model.decoder( 2025-08-14T21:56:52.1974897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1975266Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1975597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1975943Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1976314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:56:52.1976682Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:52.1976822Z 2025-08-14T21:56:52.1976921Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1977273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1977593Z return mod(**inputs) 2025-08-14T21:56:52.1977921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1978292Z outputs = self.model.decoder( 2025-08-14T21:56:52.1978640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1978993Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1979315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1979654Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1980067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.1980475Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.1980871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:56:52.1981294Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:56:52.1981456Z 2025-08-14T21:56:52.1981563Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1981904Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1982221Z return mod(**inputs) 2025-08-14T21:56:52.1982576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1982934Z outputs = self.model.decoder( 2025-08-14T21:56:52.1983296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1983653Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1983973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1984299Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1984658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.1985039Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.1985410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:56:52.1985776Z key_states = self.k_proj(current_states) 2025-08-14T21:56:52.1985909Z 2025-08-14T21:56:52.1986007Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1986344Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1986642Z return mod(**inputs) 2025-08-14T21:56:52.1986984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1987351Z outputs = self.model.decoder( 2025-08-14T21:56:52.1987702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1988067Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1988401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1988731Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1989103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.1989482Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.1989862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:56:52.1990234Z value_states = self.v_proj(current_states) 2025-08-14T21:56:52.1990382Z 2025-08-14T21:56:52.1990456Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1990656Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1990853Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.1991079Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1991405Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1991705Z return mod(**inputs) 2025-08-14T21:56:52.1992031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1992384Z outputs = self.model.decoder( 2025-08-14T21:56:52.1992742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1993102Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1993440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1993788Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1994178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.1994572Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.1994949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:56:52.1995316Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:52.1995443Z 2025-08-14T21:56:52.1995545Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1995877Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1996183Z return mod(**inputs) 2025-08-14T21:56:52.1996528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.1996901Z outputs = self.model.decoder( 2025-08-14T21:56:52.1997253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.1997620Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.1997944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.1998283Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.1998641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.1999047Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.1999206Z 2025-08-14T21:56:52.1999311Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.1999645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.1999954Z return mod(**inputs) 2025-08-14T21:56:52.2000294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2000663Z outputs = self.model.decoder( 2025-08-14T21:56:52.2001014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2001377Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2001703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2002039Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2002408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.2002815Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.2003185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:52.2003598Z return self.act(input) 2025-08-14T21:56:52.2003716Z 2025-08-14T21:56:52.2003817Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2004186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2004504Z return mod(**inputs) 2025-08-14T21:56:52.2004860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2005233Z outputs = self.model.decoder( 2025-08-14T21:56:52.2005655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2006023Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2006408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2006799Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2007168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:56:52.2007553Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:52.2007693Z 2025-08-14T21:56:52.2007792Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2008130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2008428Z return mod(**inputs) 2025-08-14T21:56:52.2008778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2009139Z outputs = self.model.decoder( 2025-08-14T21:56:52.2009492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2009849Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2010174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2010506Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2010858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2011242Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2011619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:56:52.2012012Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:56:52.2012162Z 2025-08-14T21:56:52.2012256Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2012581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2012879Z return mod(**inputs) 2025-08-14T21:56:52.2013208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2013559Z outputs = self.model.decoder( 2025-08-14T21:56:52.2013910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2014267Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2014579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2014913Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2015268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2015642Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2016038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:56:52.2016410Z key_states = self.k_proj(current_states) 2025-08-14T21:56:52.2016558Z 2025-08-14T21:56:52.2016660Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2016999Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2017352Z return mod(**inputs) 2025-08-14T21:56:52.2017695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2018065Z outputs = self.model.decoder( 2025-08-14T21:56:52.2018456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2018813Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2019134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2019480Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2019835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2020220Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2020612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:56:52.2020972Z value_states = self.v_proj(current_states) 2025-08-14T21:56:52.2021109Z 2025-08-14T21:56:52.2021183Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2021383Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2021573Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2021782Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2022111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2022410Z return mod(**inputs) 2025-08-14T21:56:52.2022731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2023088Z outputs = self.model.decoder( 2025-08-14T21:56:52.2023432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2023787Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2024100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2024430Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2024788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2025160Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2025535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:56:52.2025897Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:52.2026022Z 2025-08-14T21:56:52.2026126Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2026453Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2026758Z return mod(**inputs) 2025-08-14T21:56:52.2027102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2027455Z outputs = self.model.decoder( 2025-08-14T21:56:52.2027793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2028147Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2028471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2028798Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2029158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.2029591Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.2029753Z 2025-08-14T21:56:52.2029858Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2030205Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2030511Z return mod(**inputs) 2025-08-14T21:56:52.2030851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2031208Z outputs = self.model.decoder( 2025-08-14T21:56:52.2031566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2031932Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2032273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2032608Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2032987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.2033397Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.2033763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:52.2034082Z return self.act(input) 2025-08-14T21:56:52.2034194Z 2025-08-14T21:56:52.2034290Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2034627Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2034924Z return mod(**inputs) 2025-08-14T21:56:52.2035268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2035651Z outputs = self.model.decoder( 2025-08-14T21:56:52.2036014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2036375Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2036707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2037047Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2037406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:56:52.2037911Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:52.2038052Z 2025-08-14T21:56:52.2038151Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2038489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2038792Z return mod(**inputs) 2025-08-14T21:56:52.2039135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2039508Z outputs = self.model.decoder( 2025-08-14T21:56:52.2039872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2040235Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2040566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2040913Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2041275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2041669Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2042060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:56:52.2042515Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:56:52.2042669Z 2025-08-14T21:56:52.2042766Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2043108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2043447Z return mod(**inputs) 2025-08-14T21:56:52.2043777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2044142Z outputs = self.model.decoder( 2025-08-14T21:56:52.2044500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2044866Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2045191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2045631Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2046058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2046511Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2046918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:56:52.2047304Z key_states = self.k_proj(current_states) 2025-08-14T21:56:52.2047431Z 2025-08-14T21:56:52.2047537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2047871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2048179Z return mod(**inputs) 2025-08-14T21:56:52.2048517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2048881Z outputs = self.model.decoder( 2025-08-14T21:56:52.2049229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2049593Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2049925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2050256Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2050620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2051012Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2051390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:56:52.2051756Z value_states = self.v_proj(current_states) 2025-08-14T21:56:52.2051895Z 2025-08-14T21:56:52.2051970Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2052172Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2052367Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2052575Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2052902Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2053202Z return mod(**inputs) 2025-08-14T21:56:52.2053528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2053896Z outputs = self.model.decoder( 2025-08-14T21:56:52.2054258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2054627Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2054947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2055286Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2055658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2056047Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2056422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:56:52.2056809Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:52.2056935Z 2025-08-14T21:56:52.2057039Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2057370Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2057676Z return mod(**inputs) 2025-08-14T21:56:52.2058013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2058373Z outputs = self.model.decoder( 2025-08-14T21:56:52.2058747Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2059126Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2059462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2059789Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2060146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.2060543Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.2060700Z 2025-08-14T21:56:52.2060801Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2061119Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2061421Z return mod(**inputs) 2025-08-14T21:56:52.2061755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2062105Z outputs = self.model.decoder( 2025-08-14T21:56:52.2062457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2062816Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2063137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2063459Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2063819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.2064218Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.2064567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:52.2064880Z return self.act(input) 2025-08-14T21:56:52.2064991Z 2025-08-14T21:56:52.2065089Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2065426Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2065719Z return mod(**inputs) 2025-08-14T21:56:52.2066053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2066414Z outputs = self.model.decoder( 2025-08-14T21:56:52.2066765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2067116Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2067439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2067771Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2068124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:56:52.2068515Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:52.2068648Z 2025-08-14T21:56:52.2068743Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2069073Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2069384Z return mod(**inputs) 2025-08-14T21:56:52.2069716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2070070Z outputs = self.model.decoder( 2025-08-14T21:56:52.2070416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2070780Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2071100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2071448Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2071803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2072196Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2072573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:56:52.2072963Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:56:52.2073114Z 2025-08-14T21:56:52.2073207Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2073535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2073832Z return mod(**inputs) 2025-08-14T21:56:52.2074154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2074509Z outputs = self.model.decoder( 2025-08-14T21:56:52.2074858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2075212Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2075523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2075853Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2076201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2076570Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2076942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:56:52.2077301Z key_states = self.k_proj(current_states) 2025-08-14T21:56:52.2077423Z 2025-08-14T21:56:52.2077523Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2077845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2078142Z return mod(**inputs) 2025-08-14T21:56:52.2078471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2078830Z outputs = self.model.decoder( 2025-08-14T21:56:52.2079169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2079524Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2079841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2080161Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2080520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2080902Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2081300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:56:52.2081665Z value_states = self.v_proj(current_states) 2025-08-14T21:56:52.2081820Z 2025-08-14T21:56:52.2081896Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2082098Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2082286Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2082505Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2082835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2083135Z return mod(**inputs) 2025-08-14T21:56:52.2083466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2083844Z outputs = self.model.decoder( 2025-08-14T21:56:52.2084193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2084544Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2084885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2085279Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2085818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2086233Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2086652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:56:52.2087048Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:52.2087189Z 2025-08-14T21:56:52.2087296Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2087622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2087926Z return mod(**inputs) 2025-08-14T21:56:52.2088265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2088618Z outputs = self.model.decoder( 2025-08-14T21:56:52.2088972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2089332Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2089657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2089986Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2090347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.2090749Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.2090909Z 2025-08-14T21:56:52.2091003Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2091330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2091629Z return mod(**inputs) 2025-08-14T21:56:52.2091964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2092315Z outputs = self.model.decoder( 2025-08-14T21:56:52.2092662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2093022Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2093334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2093664Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2094025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.2094446Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.2094800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:52.2095150Z return self.act(input) 2025-08-14T21:56:52.2095252Z 2025-08-14T21:56:52.2095353Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2095690Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2095991Z return mod(**inputs) 2025-08-14T21:56:52.2096336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2096703Z outputs = self.model.decoder( 2025-08-14T21:56:52.2097071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2097451Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2097788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2098136Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2098498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:56:52.2098873Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:52.2099004Z 2025-08-14T21:56:52.2099110Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2099454Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2099762Z return mod(**inputs) 2025-08-14T21:56:52.2100104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2100471Z outputs = self.model.decoder( 2025-08-14T21:56:52.2100826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2101195Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2101523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2101868Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2102229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2102620Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2103008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:56:52.2103405Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:56:52.2103569Z 2025-08-14T21:56:52.2103668Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2104009Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2104318Z return mod(**inputs) 2025-08-14T21:56:52.2104654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2105023Z outputs = self.model.decoder( 2025-08-14T21:56:52.2105385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2105752Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2106076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2106412Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2106782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2107176Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2107558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:56:52.2107952Z key_states = self.k_proj(current_states) 2025-08-14T21:56:52.2108078Z 2025-08-14T21:56:52.2108180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2108513Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2108818Z return mod(**inputs) 2025-08-14T21:56:52.2109157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2109518Z outputs = self.model.decoder( 2025-08-14T21:56:52.2109877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2110265Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2110599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2110951Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2111309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2111688Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2112064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:56:52.2112429Z value_states = self.v_proj(current_states) 2025-08-14T21:56:52.2112566Z 2025-08-14T21:56:52.2112642Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2112843Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2113031Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2113250Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2113583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2113877Z return mod(**inputs) 2025-08-14T21:56:52.2114217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2114574Z outputs = self.model.decoder( 2025-08-14T21:56:52.2114925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2115275Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2115599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2115937Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2116303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2116691Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2117078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:56:52.2117494Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:52.2117620Z 2025-08-14T21:56:52.2117715Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2118046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2118346Z return mod(**inputs) 2025-08-14T21:56:52.2118677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2119028Z outputs = self.model.decoder( 2025-08-14T21:56:52.2119377Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2119735Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2120052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2120407Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2120771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.2121187Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.2121348Z 2025-08-14T21:56:52.2121443Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2121772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2122070Z return mod(**inputs) 2025-08-14T21:56:52.2122401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2122754Z outputs = self.model.decoder( 2025-08-14T21:56:52.2123117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2123476Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2123804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2124136Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2124491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.2124885Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.2125250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:52.2125676Z return self.act(input) 2025-08-14T21:56:52.2125792Z 2025-08-14T21:56:52.2125905Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2126262Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2126592Z return mod(**inputs) 2025-08-14T21:56:52.2126935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2127309Z outputs = self.model.decoder( 2025-08-14T21:56:52.2127670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2128061Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2128412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2128774Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2129155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:56:52.2129552Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:52.2129691Z 2025-08-14T21:56:52.2129802Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2130150Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2130475Z return mod(**inputs) 2025-08-14T21:56:52.2130835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2131227Z outputs = self.model.decoder( 2025-08-14T21:56:52.2131597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2131981Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2132326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2132675Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2133070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2133507Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2133916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:56:52.2134345Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:56:52.2134514Z 2025-08-14T21:56:52.2134615Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2134966Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2135292Z return mod(**inputs) 2025-08-14T21:56:52.2135623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2135992Z outputs = self.model.decoder( 2025-08-14T21:56:52.2136371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2136741Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2137093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2137452Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2137984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2138414Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2138810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:56:52.2139187Z key_states = self.k_proj(current_states) 2025-08-14T21:56:52.2139314Z 2025-08-14T21:56:52.2139419Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2139753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2140114Z return mod(**inputs) 2025-08-14T21:56:52.2140459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2140820Z outputs = self.model.decoder( 2025-08-14T21:56:52.2141186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2141554Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2141885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2142217Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2142584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2143005Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2143384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:56:52.2143765Z value_states = self.v_proj(current_states) 2025-08-14T21:56:52.2143903Z 2025-08-14T21:56:52.2143979Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2144186Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2144380Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2144602Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2144939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2145242Z return mod(**inputs) 2025-08-14T21:56:52.2145586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2145956Z outputs = self.model.decoder( 2025-08-14T21:56:52.2146313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2146674Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2147058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2147405Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2147766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2148187Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2148575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:56:52.2148958Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:52.2149088Z 2025-08-14T21:56:52.2149188Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2149530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2149869Z return mod(**inputs) 2025-08-14T21:56:52.2150214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2150577Z outputs = self.model.decoder( 2025-08-14T21:56:52.2150968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2151340Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2151663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2152004Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2152372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.2152773Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.2152933Z 2025-08-14T21:56:52.2153027Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2153358Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2153662Z return mod(**inputs) 2025-08-14T21:56:52.2153989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2154347Z outputs = self.model.decoder( 2025-08-14T21:56:52.2154693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2155052Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2155364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2155699Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2156064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.2156471Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.2156832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:52.2157153Z return self.act(input) 2025-08-14T21:56:52.2157257Z 2025-08-14T21:56:52.2157363Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2157699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2157999Z return mod(**inputs) 2025-08-14T21:56:52.2158329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2158689Z outputs = self.model.decoder( 2025-08-14T21:56:52.2159029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2159384Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2159707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2160061Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2160419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:56:52.2160801Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:52.2160926Z 2025-08-14T21:56:52.2161028Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2161353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2161653Z return mod(**inputs) 2025-08-14T21:56:52.2161988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2162340Z outputs = self.model.decoder( 2025-08-14T21:56:52.2162708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2163071Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2163398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2163756Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2164124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2164510Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2164897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:56:52.2165291Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:56:52.2165509Z 2025-08-14T21:56:52.2165610Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2165954Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2166271Z return mod(**inputs) 2025-08-14T21:56:52.2166654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2167045Z outputs = self.model.decoder( 2025-08-14T21:56:52.2167415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2167782Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2168127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2168461Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2168815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2169201Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2169598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:56:52.2170000Z key_states = self.k_proj(current_states) 2025-08-14T21:56:52.2170140Z 2025-08-14T21:56:52.2170250Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2170633Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2170979Z return mod(**inputs) 2025-08-14T21:56:52.2171385Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2171792Z outputs = self.model.decoder( 2025-08-14T21:56:52.2172188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2172604Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2172963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2173343Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2173742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2174137Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2174539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:56:52.2174925Z value_states = self.v_proj(current_states) 2025-08-14T21:56:52.2175058Z 2025-08-14T21:56:52.2175144Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2175351Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2175543Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2175768Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2176114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2176434Z return mod(**inputs) 2025-08-14T21:56:52.2176793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2177171Z outputs = self.model.decoder( 2025-08-14T21:56:52.2177545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2177924Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2178260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2178610Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2178979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2179386Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2179784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:56:52.2180191Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:52.2180327Z 2025-08-14T21:56:52.2180429Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2180786Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2181114Z return mod(**inputs) 2025-08-14T21:56:52.2181465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2181857Z outputs = self.model.decoder( 2025-08-14T21:56:52.2182236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2182650Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2183008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2183395Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2183806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.2184242Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.2184430Z 2025-08-14T21:56:52.2184538Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2184955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2185277Z return mod(**inputs) 2025-08-14T21:56:52.2185616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2185984Z outputs = self.model.decoder( 2025-08-14T21:56:52.2186344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2186715Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2187037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2187392Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2187777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.2188224Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.2188619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:52.2188944Z return self.act(input) 2025-08-14T21:56:52.2189049Z 2025-08-14T21:56:52.2189154Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2189492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2189802Z return mod(**inputs) 2025-08-14T21:56:52.2190162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2190517Z outputs = self.model.decoder( 2025-08-14T21:56:52.2190895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2191258Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2191578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2191905Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2192266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:56:52.2192634Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:52.2192758Z 2025-08-14T21:56:52.2192860Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2193185Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2193484Z return mod(**inputs) 2025-08-14T21:56:52.2193815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2194167Z outputs = self.model.decoder( 2025-08-14T21:56:52.2194516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2194874Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2195193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2195515Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2195878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2196273Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2196655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:56:52.2197064Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:56:52.2197224Z 2025-08-14T21:56:52.2197325Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2197662Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2197970Z return mod(**inputs) 2025-08-14T21:56:52.2198306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2198671Z outputs = self.model.decoder( 2025-08-14T21:56:52.2199025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2199383Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2199712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2200075Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2200439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2200849Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2201234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:56:52.2201609Z key_states = self.k_proj(current_states) 2025-08-14T21:56:52.2201736Z 2025-08-14T21:56:52.2201832Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2202172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2202480Z return mod(**inputs) 2025-08-14T21:56:52.2202832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2203199Z outputs = self.model.decoder( 2025-08-14T21:56:52.2203560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2203946Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2204266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2204608Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2204974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2205426Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2205816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:56:52.2206196Z value_states = self.v_proj(current_states) 2025-08-14T21:56:52.2206328Z 2025-08-14T21:56:52.2206417Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2206619Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2206821Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2207044Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2207396Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2207716Z return mod(**inputs) 2025-08-14T21:56:52.2208087Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2208466Z outputs = self.model.decoder( 2025-08-14T21:56:52.2208831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2209214Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2209553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2209899Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2210264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2210652Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2211040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:56:52.2211410Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:52.2211546Z 2025-08-14T21:56:52.2211643Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2211982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2212287Z return mod(**inputs) 2025-08-14T21:56:52.2212623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2212996Z outputs = self.model.decoder( 2025-08-14T21:56:52.2213376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2213745Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2214068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2214423Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2214789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.2215197Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.2215366Z 2025-08-14T21:56:52.2215464Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2215801Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2216120Z return mod(**inputs) 2025-08-14T21:56:52.2216456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2216819Z outputs = self.model.decoder( 2025-08-14T21:56:52.2217196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2217554Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2217886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2218217Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2218575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.2218968Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.2219329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:52.2219642Z return self.act(input) 2025-08-14T21:56:52.2219745Z 2025-08-14T21:56:52.2219850Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2220173Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2220473Z return mod(**inputs) 2025-08-14T21:56:52.2220806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2221152Z outputs = self.model.decoder( 2025-08-14T21:56:52.2221501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2221858Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2222175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2222498Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2222856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:56:52.2223221Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:52.2223346Z 2025-08-14T21:56:52.2223440Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2223772Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2224076Z return mod(**inputs) 2025-08-14T21:56:52.2224420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2224777Z outputs = self.model.decoder( 2025-08-14T21:56:52.2225134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2225510Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2225823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2226199Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2226557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2226966Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2227346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 199, in forward 2025-08-14T21:56:52.2227749Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:56:52.2227922Z 2025-08-14T21:56:52.2228018Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2228349Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2228644Z return mod(**inputs) 2025-08-14T21:56:52.2229033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2229391Z outputs = self.model.decoder( 2025-08-14T21:56:52.2229744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2230099Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2230419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2230751Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2231102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2231202Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2231433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 218, in forward 2025-08-14T21:56:52.2231510Z key_states = self.k_proj(current_states) 2025-08-14T21:56:52.2231521Z 2025-08-14T21:56:52.2231615Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2231796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2231866Z return mod(**inputs) 2025-08-14T21:56:52.2232095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2232164Z outputs = self.model.decoder( 2025-08-14T21:56:52.2232401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2232468Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2232677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2232750Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2232980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2233080Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2233311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 219, in forward 2025-08-14T21:56:52.2233391Z value_states = self.v_proj(current_states) 2025-08-14T21:56:52.2233401Z 2025-08-14T21:56:52.2233476Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2233551Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2233630Z cudagraph partition due to non gpu ops 2025-08-14T21:56:52.2233724Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2233906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2233976Z return mod(**inputs) 2025-08-14T21:56:52.2234205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2234272Z outputs = self.model.decoder( 2025-08-14T21:56:52.2234523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2234593Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2234824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2234903Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2235140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 367, in forward 2025-08-14T21:56:52.2235238Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:56:52.2235473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 291, in forward 2025-08-14T21:56:52.2235556Z attn_output = self.out_proj(attn_output) 2025-08-14T21:56:52.2235559Z 2025-08-14T21:56:52.2235676Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2250321Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2250463Z return mod(**inputs) 2025-08-14T21:56:52.2250886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2250975Z outputs = self.model.decoder( 2025-08-14T21:56:52.2251240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2251314Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2251532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2251624Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2251872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.2251996Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.2252003Z 2025-08-14T21:56:52.2252122Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2252322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2252400Z return mod(**inputs) 2025-08-14T21:56:52.2252644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2252722Z outputs = self.model.decoder( 2025-08-14T21:56:52.2252973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2253046Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2253271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2253350Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2253586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 401, in forward 2025-08-14T21:56:52.2253710Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:56:52.2253909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:56:52.2253976Z return self.act(input) 2025-08-14T21:56:52.2253986Z 2025-08-14T21:56:52.2254085Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2254275Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2254342Z return mod(**inputs) 2025-08-14T21:56:52.2254573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 823, in forward 2025-08-14T21:56:52.2254644Z outputs = self.model.decoder( 2025-08-14T21:56:52.2254886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 644, in forward 2025-08-14T21:56:52.2254993Z layer_outputs = decoder_layer( 2025-08-14T21:56:52.2255208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:56:52.2255317Z return super().__call__(*args, **kwargs) 2025-08-14T21:56:52.2255546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 403, in forward 2025-08-14T21:56:52.2255630Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:56:52.2255634Z 2025-08-14T21:56:52.2255730Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2255912Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2255981Z return mod(**inputs) 2025-08-14T21:56:52.2256245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 839, in forward 2025-08-14T21:56:52.2256342Z logits = self.output_projection(outputs[0]) 2025-08-14T21:56:52.2256345Z 2025-08-14T21:56:52.2256438Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:56:52.2256636Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:56:52.2256710Z return mod(**inputs) 2025-08-14T21:56:52.2256948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/trocr/modeling_trocr.py", line 844, in forward 2025-08-14T21:56:52.2257093Z loss = loss_fct(logits.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:56:52.2257097Z 2025-08-14T21:57:00.0803556Z Compilation time (from dynamo_timed): 13.894337947 2025-08-14T21:57:00.0842239Z pass 2025-08-14T21:57:00.0842755Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:57:00.0843702Z TIMING: _recursive_pre_grad_passes:0.00777 _recursive_joint_graph_passes:0.71034 _recursive_post_grad_passes:0.07838 async_compile.wait:0.75382 code_gen:7.26462 inductor_compile:8.41198 backend_compile:11.71567 gc:0.00053 entire_frame_compile:13.89434 total_wall_time:13.89434 2025-08-14T21:57:00.0844644Z STATS: call_* op count: 443 | FakeTensorMode.__torch_dispatch__:14347 | FakeTensor.__torch_dispatch__:4678 | ProxyTorchDispatchMode.__torch_dispatch__:5467 2025-08-14T21:57:00.0845134Z Dynamo produced 1 graphs covering 443 ops with 0 graph breaks (0 unique) 2025-08-14T21:57:05.0469930Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:57:05.0471654Z from pkg_resources import resource_filename 2025-08-14T21:57:05.6589486Z 2025-08-14T21:57:11.7840528Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:57:11.7841018Z loading model: 0it [00:06, ?it/s] 2025-08-14T21:57:11.7863275Z cpu eval XGLMForCausalLM 2025-08-14T21:57:12.1499569Z WARNING:common:fp64 golden ref were not generated for XGLMForCausalLM. Setting accuracy check to cosine 2025-08-14T21:57:12.2278829Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:57:12.7027589Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:57:13.1746516Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:57:27.1360617Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1361123Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1361480Z return mod(**inputs) 2025-08-14T21:57:27.1361894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1362641Z outputs = self.model( 2025-08-14T21:57:27.1363021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1363495Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1363840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1364211Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1364610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1365028Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1365559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1366069Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1366246Z 2025-08-14T21:57:27.1366365Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1366811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1367135Z return mod(**inputs) 2025-08-14T21:57:27.1367476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1367845Z outputs = self.model( 2025-08-14T21:57:27.1368207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1368568Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1368906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1369253Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1369612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1370030Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1370411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.1370773Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.1370911Z 2025-08-14T21:57:27.1371010Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1371352Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1371656Z return mod(**inputs) 2025-08-14T21:57:27.1371982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1372341Z outputs = self.model( 2025-08-14T21:57:27.1372690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1373050Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1373388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1373746Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1374109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1374493Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1374881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1375284Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1375448Z 2025-08-14T21:57:27.1375552Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1375884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1376188Z return mod(**inputs) 2025-08-14T21:57:27.1376554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1376914Z outputs = self.model( 2025-08-14T21:57:27.1377305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1377671Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1378013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1378366Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1378736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1379118Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1379527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.1379954Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.1380147Z 2025-08-14T21:57:27.1380270Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1380620Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1380924Z return mod(**inputs) 2025-08-14T21:57:27.1381266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1381625Z outputs = self.model( 2025-08-14T21:57:27.1381968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1382327Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1382662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1383011Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1383372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1383764Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1384174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.1384547Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.1384686Z 2025-08-14T21:57:27.1384785Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1385142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1385446Z return mod(**inputs) 2025-08-14T21:57:27.1385784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1386144Z outputs = self.model( 2025-08-14T21:57:27.1386486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1386843Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1387180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1387528Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1387900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1388281Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1388666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.1389079Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.1389220Z 2025-08-14T21:57:27.1389318Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1389673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1389976Z return mod(**inputs) 2025-08-14T21:57:27.1390308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1390669Z outputs = self.model( 2025-08-14T21:57:27.1391003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1391368Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1391678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1392053Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1392403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1392790Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1393152Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.1393566Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.1393730Z 2025-08-14T21:57:27.1393831Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1394158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1394463Z return mod(**inputs) 2025-08-14T21:57:27.1394803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1395166Z outputs = self.model( 2025-08-14T21:57:27.1395502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1395871Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1396214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1396554Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1396905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1397301Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1397677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.1398032Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.1398166Z 2025-08-14T21:57:27.1398261Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1398597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1398907Z return mod(**inputs) 2025-08-14T21:57:27.1399240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1399601Z outputs = self.model( 2025-08-14T21:57:27.1399940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1400307Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1400634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1400976Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1401340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1401752Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1401925Z 2025-08-14T21:57:27.1402024Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1402391Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1402755Z return mod(**inputs) 2025-08-14T21:57:27.1403132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1403555Z outputs = self.model( 2025-08-14T21:57:27.1403927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1404325Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1404702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1405083Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1405587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1406047Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1406479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.1406847Z return self.act(input) 2025-08-14T21:57:27.1406962Z 2025-08-14T21:57:27.1407094Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1407466Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1407819Z return mod(**inputs) 2025-08-14T21:57:27.1408200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1408588Z outputs = self.model( 2025-08-14T21:57:27.1408973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1409375Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1409743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1410115Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1410519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.1410932Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.1411075Z 2025-08-14T21:57:27.1411183Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1411560Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1411903Z return mod(**inputs) 2025-08-14T21:57:27.1412278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1412676Z outputs = self.model( 2025-08-14T21:57:27.1413049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1413449Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1413806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1414197Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1414601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1415027Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1415444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1415848Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1416012Z 2025-08-14T21:57:27.1416114Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1416460Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1416766Z return mod(**inputs) 2025-08-14T21:57:27.1417106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1417487Z outputs = self.model( 2025-08-14T21:57:27.1417820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1418215Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1418541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1418882Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1419233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1419616Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1420009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.1420398Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.1420528Z 2025-08-14T21:57:27.1420626Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1420994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1421305Z return mod(**inputs) 2025-08-14T21:57:27.1421636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1421990Z outputs = self.model( 2025-08-14T21:57:27.1422326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1422682Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1423004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1423344Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1423704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1424088Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1424479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1424884Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1425039Z 2025-08-14T21:57:27.1425145Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1425487Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1425811Z return mod(**inputs) 2025-08-14T21:57:27.1426157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1426524Z outputs = self.model( 2025-08-14T21:57:27.1426854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1427212Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1427537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1427874Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1428236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1428616Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1429005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.1429436Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.1429636Z 2025-08-14T21:57:27.1429732Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1430072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1430414Z return mod(**inputs) 2025-08-14T21:57:27.1430756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1431125Z outputs = self.model( 2025-08-14T21:57:27.1431493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1431865Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1432208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1432557Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1432921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1433296Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1434494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.1434927Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.1435067Z 2025-08-14T21:57:27.1435204Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1435575Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1435900Z return mod(**inputs) 2025-08-14T21:57:27.1436254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1436618Z outputs = self.model( 2025-08-14T21:57:27.1436966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1437342Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1437953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1438323Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1438700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1439100Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1439490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.1439891Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.1440040Z 2025-08-14T21:57:27.1440149Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1440508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1440823Z return mod(**inputs) 2025-08-14T21:57:27.1441176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1441572Z outputs = self.model( 2025-08-14T21:57:27.1441937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1442340Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1442717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1443490Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1443897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1444323Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1444741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.1445197Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.1445384Z 2025-08-14T21:57:27.1445581Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1446046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1446422Z return mod(**inputs) 2025-08-14T21:57:27.1446805Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1447236Z outputs = self.model( 2025-08-14T21:57:27.1447587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1447969Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1448309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1448670Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1449077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1449475Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1449878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.1450308Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.1450442Z 2025-08-14T21:57:27.1450551Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1450892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1451209Z return mod(**inputs) 2025-08-14T21:57:27.1451553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1451917Z outputs = self.model( 2025-08-14T21:57:27.1452254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1452629Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1452976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1453329Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1453710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1454147Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1454319Z 2025-08-14T21:57:27.1454427Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1454775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1455103Z return mod(**inputs) 2025-08-14T21:57:27.1455458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1455835Z outputs = self.model( 2025-08-14T21:57:27.1456185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1456583Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1456948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1457321Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1457726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1458183Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1458566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.1458918Z return self.act(input) 2025-08-14T21:57:27.1459055Z 2025-08-14T21:57:27.1459165Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1459549Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1459903Z return mod(**inputs) 2025-08-14T21:57:27.1460270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1460650Z outputs = self.model( 2025-08-14T21:57:27.1461023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1461393Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1461739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1462110Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1462504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.1462904Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.1463045Z 2025-08-14T21:57:27.1463163Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1463524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1463831Z return mod(**inputs) 2025-08-14T21:57:27.1464190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1464576Z outputs = self.model( 2025-08-14T21:57:27.1464996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1465375Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1465733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1466112Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1466502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1466933Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1467353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1467792Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1467963Z 2025-08-14T21:57:27.1468068Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1468447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1468784Z return mod(**inputs) 2025-08-14T21:57:27.1469147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1469535Z outputs = self.model( 2025-08-14T21:57:27.1469928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1470323Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1470689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1471066Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1471473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1471897Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1472306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.1472711Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.1472850Z 2025-08-14T21:57:27.1472963Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1473342Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1473681Z return mod(**inputs) 2025-08-14T21:57:27.1474052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1474489Z outputs = self.model( 2025-08-14T21:57:27.1474865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1475281Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1475642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1476009Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1476408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1476828Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1477246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1477655Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1477823Z 2025-08-14T21:57:27.1477920Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1478303Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1478619Z return mod(**inputs) 2025-08-14T21:57:27.1478953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1479316Z outputs = self.model( 2025-08-14T21:57:27.1479658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1480029Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1480371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1480728Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1481107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1481511Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1481895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.1482315Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.1482493Z 2025-08-14T21:57:27.1482599Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1482932Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1483244Z return mod(**inputs) 2025-08-14T21:57:27.1483583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1483940Z outputs = self.model( 2025-08-14T21:57:27.1484289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1484664Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1485004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1485353Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1485813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1486214Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1486598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.1486994Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.1487138Z 2025-08-14T21:57:27.1487237Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1487592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1487938Z return mod(**inputs) 2025-08-14T21:57:27.1488308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1488692Z outputs = self.model( 2025-08-14T21:57:27.1489025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1489395Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1489729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1490081Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1490443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1490836Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1491237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.1491631Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.1491776Z 2025-08-14T21:57:27.1491893Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1492240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1492554Z return mod(**inputs) 2025-08-14T21:57:27.1492886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1493252Z outputs = self.model( 2025-08-14T21:57:27.1493595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1493962Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1494295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1494649Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1495023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1495408Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1495796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.1496216Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.1496387Z 2025-08-14T21:57:27.1496493Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1496831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1497145Z return mod(**inputs) 2025-08-14T21:57:27.1497490Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1497858Z outputs = self.model( 2025-08-14T21:57:27.1498191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1498558Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1498905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1499236Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1499595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1499974Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1500346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.1500703Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.1500838Z 2025-08-14T21:57:27.1500935Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1501290Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1501589Z return mod(**inputs) 2025-08-14T21:57:27.1501918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1502290Z outputs = self.model( 2025-08-14T21:57:27.1502626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1502983Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1503318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1503663Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1504048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1504453Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1504625Z 2025-08-14T21:57:27.1504728Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1505101Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1505411Z return mod(**inputs) 2025-08-14T21:57:27.1505767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1506122Z outputs = self.model( 2025-08-14T21:57:27.1506456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1506807Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1507133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1507474Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1507828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1508234Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1508599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.1508920Z return self.act(input) 2025-08-14T21:57:27.1509023Z 2025-08-14T21:57:27.1509118Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1509468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1509777Z return mod(**inputs) 2025-08-14T21:57:27.1510108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1510460Z outputs = self.model( 2025-08-14T21:57:27.1510792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1511150Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1511470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1511820Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1512191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.1512567Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.1512697Z 2025-08-14T21:57:27.1512796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1513154Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1513461Z return mod(**inputs) 2025-08-14T21:57:27.1513788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1514168Z outputs = self.model( 2025-08-14T21:57:27.1514511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1514875Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1515221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1515578Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1515939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-14T21:57:27.1516299Z hidden_states = residual + hidden_states 2025-08-14T21:57:27.1516434Z 2025-08-14T21:57:27.1516532Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1516875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1517202Z return mod(**inputs) 2025-08-14T21:57:27.1517544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1517914Z outputs = self.model( 2025-08-14T21:57:27.1518278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1518648Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1518980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1519325Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1519691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1520074Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1520466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1520873Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1521029Z 2025-08-14T21:57:27.1521136Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1521479Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1521796Z return mod(**inputs) 2025-08-14T21:57:27.1522139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1522496Z outputs = self.model( 2025-08-14T21:57:27.1522842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1523209Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1523547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1523889Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1524261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1524653Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1525043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.1525476Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.1525626Z 2025-08-14T21:57:27.1525736Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1526121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1526462Z return mod(**inputs) 2025-08-14T21:57:27.1526820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1527245Z outputs = self.model( 2025-08-14T21:57:27.1527596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1527982Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1528321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1528694Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1529060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1529483Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1529852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1530235Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1530383Z 2025-08-14T21:57:27.1530475Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1530830Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1531133Z return mod(**inputs) 2025-08-14T21:57:27.1531468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1531805Z outputs = self.model( 2025-08-14T21:57:27.1532128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1532476Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1532786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1533113Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1533460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1533832Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1534196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.1534609Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.1534783Z 2025-08-14T21:57:27.1534889Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1535226Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1535544Z return mod(**inputs) 2025-08-14T21:57:27.1535874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1536226Z outputs = self.model( 2025-08-14T21:57:27.1536551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1536908Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1537235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1537578Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1538072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1538449Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1538819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.1539176Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.1539316Z 2025-08-14T21:57:27.1539412Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1539743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1540036Z return mod(**inputs) 2025-08-14T21:57:27.1540354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1540750Z outputs = self.model( 2025-08-14T21:57:27.1541080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1541453Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1541781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1542118Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1542476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1542846Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1543222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.1543623Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.1543763Z 2025-08-14T21:57:27.1543865Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1544191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1544522Z return mod(**inputs) 2025-08-14T21:57:27.1544860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1545205Z outputs = self.model( 2025-08-14T21:57:27.1545548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1545919Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1546243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1546575Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1546935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1547313Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1547690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.1548110Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.1548288Z 2025-08-14T21:57:27.1548387Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1548727Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1549027Z return mod(**inputs) 2025-08-14T21:57:27.1549363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1549722Z outputs = self.model( 2025-08-14T21:57:27.1550060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1550421Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1550752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1551097Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1551455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1551842Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1552223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.1552597Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.1552727Z 2025-08-14T21:57:27.1552825Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1553170Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1553483Z return mod(**inputs) 2025-08-14T21:57:27.1553837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1554195Z outputs = self.model( 2025-08-14T21:57:27.1554540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1554930Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1555261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1555611Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1555978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1556391Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1556557Z 2025-08-14T21:57:27.1556673Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1557017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1557326Z return mod(**inputs) 2025-08-14T21:57:27.1557671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1558077Z outputs = self.model( 2025-08-14T21:57:27.1558423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1558791Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1559119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1559467Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1559838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1560246Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1560623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.1560950Z return self.act(input) 2025-08-14T21:57:27.1561055Z 2025-08-14T21:57:27.1561159Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1561489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1561795Z return mod(**inputs) 2025-08-14T21:57:27.1562131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1562481Z outputs = self.model( 2025-08-14T21:57:27.1562817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1563174Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1563505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1563839Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1564211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.1564593Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.1564722Z 2025-08-14T21:57:27.1564825Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1565163Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1565542Z return mod(**inputs) 2025-08-14T21:57:27.1565894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1566259Z outputs = self.model( 2025-08-14T21:57:27.1566633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1567032Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1567381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1567713Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1568092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1568475Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1568845Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1569232Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1569389Z 2025-08-14T21:57:27.1569483Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1569859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1570159Z return mod(**inputs) 2025-08-14T21:57:27.1570496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1570869Z outputs = self.model( 2025-08-14T21:57:27.1571201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1571560Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1571891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1572225Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1572575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1572953Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1573329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.1573693Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.1573818Z 2025-08-14T21:57:27.1573912Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1574255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1574559Z return mod(**inputs) 2025-08-14T21:57:27.1574881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1575231Z outputs = self.model( 2025-08-14T21:57:27.1575564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1575971Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1576288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1576622Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1576979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1577356Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1577723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1578113Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1578266Z 2025-08-14T21:57:27.1578368Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1578695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1579000Z return mod(**inputs) 2025-08-14T21:57:27.1579329Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1579681Z outputs = self.model( 2025-08-14T21:57:27.1580002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1580376Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1580700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1581063Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1581429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1581815Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1582196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.1582609Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.1582795Z 2025-08-14T21:57:27.1582909Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1583248Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1583551Z return mod(**inputs) 2025-08-14T21:57:27.1583890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1584254Z outputs = self.model( 2025-08-14T21:57:27.1584595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1584956Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1585285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1585631Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1586005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1586378Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1586752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.1587123Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.1587255Z 2025-08-14T21:57:27.1587351Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1587689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1587996Z return mod(**inputs) 2025-08-14T21:57:27.1588328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1588675Z outputs = self.model( 2025-08-14T21:57:27.1589021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1589394Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1589716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1590063Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1590440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1590819Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1591188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.1591566Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.1591711Z 2025-08-14T21:57:27.1591805Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1592142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1592437Z return mod(**inputs) 2025-08-14T21:57:27.1592770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1593146Z outputs = self.model( 2025-08-14T21:57:27.1593475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1593855Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1594180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1594522Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1594876Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1595256Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1595633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.1596056Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.1596224Z 2025-08-14T21:57:27.1596319Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1596671Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1596978Z return mod(**inputs) 2025-08-14T21:57:27.1597302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1597656Z outputs = self.model( 2025-08-14T21:57:27.1597987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1598339Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1598657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1598995Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1599353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1599723Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1600099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.1600466Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.1600591Z 2025-08-14T21:57:27.1600693Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1601023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1601328Z return mod(**inputs) 2025-08-14T21:57:27.1601660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1602019Z outputs = self.model( 2025-08-14T21:57:27.1602356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1602735Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1603078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1603426Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1603806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1604232Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1604403Z 2025-08-14T21:57:27.1604511Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1604858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1605174Z return mod(**inputs) 2025-08-14T21:57:27.1605601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1605985Z outputs = self.model( 2025-08-14T21:57:27.1606370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1606761Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1607110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1607473Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1607851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1608262Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1608635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.1608958Z return self.act(input) 2025-08-14T21:57:27.1609072Z 2025-08-14T21:57:27.1609199Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1609538Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1609836Z return mod(**inputs) 2025-08-14T21:57:27.1610185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1610543Z outputs = self.model( 2025-08-14T21:57:27.1610875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1611226Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1611551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1611888Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1612240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.1612605Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.1612741Z 2025-08-14T21:57:27.1612839Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1613190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1613494Z return mod(**inputs) 2025-08-14T21:57:27.1613835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1614194Z outputs = self.model( 2025-08-14T21:57:27.1614526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1614894Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1615226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1615571Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1615931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-14T21:57:27.1616306Z hidden_states = residual + hidden_states 2025-08-14T21:57:27.1616433Z 2025-08-14T21:57:27.1616540Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1616886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1617191Z return mod(**inputs) 2025-08-14T21:57:27.1617532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1617890Z outputs = self.model( 2025-08-14T21:57:27.1618225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1618590Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1618926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1619274Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1619657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1620048Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1620453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1620847Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1621012Z 2025-08-14T21:57:27.1621111Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1621456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1621769Z return mod(**inputs) 2025-08-14T21:57:27.1622102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1622481Z outputs = self.model( 2025-08-14T21:57:27.1622830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1623190Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1623540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1623893Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1624264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1624647Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1625034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.1625414Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.1625543Z 2025-08-14T21:57:27.1625650Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1625992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1626307Z return mod(**inputs) 2025-08-14T21:57:27.1626660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1627019Z outputs = self.model( 2025-08-14T21:57:27.1627367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1627741Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1628081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1628426Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1628800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1629207Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1629583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1629991Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1630148Z 2025-08-14T21:57:27.1630241Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1630574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1630867Z return mod(**inputs) 2025-08-14T21:57:27.1631195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1631541Z outputs = self.model( 2025-08-14T21:57:27.1631874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1632228Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1632558Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1632926Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1633281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1633691Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1634060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.1634475Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.1634650Z 2025-08-14T21:57:27.1634746Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1635086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1635396Z return mod(**inputs) 2025-08-14T21:57:27.1635736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1636108Z outputs = self.model( 2025-08-14T21:57:27.1636455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1636814Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1637131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1637463Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1638045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1638443Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1638819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.1639214Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.1639354Z 2025-08-14T21:57:27.1639462Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1639809Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1640129Z return mod(**inputs) 2025-08-14T21:57:27.1640476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1640879Z outputs = self.model( 2025-08-14T21:57:27.1641208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1641566Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1641895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1642233Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1642598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1642983Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1643363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.1643738Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.1643887Z 2025-08-14T21:57:27.1643983Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1644326Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1644651Z return mod(**inputs) 2025-08-14T21:57:27.1645003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1645363Z outputs = self.model( 2025-08-14T21:57:27.1645784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1646239Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1646587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1646974Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1647384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1647766Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1648185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.1648610Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.1648783Z 2025-08-14T21:57:27.1648889Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1649253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1649574Z return mod(**inputs) 2025-08-14T21:57:27.1649943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1650300Z outputs = self.model( 2025-08-14T21:57:27.1650649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1651072Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1651406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1651746Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1652115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1652503Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1652883Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.1653260Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.1653402Z 2025-08-14T21:57:27.1653502Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1653847Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1654152Z return mod(**inputs) 2025-08-14T21:57:27.1654495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1654859Z outputs = self.model( 2025-08-14T21:57:27.1655197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1655559Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1655897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1656246Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1656606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1657017Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1657181Z 2025-08-14T21:57:27.1657285Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1657631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1657933Z return mod(**inputs) 2025-08-14T21:57:27.1658277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1658642Z outputs = self.model( 2025-08-14T21:57:27.1658981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1659347Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1659711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1660040Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1660383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1660790Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1661142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.1661447Z return self.act(input) 2025-08-14T21:57:27.1661553Z 2025-08-14T21:57:27.1661645Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1661976Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1662279Z return mod(**inputs) 2025-08-14T21:57:27.1662620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1662984Z outputs = self.model( 2025-08-14T21:57:27.1663343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1663701Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1664022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1664357Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1664711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.1665068Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.1665203Z 2025-08-14T21:57:27.1665299Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1665644Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1665940Z return mod(**inputs) 2025-08-14T21:57:27.1666257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1666599Z outputs = self.model( 2025-08-14T21:57:27.1666924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1667266Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1667586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1667915Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1668265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1668630Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1669002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1669390Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1669539Z 2025-08-14T21:57:27.1669640Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1669964Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1670266Z return mod(**inputs) 2025-08-14T21:57:27.1670592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1670934Z outputs = self.model( 2025-08-14T21:57:27.1671295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1671651Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1671978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1672339Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1672708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1673116Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1673474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.1673827Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.1673955Z 2025-08-14T21:57:27.1674050Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1674381Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1674677Z return mod(**inputs) 2025-08-14T21:57:27.1675027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1675382Z outputs = self.model( 2025-08-14T21:57:27.1675712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1676087Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1676410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1676746Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1677094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1677471Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1677843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1678233Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1678384Z 2025-08-14T21:57:27.1678479Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1678813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1679113Z return mod(**inputs) 2025-08-14T21:57:27.1679436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1679789Z outputs = self.model( 2025-08-14T21:57:27.1680118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1680472Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1680785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1681122Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1681481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1681851Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1682226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.1682638Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.1682811Z 2025-08-14T21:57:27.1682914Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1683238Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1683538Z return mod(**inputs) 2025-08-14T21:57:27.1683875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1684235Z outputs = self.model( 2025-08-14T21:57:27.1684568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1684933Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1685292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1685693Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1686061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1686473Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1686860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.1687239Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.1687394Z 2025-08-14T21:57:27.1687491Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1687837Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1688148Z return mod(**inputs) 2025-08-14T21:57:27.1688492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1688850Z outputs = self.model( 2025-08-14T21:57:27.1689198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1689552Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1689878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1690218Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1690577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1690951Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1691326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.1691703Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.1691842Z 2025-08-14T21:57:27.1691937Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1692276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1692579Z return mod(**inputs) 2025-08-14T21:57:27.1692911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1693259Z outputs = self.model( 2025-08-14T21:57:27.1693593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1693949Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1694313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1694656Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1695016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1695397Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1695764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.1696170Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.1696341Z 2025-08-14T21:57:27.1696437Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1696771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1697067Z return mod(**inputs) 2025-08-14T21:57:27.1697400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1697754Z outputs = self.model( 2025-08-14T21:57:27.1698083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1698466Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1698798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1699160Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1699515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1699897Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1700275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.1700658Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.1700784Z 2025-08-14T21:57:27.1700877Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1701227Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1701527Z return mod(**inputs) 2025-08-14T21:57:27.1701859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1702206Z outputs = self.model( 2025-08-14T21:57:27.1702530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1702877Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1703188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1703515Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1703871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1704262Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1704431Z 2025-08-14T21:57:27.1704528Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1704863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1705171Z return mod(**inputs) 2025-08-14T21:57:27.1705497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1705848Z outputs = self.model( 2025-08-14T21:57:27.1706178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1706526Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1706853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1707188Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1707543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1707934Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1708300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.1708623Z return self.act(input) 2025-08-14T21:57:27.1708726Z 2025-08-14T21:57:27.1708830Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1709161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1709465Z return mod(**inputs) 2025-08-14T21:57:27.1709793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1710137Z outputs = self.model( 2025-08-14T21:57:27.1710472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1710829Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1711183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1711521Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1711889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.1712284Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.1712416Z 2025-08-14T21:57:27.1712515Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1712863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1713176Z return mod(**inputs) 2025-08-14T21:57:27.1713527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1713874Z outputs = self.model( 2025-08-14T21:57:27.1714226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1714591Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1714930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1715274Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1715631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-14T21:57:27.1715995Z hidden_states = residual + hidden_states 2025-08-14T21:57:27.1716125Z 2025-08-14T21:57:27.1716222Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1716559Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1716864Z return mod(**inputs) 2025-08-14T21:57:27.1717206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1717570Z outputs = self.model( 2025-08-14T21:57:27.1717899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1718257Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1718576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1718909Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1719267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1719652Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1720030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1720432Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1720588Z 2025-08-14T21:57:27.1720694Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1721034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1721347Z return mod(**inputs) 2025-08-14T21:57:27.1721732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1722084Z outputs = self.model( 2025-08-14T21:57:27.1722410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1722487Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1722694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1722768Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1723007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1723121Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1723362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.1723455Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.1723459Z 2025-08-14T21:57:27.1723555Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1723751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1723813Z return mod(**inputs) 2025-08-14T21:57:27.1724053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1724117Z outputs = self.model( 2025-08-14T21:57:27.1724349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1724452Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1724666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1724758Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1725004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1725097Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1725344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1725521Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1725528Z 2025-08-14T21:57:27.1725633Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1725839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1725908Z return mod(**inputs) 2025-08-14T21:57:27.1726160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1726230Z outputs = self.model( 2025-08-14T21:57:27.1726508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1726586Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1726792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1726865Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1727104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1727196Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1727436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.1727564Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.1727569Z 2025-08-14T21:57:27.1727664Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1727860Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1727923Z return mod(**inputs) 2025-08-14T21:57:27.1728160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1728224Z outputs = self.model( 2025-08-14T21:57:27.1728453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1728527Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1728733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1728807Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1729069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1729160Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1729413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.1729494Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.1729498Z 2025-08-14T21:57:27.1729593Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1729785Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1729848Z return mod(**inputs) 2025-08-14T21:57:27.1730079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1730150Z outputs = self.model( 2025-08-14T21:57:27.1730395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1730473Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1730699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1730774Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1731008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1731096Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1731331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.1731420Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.1731424Z 2025-08-14T21:57:27.1731518Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1731710Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1731773Z return mod(**inputs) 2025-08-14T21:57:27.1732002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1732076Z outputs = self.model( 2025-08-14T21:57:27.1732302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1732378Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1732580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1732652Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1732885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1732975Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1733210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.1733330Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.1733333Z 2025-08-14T21:57:27.1733429Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1733618Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1733683Z return mod(**inputs) 2025-08-14T21:57:27.1733912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1733982Z outputs = self.model( 2025-08-14T21:57:27.1734210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1734286Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1734491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1734582Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1734826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1734933Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1735170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.1735264Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.1735267Z 2025-08-14T21:57:27.1735363Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1735554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1735614Z return mod(**inputs) 2025-08-14T21:57:27.1735856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1735931Z outputs = self.model( 2025-08-14T21:57:27.1736176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1736252Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1736457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1736529Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1736765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1736878Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1736881Z 2025-08-14T21:57:27.1736977Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1737172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1737235Z return mod(**inputs) 2025-08-14T21:57:27.1737475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1737541Z outputs = self.model( 2025-08-14T21:57:27.1737956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1738036Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1738243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1738325Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1738553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1738665Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1738875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.1738942Z return self.act(input) 2025-08-14T21:57:27.1738946Z 2025-08-14T21:57:27.1739044Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1739239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1739303Z return mod(**inputs) 2025-08-14T21:57:27.1739539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1739602Z outputs = self.model( 2025-08-14T21:57:27.1739832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1739906Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1740112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1740187Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1740469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.1740548Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.1740553Z 2025-08-14T21:57:27.1740693Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1740881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1740943Z return mod(**inputs) 2025-08-14T21:57:27.1741183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1741248Z outputs = self.model( 2025-08-14T21:57:27.1741487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1741555Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1741781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1741864Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1742119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1742212Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1742450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1742565Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1742569Z 2025-08-14T21:57:27.1742668Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1742850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1742910Z return mod(**inputs) 2025-08-14T21:57:27.1743140Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1743203Z outputs = self.model( 2025-08-14T21:57:27.1743427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1743502Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1743704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1743785Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1744011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1744101Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1744336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.1744414Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.1744418Z 2025-08-14T21:57:27.1744521Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1744706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1744769Z return mod(**inputs) 2025-08-14T21:57:27.1745011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1745074Z outputs = self.model( 2025-08-14T21:57:27.1745294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1745369Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1745567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1745646Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1745867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1745969Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1746200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1746345Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1746348Z 2025-08-14T21:57:27.1746447Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1746629Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1746688Z return mod(**inputs) 2025-08-14T21:57:27.1746918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1746979Z outputs = self.model( 2025-08-14T21:57:27.1747221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1747298Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1747507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1747607Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1747841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1747931Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1748166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.1748290Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.1748294Z 2025-08-14T21:57:27.1748398Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1748585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1748647Z return mod(**inputs) 2025-08-14T21:57:27.1748885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1748950Z outputs = self.model( 2025-08-14T21:57:27.1749181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1749256Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1749461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1749542Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1749774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1749861Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1750091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.1750170Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.1750173Z 2025-08-14T21:57:27.1750265Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1750456Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1750518Z return mod(**inputs) 2025-08-14T21:57:27.1750748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1750811Z outputs = self.model( 2025-08-14T21:57:27.1751034Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1751107Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1751306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1751383Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1751626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1751713Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1751960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.1752047Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.1752051Z 2025-08-14T21:57:27.1752142Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1752330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1752390Z return mod(**inputs) 2025-08-14T21:57:27.1752616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1752696Z outputs = self.model( 2025-08-14T21:57:27.1752922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1752997Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1753216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1753291Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1753519Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1753607Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1753836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.1753948Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.1753952Z 2025-08-14T21:57:27.1754046Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1754239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1754303Z return mod(**inputs) 2025-08-14T21:57:27.1754539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1754602Z outputs = self.model( 2025-08-14T21:57:27.1754830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1754903Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1755105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1755178Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1755411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1755502Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1755737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.1755825Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.1755829Z 2025-08-14T21:57:27.1755919Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1756106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1756165Z return mod(**inputs) 2025-08-14T21:57:27.1756392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1756453Z outputs = self.model( 2025-08-14T21:57:27.1756674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1756746Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1756944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1757032Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1757266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1757389Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1757392Z 2025-08-14T21:57:27.1757491Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1757673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1757733Z return mod(**inputs) 2025-08-14T21:57:27.1757968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1758029Z outputs = self.model( 2025-08-14T21:57:27.1758287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1758357Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1758579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1758662Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1758894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1759002Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1759207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.1759272Z return self.act(input) 2025-08-14T21:57:27.1759275Z 2025-08-14T21:57:27.1759376Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1759563Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1759626Z return mod(**inputs) 2025-08-14T21:57:27.1759865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1759931Z outputs = self.model( 2025-08-14T21:57:27.1760160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1760235Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1760440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1760520Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1760751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.1760825Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.1760828Z 2025-08-14T21:57:27.1760937Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1761121Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1761192Z return mod(**inputs) 2025-08-14T21:57:27.1761425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1761488Z outputs = self.model( 2025-08-14T21:57:27.1761724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1761792Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1761995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1762075Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1762305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-14T21:57:27.1762387Z hidden_states = residual + hidden_states 2025-08-14T21:57:27.1762407Z 2025-08-14T21:57:27.1762504Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1762695Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1762781Z return mod(**inputs) 2025-08-14T21:57:27.1763018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1763081Z outputs = self.model( 2025-08-14T21:57:27.1763325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1763392Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1763607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1763679Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1763933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1764037Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1764288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1764404Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1764407Z 2025-08-14T21:57:27.1764504Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1764693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1764763Z return mod(**inputs) 2025-08-14T21:57:27.1764999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1765063Z outputs = self.model( 2025-08-14T21:57:27.1765312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1765384Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1765679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1765761Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1766008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1766113Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1766355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.1766450Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.1766454Z 2025-08-14T21:57:27.1766550Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1766735Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1766807Z return mod(**inputs) 2025-08-14T21:57:27.1767039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1767104Z outputs = self.model( 2025-08-14T21:57:27.1767340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1767407Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1767619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1767691Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1767919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1768018Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1768245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1768376Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1768385Z 2025-08-14T21:57:27.1768479Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1768683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1768750Z return mod(**inputs) 2025-08-14T21:57:27.1768978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1769040Z outputs = self.model( 2025-08-14T21:57:27.1769276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1769344Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1769570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1769646Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1769875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1769988Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1770228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.1770352Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.1770363Z 2025-08-14T21:57:27.1770459Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1770646Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1770715Z return mod(**inputs) 2025-08-14T21:57:27.1770945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1771011Z outputs = self.model( 2025-08-14T21:57:27.1771250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1771320Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1771533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1771606Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1771835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1771930Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1772159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.1772239Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.1772242Z 2025-08-14T21:57:27.1772345Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1772535Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1772604Z return mod(**inputs) 2025-08-14T21:57:27.1772836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1772902Z outputs = self.model( 2025-08-14T21:57:27.1773141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1773209Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1773415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1773494Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1773724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1773823Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1774070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.1774159Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.1774178Z 2025-08-14T21:57:27.1774282Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1774468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1774538Z return mod(**inputs) 2025-08-14T21:57:27.1774770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1774833Z outputs = self.model( 2025-08-14T21:57:27.1775068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1775150Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1775357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1775437Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1775742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1775843Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1776074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.1776192Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.1776196Z 2025-08-14T21:57:27.1776297Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1776482Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1776552Z return mod(**inputs) 2025-08-14T21:57:27.1776783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1776848Z outputs = self.model( 2025-08-14T21:57:27.1777088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1777157Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1777362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1777441Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1777675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1777773Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1778003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.1778080Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.1778085Z 2025-08-14T21:57:27.1778187Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1778373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1778443Z return mod(**inputs) 2025-08-14T21:57:27.1778675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1778738Z outputs = self.model( 2025-08-14T21:57:27.1778975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1779042Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1779244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1779325Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1779567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1779699Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1779703Z 2025-08-14T21:57:27.1779797Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1779993Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1780061Z return mod(**inputs) 2025-08-14T21:57:27.1780284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1780346Z outputs = self.model( 2025-08-14T21:57:27.1780577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1780642Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1780863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1780935Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1781173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1781289Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1781482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.1781551Z return self.act(input) 2025-08-14T21:57:27.1781554Z 2025-08-14T21:57:27.1781644Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1781821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1781887Z return mod(**inputs) 2025-08-14T21:57:27.1782117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1782180Z outputs = self.model( 2025-08-14T21:57:27.1782418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1782484Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1782696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1782769Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1782995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.1783075Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.1783079Z 2025-08-14T21:57:27.1783172Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1783353Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1783420Z return mod(**inputs) 2025-08-14T21:57:27.1783701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1783773Z outputs = self.model( 2025-08-14T21:57:27.1784008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1784077Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1784293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1784367Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1784603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1784694Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1784924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1785036Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1785058Z 2025-08-14T21:57:27.1785158Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1785347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1785444Z return mod(**inputs) 2025-08-14T21:57:27.1785674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1785744Z outputs = self.model( 2025-08-14T21:57:27.1785972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1786038Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1786249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1786323Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1786566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1786669Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1786929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.1787011Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.1787014Z 2025-08-14T21:57:27.1787107Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1787289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1787356Z return mod(**inputs) 2025-08-14T21:57:27.1787578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1787647Z outputs = self.model( 2025-08-14T21:57:27.1787872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1787939Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1788146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1788218Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1788439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1788534Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1788755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1788860Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1788863Z 2025-08-14T21:57:27.1788956Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1789143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1789213Z return mod(**inputs) 2025-08-14T21:57:27.1789441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1789513Z outputs = self.model( 2025-08-14T21:57:27.1789743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1789810Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1790020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1790101Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1790323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1790421Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1790642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.1790791Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.1790794Z 2025-08-14T21:57:27.1790889Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1791085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1791155Z return mod(**inputs) 2025-08-14T21:57:27.1791380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1791448Z outputs = self.model( 2025-08-14T21:57:27.1791673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1791742Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1791973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1792049Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1792293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1792395Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1792625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.1792711Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.1792715Z 2025-08-14T21:57:27.1792810Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1792995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1793064Z return mod(**inputs) 2025-08-14T21:57:27.1793294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1793356Z outputs = self.model( 2025-08-14T21:57:27.1793588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1793656Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1793867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1793940Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1794163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1794259Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1794485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.1794579Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.1794583Z 2025-08-14T21:57:27.1794678Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1794864Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1794932Z return mod(**inputs) 2025-08-14T21:57:27.1795159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1795225Z outputs = self.model( 2025-08-14T21:57:27.1795458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1795525Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1795735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1795808Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1796036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1796133Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1796406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.1796530Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.1796551Z 2025-08-14T21:57:27.1796648Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1796835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1796903Z return mod(**inputs) 2025-08-14T21:57:27.1797135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1797200Z outputs = self.model( 2025-08-14T21:57:27.1797438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1797523Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1797748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1797823Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1798073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1798178Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1798412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.1798489Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.1798500Z 2025-08-14T21:57:27.1798596Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1798794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1798862Z return mod(**inputs) 2025-08-14T21:57:27.1799088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1799152Z outputs = self.model( 2025-08-14T21:57:27.1799387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1799456Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1799665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1799737Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1799962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1800078Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1800081Z 2025-08-14T21:57:27.1800176Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1800361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1800431Z return mod(**inputs) 2025-08-14T21:57:27.1800666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1800741Z outputs = self.model( 2025-08-14T21:57:27.1800974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1801044Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1801265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1801336Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1801570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1801678Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1801874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.1801966Z return self.act(input) 2025-08-14T21:57:27.1801969Z 2025-08-14T21:57:27.1802065Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1802268Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1802335Z return mod(**inputs) 2025-08-14T21:57:27.1802563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1802632Z outputs = self.model( 2025-08-14T21:57:27.1802860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1802927Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1803153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1803227Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1803468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.1803555Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.1803559Z 2025-08-14T21:57:27.1803655Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1803853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1803915Z return mod(**inputs) 2025-08-14T21:57:27.1804148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1804218Z outputs = self.model( 2025-08-14T21:57:27.1804452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1804530Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1804745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1804822Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1805073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-14T21:57:27.1805154Z hidden_states = residual + hidden_states 2025-08-14T21:57:27.1805157Z 2025-08-14T21:57:27.1805256Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1805534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1805609Z return mod(**inputs) 2025-08-14T21:57:27.1805860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1805929Z outputs = self.model( 2025-08-14T21:57:27.1806176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1806257Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1806469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1806559Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1806797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1806889Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1807125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1807229Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1807233Z 2025-08-14T21:57:27.1807326Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1807523Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1807610Z return mod(**inputs) 2025-08-14T21:57:27.1807851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1807931Z outputs = self.model( 2025-08-14T21:57:27.1808160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1808239Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1808440Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1808513Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1808746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1808858Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1809096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.1809175Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.1809179Z 2025-08-14T21:57:27.1809290Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1809489Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1809553Z return mod(**inputs) 2025-08-14T21:57:27.1809801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1809866Z outputs = self.model( 2025-08-14T21:57:27.1810092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1810166Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1810370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1810443Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1810679Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1810770Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1811005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1811106Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1811109Z 2025-08-14T21:57:27.1811205Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1811395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1811457Z return mod(**inputs) 2025-08-14T21:57:27.1811683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1811754Z outputs = self.model( 2025-08-14T21:57:27.1811980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1812056Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1812262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1812333Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1812570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1812662Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1812898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.1813025Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.1813030Z 2025-08-14T21:57:27.1813126Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1813338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1813402Z return mod(**inputs) 2025-08-14T21:57:27.1813648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1813720Z outputs = self.model( 2025-08-14T21:57:27.1813947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1814023Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1814227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1814301Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1814556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1814650Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1814898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.1814981Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.1814984Z 2025-08-14T21:57:27.1815080Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1815271Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1815332Z return mod(**inputs) 2025-08-14T21:57:27.1815562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1815637Z outputs = self.model( 2025-08-14T21:57:27.1815867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1815941Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1816148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1816221Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1816457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1816547Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1816775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.1816871Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.1816874Z 2025-08-14T21:57:27.1816968Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1817171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1817232Z return mod(**inputs) 2025-08-14T21:57:27.1817454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1817525Z outputs = self.model( 2025-08-14T21:57:27.1817745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1817819Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1818021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1818093Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1818328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1818417Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1818651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.1818787Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.1818790Z 2025-08-14T21:57:27.1818881Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1819070Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1819730Z return mod(**inputs) 2025-08-14T21:57:27.1819956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1820028Z outputs = self.model( 2025-08-14T21:57:27.1820252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1820328Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1820531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1820621Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1820858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1820967Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1821191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.1821277Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.1821281Z 2025-08-14T21:57:27.1821376Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1821574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1821634Z return mod(**inputs) 2025-08-14T21:57:27.1821867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1821937Z outputs = self.model( 2025-08-14T21:57:27.1822172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1822242Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1822460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1822533Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1822771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1822878Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1822882Z 2025-08-14T21:57:27.1822976Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1823167Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1823227Z return mod(**inputs) 2025-08-14T21:57:27.1823471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1823536Z outputs = self.model( 2025-08-14T21:57:27.1823777Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1823851Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1824054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1824125Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1824357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1824461Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1824661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.1824726Z return self.act(input) 2025-08-14T21:57:27.1824731Z 2025-08-14T21:57:27.1824827Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1825040Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1825103Z return mod(**inputs) 2025-08-14T21:57:27.1825356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1825418Z outputs = self.model( 2025-08-14T21:57:27.1825647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1825720Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1825925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1825997Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1826254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.1826331Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.1826334Z 2025-08-14T21:57:27.1826435Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1826630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1826693Z return mod(**inputs) 2025-08-14T21:57:27.1826928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1826990Z outputs = self.model( 2025-08-14T21:57:27.1827213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1827285Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1827486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1827568Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1827793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1827884Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1828118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1828220Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1828223Z 2025-08-14T21:57:27.1828323Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1828499Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1828558Z return mod(**inputs) 2025-08-14T21:57:27.1828785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1828847Z outputs = self.model( 2025-08-14T21:57:27.1829071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1829147Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1829368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1829448Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1829672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1829760Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1829988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.1830060Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.1830063Z 2025-08-14T21:57:27.1830161Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1830341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1830417Z return mod(**inputs) 2025-08-14T21:57:27.1830650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1830739Z outputs = self.model( 2025-08-14T21:57:27.1830963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1831039Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1831235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1831312Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1831535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1831638Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1831870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1831971Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1831989Z 2025-08-14T21:57:27.1832091Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1832272Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1832334Z return mod(**inputs) 2025-08-14T21:57:27.1832567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1832631Z outputs = self.model( 2025-08-14T21:57:27.1832856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1832933Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1833137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1833219Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1833448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1833540Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1833771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.1833898Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.1833902Z 2025-08-14T21:57:27.1834000Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1834193Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1834256Z return mod(**inputs) 2025-08-14T21:57:27.1834495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1834562Z outputs = self.model( 2025-08-14T21:57:27.1834794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1834873Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1835080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1835162Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1835398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1835489Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1835724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.1835809Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.1835812Z 2025-08-14T21:57:27.1835937Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1836128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1836190Z return mod(**inputs) 2025-08-14T21:57:27.1836439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1836503Z outputs = self.model( 2025-08-14T21:57:27.1836731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1836806Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1837008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1837079Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1837333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1837428Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1837918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.1838016Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.1838019Z 2025-08-14T21:57:27.1838116Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1838311Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1838373Z return mod(**inputs) 2025-08-14T21:57:27.1838609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1838673Z outputs = self.model( 2025-08-14T21:57:27.1838909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1838983Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1839188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1839262Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1839498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1839589Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1839828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.1839946Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.1839949Z 2025-08-14T21:57:27.1840045Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1840239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1840301Z return mod(**inputs) 2025-08-14T21:57:27.1840540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1840604Z outputs = self.model( 2025-08-14T21:57:27.1840834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1840910Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1841116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1841188Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1841424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1841514Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1841750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.1841853Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.1841856Z 2025-08-14T21:57:27.1841950Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1842143Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1842230Z return mod(**inputs) 2025-08-14T21:57:27.1842468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1842530Z outputs = self.model( 2025-08-14T21:57:27.1842759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1842834Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1843039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1843143Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1843386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1843514Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1843519Z 2025-08-14T21:57:27.1843624Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1843808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1843870Z return mod(**inputs) 2025-08-14T21:57:27.1844109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1844174Z outputs = self.model( 2025-08-14T21:57:27.1844410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1844487Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1844698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1844781Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1845012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1845124Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1845332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.1845441Z return self.act(input) 2025-08-14T21:57:27.1845447Z 2025-08-14T21:57:27.1845563Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1845754Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1845817Z return mod(**inputs) 2025-08-14T21:57:27.1846062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1846128Z outputs = self.model( 2025-08-14T21:57:27.1846367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1846446Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1846655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1846736Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1846972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.1847050Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.1847053Z 2025-08-14T21:57:27.1847159Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1847347Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1847410Z return mod(**inputs) 2025-08-14T21:57:27.1847678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1847745Z outputs = self.model( 2025-08-14T21:57:27.1847988Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1848074Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1848295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1848375Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1848608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-14T21:57:27.1848689Z hidden_states = residual + hidden_states 2025-08-14T21:57:27.1848692Z 2025-08-14T21:57:27.1848802Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1848992Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1849061Z return mod(**inputs) 2025-08-14T21:57:27.1849306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1849370Z outputs = self.model( 2025-08-14T21:57:27.1849608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1849675Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1849885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1849957Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1850193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1850293Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1850531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1850647Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1850661Z 2025-08-14T21:57:27.1850756Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1850940Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1851006Z return mod(**inputs) 2025-08-14T21:57:27.1851234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1851297Z outputs = self.model( 2025-08-14T21:57:27.1851532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1851599Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1851860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1851936Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1852166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1852265Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1852495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.1852570Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.1852581Z 2025-08-14T21:57:27.1852675Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1852861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1852931Z return mod(**inputs) 2025-08-14T21:57:27.1853160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1853251Z outputs = self.model( 2025-08-14T21:57:27.1853494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1853579Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1853792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1853864Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1854093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1854191Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1854428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1854548Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1854553Z 2025-08-14T21:57:27.1854660Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1854908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1854981Z return mod(**inputs) 2025-08-14T21:57:27.1855217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1855280Z outputs = self.model( 2025-08-14T21:57:27.1855522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1855589Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1855799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1855881Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1856115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1856215Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1856450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.1856575Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.1856578Z 2025-08-14T21:57:27.1856684Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1856872Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1856939Z return mod(**inputs) 2025-08-14T21:57:27.1857172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1857233Z outputs = self.model( 2025-08-14T21:57:27.1857473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1857542Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1857753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1857835Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1858067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1858164Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1858399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.1858480Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.1858484Z 2025-08-14T21:57:27.1858595Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1858779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1858861Z return mod(**inputs) 2025-08-14T21:57:27.1859088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1859153Z outputs = self.model( 2025-08-14T21:57:27.1859402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1859468Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1859670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1859752Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1859980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1860074Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1860317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.1860410Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.1860414Z 2025-08-14T21:57:27.1860531Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1860717Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1860787Z return mod(**inputs) 2025-08-14T21:57:27.1861020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1861081Z outputs = self.model( 2025-08-14T21:57:27.1861309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1861377Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1861576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1861656Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1861881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1861978Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1862200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.1862312Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.1862315Z 2025-08-14T21:57:27.1862414Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1862590Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1862650Z return mod(**inputs) 2025-08-14T21:57:27.1862886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1862949Z outputs = self.model( 2025-08-14T21:57:27.1863184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1863253Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1863455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1863537Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1863763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1863857Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1864083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.1864157Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.1864160Z 2025-08-14T21:57:27.1864262Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1864463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1864523Z return mod(**inputs) 2025-08-14T21:57:27.1864760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1864839Z outputs = self.model( 2025-08-14T21:57:27.1865079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1865146Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1865347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1865425Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1865653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1865782Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1865787Z 2025-08-14T21:57:27.1865883Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1866081Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1866160Z return mod(**inputs) 2025-08-14T21:57:27.1866387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1866450Z outputs = self.model( 2025-08-14T21:57:27.1866682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1866748Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1866954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1867025Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1867248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1867361Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1867555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.1867622Z return self.act(input) 2025-08-14T21:57:27.1867633Z 2025-08-14T21:57:27.1867727Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1867908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1867976Z return mod(**inputs) 2025-08-14T21:57:27.1868208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1868271Z outputs = self.model( 2025-08-14T21:57:27.1868508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1868576Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1868789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1868864Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1869093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.1869178Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.1869181Z 2025-08-14T21:57:27.1869273Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1869458Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1869537Z return mod(**inputs) 2025-08-14T21:57:27.1869759Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1869829Z outputs = self.model( 2025-08-14T21:57:27.1870072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1870139Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1870348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1870434Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1870662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1870765Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1870990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1871095Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1871098Z 2025-08-14T21:57:27.1871206Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1871389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1871456Z return mod(**inputs) 2025-08-14T21:57:27.1871698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1871770Z outputs = self.model( 2025-08-14T21:57:27.1871992Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1872059Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1872265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1872337Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1872565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1872663Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1872894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.1872979Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.1872983Z 2025-08-14T21:57:27.1873088Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1873269Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1873339Z return mod(**inputs) 2025-08-14T21:57:27.1873563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1873634Z outputs = self.model( 2025-08-14T21:57:27.1873864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1873933Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1874143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1874219Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1874452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1874553Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1874787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1874897Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1874900Z 2025-08-14T21:57:27.1874998Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1875187Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1875258Z return mod(**inputs) 2025-08-14T21:57:27.1875501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1875590Z outputs = self.model( 2025-08-14T21:57:27.1875820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1875904Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1876126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1876197Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1876420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1876513Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1876748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.1876900Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.1876906Z 2025-08-14T21:57:27.1877004Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1877207Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1877279Z return mod(**inputs) 2025-08-14T21:57:27.1877517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1877582Z outputs = self.model( 2025-08-14T21:57:27.1877822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1877889Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1878106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1878180Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1878414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1878515Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1878753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.1878842Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.1878846Z 2025-08-14T21:57:27.1878945Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1879135Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1879206Z return mod(**inputs) 2025-08-14T21:57:27.1879444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1879508Z outputs = self.model( 2025-08-14T21:57:27.1879751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1879821Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1880041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1880117Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1880357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1880457Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1880695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.1880795Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.1880799Z 2025-08-14T21:57:27.1880893Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1881084Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1881172Z return mod(**inputs) 2025-08-14T21:57:27.1881408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1881474Z outputs = self.model( 2025-08-14T21:57:27.1881737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1881807Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1882024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1882098Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1882332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1882434Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1882685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.1882810Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.1882820Z 2025-08-14T21:57:27.1882933Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1883126Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1883197Z return mod(**inputs) 2025-08-14T21:57:27.1883432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1883497Z outputs = self.model( 2025-08-14T21:57:27.1883739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1883807Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1884024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1884100Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1884335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1884434Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1884670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.1884749Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.1884760Z 2025-08-14T21:57:27.1884857Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1885046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1885114Z return mod(**inputs) 2025-08-14T21:57:27.1885352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1885539Z outputs = self.model( 2025-08-14T21:57:27.1885798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1885869Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1886091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1886168Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1886404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1886524Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1886527Z 2025-08-14T21:57:27.1886628Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1886819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1886893Z return mod(**inputs) 2025-08-14T21:57:27.1887127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1887217Z outputs = self.model( 2025-08-14T21:57:27.1887454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1887540Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1887760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1887836Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1888071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1888189Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1888412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.1888487Z return self.act(input) 2025-08-14T21:57:27.1888493Z 2025-08-14T21:57:27.1888591Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1888803Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1888877Z return mod(**inputs) 2025-08-14T21:57:27.1889128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1889200Z outputs = self.model( 2025-08-14T21:57:27.1889436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1889503Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1889719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1889790Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1890027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.1890111Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.1890114Z 2025-08-14T21:57:27.1890208Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1890398Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1890459Z return mod(**inputs) 2025-08-14T21:57:27.1890695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1890767Z outputs = self.model( 2025-08-14T21:57:27.1891002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1891067Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1891285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1891357Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1891656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-14T21:57:27.1891730Z hidden_states = residual + hidden_states 2025-08-14T21:57:27.1891734Z 2025-08-14T21:57:27.1891827Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1892023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1892083Z return mod(**inputs) 2025-08-14T21:57:27.1892325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1892391Z outputs = self.model( 2025-08-14T21:57:27.1892625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1892700Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1892927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1892999Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1893242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1893349Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1893586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1893687Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1893690Z 2025-08-14T21:57:27.1893785Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1893979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1894062Z return mod(**inputs) 2025-08-14T21:57:27.1894302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1894368Z outputs = self.model( 2025-08-14T21:57:27.1894616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1894693Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1894899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1894971Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1895208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1895298Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1895536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.1895612Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.1895617Z 2025-08-14T21:57:27.1895711Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1895900Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1895964Z return mod(**inputs) 2025-08-14T21:57:27.1896195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1896269Z outputs = self.model( 2025-08-14T21:57:27.1896496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1896572Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1896776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1896848Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1897083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1897175Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1897412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1897514Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1897518Z 2025-08-14T21:57:27.1897612Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1897802Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1897862Z return mod(**inputs) 2025-08-14T21:57:27.1898147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1898218Z outputs = self.model( 2025-08-14T21:57:27.1898448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1898542Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1898751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1898839Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1899074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1899164Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1899402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.1899522Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.1899525Z 2025-08-14T21:57:27.1899619Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1899826Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1899890Z return mod(**inputs) 2025-08-14T21:57:27.1900132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1900204Z outputs = self.model( 2025-08-14T21:57:27.1900439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1900515Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1900721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1900792Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1901025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1901119Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1901344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.1901433Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.1901438Z 2025-08-14T21:57:27.1901533Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1901725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1901786Z return mod(**inputs) 2025-08-14T21:57:27.1902016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1902088Z outputs = self.model( 2025-08-14T21:57:27.1902319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1902391Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1902596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1902669Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1902907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1902997Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1903227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.1903323Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.1903327Z 2025-08-14T21:57:27.1903420Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1903609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1903670Z return mod(**inputs) 2025-08-14T21:57:27.1903899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1903988Z outputs = self.model( 2025-08-14T21:57:27.1904218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1904293Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1904520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1904593Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1904830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1904918Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1905147Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.1905269Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.1905287Z 2025-08-14T21:57:27.1905382Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1905574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1905651Z return mod(**inputs) 2025-08-14T21:57:27.1905882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1905956Z outputs = self.model( 2025-08-14T21:57:27.1906186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1906259Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1906466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1906539Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1906775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1906866Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1907093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.1907176Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.1907179Z 2025-08-14T21:57:27.1907272Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1907462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1907523Z return mod(**inputs) 2025-08-14T21:57:27.1907751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1907820Z outputs = self.model( 2025-08-14T21:57:27.1908050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1908117Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1908331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1908405Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1908641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1908749Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1908753Z 2025-08-14T21:57:27.1908859Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1909046Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1909106Z return mod(**inputs) 2025-08-14T21:57:27.1909337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1909399Z outputs = self.model( 2025-08-14T21:57:27.1909621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1909714Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1909918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1910004Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1910237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1910343Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1910543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.1910608Z return self.act(input) 2025-08-14T21:57:27.1910612Z 2025-08-14T21:57:27.1910704Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1910907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1910969Z return mod(**inputs) 2025-08-14T21:57:27.1911209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1911282Z outputs = self.model( 2025-08-14T21:57:27.1911513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1911588Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1911794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1911865Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1912101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.1912177Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.1912182Z 2025-08-14T21:57:27.1912282Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1912467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1912530Z return mod(**inputs) 2025-08-14T21:57:27.1912766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1912828Z outputs = self.model( 2025-08-14T21:57:27.1913061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1913134Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1913330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1913409Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1913631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1913719Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1913949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1914048Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1914052Z 2025-08-14T21:57:27.1914151Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1914330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1914388Z return mod(**inputs) 2025-08-14T21:57:27.1914618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1914680Z outputs = self.model( 2025-08-14T21:57:27.1914908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1914985Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1915449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1915530Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1915780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1915870Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1916105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.1916178Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.1916181Z 2025-08-14T21:57:27.1916281Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1916461Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1916539Z return mod(**inputs) 2025-08-14T21:57:27.1916769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1916832Z outputs = self.model( 2025-08-14T21:57:27.1917076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1917153Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1917357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1917435Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1917660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1917746Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1917978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1918077Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1918080Z 2025-08-14T21:57:27.1918178Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1918360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1918419Z return mod(**inputs) 2025-08-14T21:57:27.1918652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1918715Z outputs = self.model( 2025-08-14T21:57:27.1918937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1919009Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1919210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1919290Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1919517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1919606Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1919841Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.1919962Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.1919965Z 2025-08-14T21:57:27.1920065Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1920245Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1920304Z return mod(**inputs) 2025-08-14T21:57:27.1920537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1920600Z outputs = self.model( 2025-08-14T21:57:27.1920824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1920913Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1921116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1921210Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1921436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1921524Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1921757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.1921834Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.1921837Z 2025-08-14T21:57:27.1921931Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1922137Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1922202Z return mod(**inputs) 2025-08-14T21:57:27.1922467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1922531Z outputs = self.model( 2025-08-14T21:57:27.1922755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1922829Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1923030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1923102Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1923334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1923422Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1923652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.1923739Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.1923744Z 2025-08-14T21:57:27.1923837Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1924030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1924091Z return mod(**inputs) 2025-08-14T21:57:27.1924328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1924394Z outputs = self.model( 2025-08-14T21:57:27.1924623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1924699Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1924906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1924981Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1925223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1925312Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1925616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.1925741Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.1925744Z 2025-08-14T21:57:27.1925840Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1926037Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1926100Z return mod(**inputs) 2025-08-14T21:57:27.1926344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1926431Z outputs = self.model( 2025-08-14T21:57:27.1926673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1926768Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1926980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1927055Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1927307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1927397Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1927629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.1927721Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.1927725Z 2025-08-14T21:57:27.1927819Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1928007Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1928085Z return mod(**inputs) 2025-08-14T21:57:27.1928321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1928386Z outputs = self.model( 2025-08-14T21:57:27.1928619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1928695Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1928903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1928977Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1929220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1929331Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1929335Z 2025-08-14T21:57:27.1929443Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1929631Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1929694Z return mod(**inputs) 2025-08-14T21:57:27.1929937Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1930002Z outputs = self.model( 2025-08-14T21:57:27.1930239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1930315Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1930527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1930608Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1930839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1930950Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1931163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.1931229Z return self.act(input) 2025-08-14T21:57:27.1931233Z 2025-08-14T21:57:27.1931337Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1931529Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1931592Z return mod(**inputs) 2025-08-14T21:57:27.1931834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1931899Z outputs = self.model( 2025-08-14T21:57:27.1932135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1932230Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1932443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1932540Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1932775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.1932852Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.1932855Z 2025-08-14T21:57:27.1932957Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1933142Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1933205Z return mod(**inputs) 2025-08-14T21:57:27.1933462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1933530Z outputs = self.model( 2025-08-14T21:57:27.1933799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1933872Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1934080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1934161Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1934397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-14T21:57:27.1934479Z hidden_states = residual + hidden_states 2025-08-14T21:57:27.1934483Z 2025-08-14T21:57:27.1934578Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1934766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1934837Z return mod(**inputs) 2025-08-14T21:57:27.1935073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1935138Z outputs = self.model( 2025-08-14T21:57:27.1935381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1935450Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1935666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1935739Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1935975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1936074Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1936309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1936421Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1936425Z 2025-08-14T21:57:27.1936522Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1936712Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1936783Z return mod(**inputs) 2025-08-14T21:57:27.1937016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1937092Z outputs = self.model( 2025-08-14T21:57:27.1937319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1937386Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1937702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1937785Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1938064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1938162Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1938412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.1938487Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.1938499Z 2025-08-14T21:57:27.1938593Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1938775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1938844Z return mod(**inputs) 2025-08-14T21:57:27.1939067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1939151Z outputs = self.model( 2025-08-14T21:57:27.1939386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1939456Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1939686Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1939761Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1939986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1940080Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1940302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1940402Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1940411Z 2025-08-14T21:57:27.1940506Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1940687Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1940757Z return mod(**inputs) 2025-08-14T21:57:27.1940982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1941045Z outputs = self.model( 2025-08-14T21:57:27.1941276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1941342Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1941547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1941619Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1941840Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1941936Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1942160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.1942281Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.1942286Z 2025-08-14T21:57:27.1942387Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1942564Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1942632Z return mod(**inputs) 2025-08-14T21:57:27.1942858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1942919Z outputs = self.model( 2025-08-14T21:57:27.1943149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1943214Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1943416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1943509Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1943733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1943843Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1944066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.1944144Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.1944148Z 2025-08-14T21:57:27.1944245Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1944424Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1944489Z return mod(**inputs) 2025-08-14T21:57:27.1944727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1944790Z outputs = self.model( 2025-08-14T21:57:27.1945035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1945103Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1945302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1945379Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1945603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1945697Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1945918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.1946005Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.1946010Z 2025-08-14T21:57:27.1946109Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1946287Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1946354Z return mod(**inputs) 2025-08-14T21:57:27.1946572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1946633Z outputs = self.model( 2025-08-14T21:57:27.1946862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1946925Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1947123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1947202Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1947423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1947516Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1947739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.1947855Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.1947858Z 2025-08-14T21:57:27.1947957Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1948136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1948203Z return mod(**inputs) 2025-08-14T21:57:27.1948427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1948489Z outputs = self.model( 2025-08-14T21:57:27.1948719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1948803Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1949005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1949100Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1949323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1949417Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1949639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.1949713Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.1949716Z 2025-08-14T21:57:27.1949816Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1950011Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1950075Z return mod(**inputs) 2025-08-14T21:57:27.1950309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1950389Z outputs = self.model( 2025-08-14T21:57:27.1950626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1950693Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1950894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1950971Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1951197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1951309Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1951313Z 2025-08-14T21:57:27.1951406Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1951584Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1951651Z return mod(**inputs) 2025-08-14T21:57:27.1951878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1951939Z outputs = self.model( 2025-08-14T21:57:27.1952170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1952235Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1952441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1952510Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1952734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1952847Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1953042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.1953107Z return self.act(input) 2025-08-14T21:57:27.1953117Z 2025-08-14T21:57:27.1953211Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1953389Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1953457Z return mod(**inputs) 2025-08-14T21:57:27.1953680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1953742Z outputs = self.model( 2025-08-14T21:57:27.1953973Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1954040Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1954249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1954336Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1954560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.1954654Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.1954657Z 2025-08-14T21:57:27.1954749Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1954931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1954999Z return mod(**inputs) 2025-08-14T21:57:27.1955224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1955293Z outputs = self.model( 2025-08-14T21:57:27.1955529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1955597Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1955825Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1955899Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1956123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1956220Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1956443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1956549Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1956552Z 2025-08-14T21:57:27.1956644Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1956825Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1956893Z return mod(**inputs) 2025-08-14T21:57:27.1957117Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1957185Z outputs = self.model( 2025-08-14T21:57:27.1957408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1957475Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1957683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1957752Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1957971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1958066Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1958291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.1958372Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.1958375Z 2025-08-14T21:57:27.1958467Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1958645Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1958711Z return mod(**inputs) 2025-08-14T21:57:27.1958931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1958998Z outputs = self.model( 2025-08-14T21:57:27.1959222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1959287Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1959496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1959567Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1959808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1959904Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1960144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1960249Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1960253Z 2025-08-14T21:57:27.1960343Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1960524Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1960590Z return mod(**inputs) 2025-08-14T21:57:27.1960813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1960906Z outputs = self.model( 2025-08-14T21:57:27.1961130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1961199Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1961418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1961493Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1961717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1961817Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1962040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.1962168Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.1962171Z 2025-08-14T21:57:27.1962265Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1962449Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1962516Z return mod(**inputs) 2025-08-14T21:57:27.1962741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1962804Z outputs = self.model( 2025-08-14T21:57:27.1963039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1963106Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1963312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1963383Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1963607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1963703Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1963929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.1964015Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.1964019Z 2025-08-14T21:57:27.1964116Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1964304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1964373Z return mod(**inputs) 2025-08-14T21:57:27.1964606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1964668Z outputs = self.model( 2025-08-14T21:57:27.1964908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1964979Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1965192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1965285Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1965583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1965710Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1965951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.1966053Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.1966057Z 2025-08-14T21:57:27.1966157Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1966351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1966423Z return mod(**inputs) 2025-08-14T21:57:27.1966680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1966751Z outputs = self.model( 2025-08-14T21:57:27.1967021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1967094Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1967321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1967395Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1967627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1967728Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1967968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.1968092Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.1968103Z 2025-08-14T21:57:27.1968204Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1968447Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1968519Z return mod(**inputs) 2025-08-14T21:57:27.1968763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1968831Z outputs = self.model( 2025-08-14T21:57:27.1969080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1969152Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1969375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1969452Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1969695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1969799Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1970044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.1970125Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.1970137Z 2025-08-14T21:57:27.1970237Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1970430Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1970502Z return mod(**inputs) 2025-08-14T21:57:27.1970744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1970809Z outputs = self.model( 2025-08-14T21:57:27.1971060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1971149Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1971379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1971472Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1971720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1971843Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1971846Z 2025-08-14T21:57:27.1971946Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1972140Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1972213Z return mod(**inputs) 2025-08-14T21:57:27.1972472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1972547Z outputs = self.model( 2025-08-14T21:57:27.1972789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1972876Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1973101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1973175Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1973418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1973536Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1973741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.1973816Z return self.act(input) 2025-08-14T21:57:27.1973819Z 2025-08-14T21:57:27.1973920Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1974114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1974186Z return mod(**inputs) 2025-08-14T21:57:27.1974432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1974508Z outputs = self.model( 2025-08-14T21:57:27.1974750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1974820Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1975041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1975117Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1975364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.1975449Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.1975454Z 2025-08-14T21:57:27.1975552Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1975758Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1975823Z return mod(**inputs) 2025-08-14T21:57:27.1976066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1976142Z outputs = self.model( 2025-08-14T21:57:27.1976383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1976460Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1976667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1976737Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1976967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-14T21:57:27.1977058Z hidden_states = residual + hidden_states 2025-08-14T21:57:27.1977061Z 2025-08-14T21:57:27.1977155Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1977361Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1977422Z return mod(**inputs) 2025-08-14T21:57:27.1977655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1977717Z outputs = self.model( 2025-08-14T21:57:27.1977938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1978010Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1978232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1978305Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1978552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1978645Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1978878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1978978Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1978982Z 2025-08-14T21:57:27.1979075Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1979264Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1979322Z return mod(**inputs) 2025-08-14T21:57:27.1979566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1979627Z outputs = self.model( 2025-08-14T21:57:27.1979852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1979924Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1980128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1980199Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1980432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1980520Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1980755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.1980827Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.1980830Z 2025-08-14T21:57:27.1980923Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1981110Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1981174Z return mod(**inputs) 2025-08-14T21:57:27.1981404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1981467Z outputs = self.model( 2025-08-14T21:57:27.1981697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1981771Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1981971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1982042Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1982274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1982362Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1982612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1982710Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1982728Z 2025-08-14T21:57:27.1982822Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1983010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1983069Z return mod(**inputs) 2025-08-14T21:57:27.1983289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1983360Z outputs = self.model( 2025-08-14T21:57:27.1983581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1983670Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1983873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1983943Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1984192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1984284Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1984514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.1984634Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.1984638Z 2025-08-14T21:57:27.1984730Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1984918Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1984978Z return mod(**inputs) 2025-08-14T21:57:27.1985200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1985269Z outputs = self.model( 2025-08-14T21:57:27.1985494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1985570Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1985767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1985837Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1986064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1986149Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1986379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.1986457Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.1986462Z 2025-08-14T21:57:27.1986554Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1986739Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1986799Z return mod(**inputs) 2025-08-14T21:57:27.1987021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1987089Z outputs = self.model( 2025-08-14T21:57:27.1987310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1987384Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1987582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1987652Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1987882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1987985Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1988207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.1988317Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.1988320Z 2025-08-14T21:57:27.1988411Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1988597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1988657Z return mod(**inputs) 2025-08-14T21:57:27.1988880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1988949Z outputs = self.model( 2025-08-14T21:57:27.1989184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1989263Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1989483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1989556Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1989788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1989874Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1990097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.1990221Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.1990224Z 2025-08-14T21:57:27.1990316Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1990502Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1990563Z return mod(**inputs) 2025-08-14T21:57:27.1990786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1990860Z outputs = self.model( 2025-08-14T21:57:27.1991081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1991155Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1991353Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1991424Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1991656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1991744Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1991966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.1992049Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.1992052Z 2025-08-14T21:57:27.1992145Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1992331Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1992391Z return mod(**inputs) 2025-08-14T21:57:27.1992609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1992678Z outputs = self.model( 2025-08-14T21:57:27.1992898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1992964Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1993170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1993262Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1993495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1993618Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1993621Z 2025-08-14T21:57:27.1993714Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1993901Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1993961Z return mod(**inputs) 2025-08-14T21:57:27.1994191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1994252Z outputs = self.model( 2025-08-14T21:57:27.1994489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1994566Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1994769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1994856Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1995092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.1995198Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.1995397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.1995459Z return self.act(input) 2025-08-14T21:57:27.1995462Z 2025-08-14T21:57:27.1995555Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1995743Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1995805Z return mod(**inputs) 2025-08-14T21:57:27.1996040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1996102Z outputs = self.model( 2025-08-14T21:57:27.1996326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1996401Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1996603Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1996674Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1996905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.1996982Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.1996985Z 2025-08-14T21:57:27.1997083Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1997265Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1997326Z return mod(**inputs) 2025-08-14T21:57:27.1997561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1997622Z outputs = self.model( 2025-08-14T21:57:27.1997846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1997920Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.1998118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.1998193Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.1998417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.1998504Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.1998733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.1998851Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.1998857Z 2025-08-14T21:57:27.1998979Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.1999158Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.1999219Z return mod(**inputs) 2025-08-14T21:57:27.1999448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.1999509Z outputs = self.model( 2025-08-14T21:57:27.1999727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.1999800Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2000015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2000097Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2000333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2000422Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2000654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.2000725Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.2000728Z 2025-08-14T21:57:27.2000826Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2001006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2001065Z return mod(**inputs) 2025-08-14T21:57:27.2001298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2001360Z outputs = self.model( 2025-08-14T21:57:27.2001583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2001658Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2001860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2001937Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2002163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2002251Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2002483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.2002581Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.2002586Z 2025-08-14T21:57:27.2002679Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2002869Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2002930Z return mod(**inputs) 2025-08-14T21:57:27.2003162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2003223Z outputs = self.model( 2025-08-14T21:57:27.2003447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2003520Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2003764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2003843Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2004076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2004183Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2004420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.2004560Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.2004563Z 2025-08-14T21:57:27.2004660Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2004854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2004916Z return mod(**inputs) 2025-08-14T21:57:27.2005154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2005217Z outputs = self.model( 2025-08-14T21:57:27.2005533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2005617Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2005837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2005931Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2006186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2006282Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2006534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.2006618Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.2006623Z 2025-08-14T21:57:27.2006724Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2006938Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2007005Z return mod(**inputs) 2025-08-14T21:57:27.2007245Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2007310Z outputs = self.model( 2025-08-14T21:57:27.2007544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2007620Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2007827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2007899Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2008139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2008229Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2008470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.2008560Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.2008563Z 2025-08-14T21:57:27.2008658Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2008853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2008918Z return mod(**inputs) 2025-08-14T21:57:27.2009157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2009220Z outputs = self.model( 2025-08-14T21:57:27.2009450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2009526Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2009735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2009810Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2010069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2010162Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2010416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.2010533Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.2010536Z 2025-08-14T21:57:27.2010630Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2010822Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2010884Z return mod(**inputs) 2025-08-14T21:57:27.2011120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2011218Z outputs = self.model( 2025-08-14T21:57:27.2011455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2011531Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2011757Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2011833Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2012071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2012160Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2012397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.2012473Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.2012476Z 2025-08-14T21:57:27.2012571Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2012766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2012829Z return mod(**inputs) 2025-08-14T21:57:27.2013058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2013129Z outputs = self.model( 2025-08-14T21:57:27.2013359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2013432Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2013635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2013706Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2013939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.2014049Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.2014054Z 2025-08-14T21:57:27.2014157Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2014341Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2014404Z return mod(**inputs) 2025-08-14T21:57:27.2014640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2014703Z outputs = self.model( 2025-08-14T21:57:27.2014932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2015006Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2015208Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2015286Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2015514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.2015638Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.2015847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.2015928Z return self.act(input) 2025-08-14T21:57:27.2015931Z 2025-08-14T21:57:27.2016035Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2016219Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2016280Z return mod(**inputs) 2025-08-14T21:57:27.2016516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2016579Z outputs = self.model( 2025-08-14T21:57:27.2016820Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2016899Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2017110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2017216Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2017446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.2017522Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.2017526Z 2025-08-14T21:57:27.2017627Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2017811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2017873Z return mod(**inputs) 2025-08-14T21:57:27.2018110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2018174Z outputs = self.model( 2025-08-14T21:57:27.2018412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2018483Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2018688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2018768Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2018999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-14T21:57:27.2019080Z hidden_states = residual + hidden_states 2025-08-14T21:57:27.2019083Z 2025-08-14T21:57:27.2019175Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2019354Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2019421Z return mod(**inputs) 2025-08-14T21:57:27.2019643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2019705Z outputs = self.model( 2025-08-14T21:57:27.2019935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2020002Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2020209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2020278Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2020501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2020596Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2020815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.2020915Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.2020925Z 2025-08-14T21:57:27.2021042Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2021223Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2021292Z return mod(**inputs) 2025-08-14T21:57:27.2021535Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2021596Z outputs = self.model( 2025-08-14T21:57:27.2021828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2021893Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2022101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2022172Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2022410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2022510Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2022748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.2022824Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.2022834Z 2025-08-14T21:57:27.2022926Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2023106Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2023174Z return mod(**inputs) 2025-08-14T21:57:27.2023394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2023456Z outputs = self.model( 2025-08-14T21:57:27.2023685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2023749Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2023948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2024026Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2024248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2024343Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2024563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.2024661Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.2024664Z 2025-08-14T21:57:27.2024765Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2024950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2025016Z return mod(**inputs) 2025-08-14T21:57:27.2025246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2025310Z outputs = self.model( 2025-08-14T21:57:27.2025542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2025610Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2025814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2025893Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2026118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2026214Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2026477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.2026616Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.2026619Z 2025-08-14T21:57:27.2026720Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2026915Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2026982Z return mod(**inputs) 2025-08-14T21:57:27.2027203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2027265Z outputs = self.model( 2025-08-14T21:57:27.2027491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2027556Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2027769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2027851Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2028072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2028182Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2028410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.2028488Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.2028492Z 2025-08-14T21:57:27.2028593Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2028771Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2028838Z return mod(**inputs) 2025-08-14T21:57:27.2029064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2029127Z outputs = self.model( 2025-08-14T21:57:27.2029359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2029426Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2029627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2029708Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2029933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2030029Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2030253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.2030340Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.2030344Z 2025-08-14T21:57:27.2030445Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2030625Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2030696Z return mod(**inputs) 2025-08-14T21:57:27.2030923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2030987Z outputs = self.model( 2025-08-14T21:57:27.2031222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2031289Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2031488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2031568Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2031793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2031891Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2032138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.2032252Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.2032271Z 2025-08-14T21:57:27.2032375Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2032554Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2032614Z return mod(**inputs) 2025-08-14T21:57:27.2032846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2032908Z outputs = self.model( 2025-08-14T21:57:27.2033142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2033208Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2033423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2033506Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2033743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2033842Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2034065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.2034139Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.2034142Z 2025-08-14T21:57:27.2034238Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2034416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2034475Z return mod(**inputs) 2025-08-14T21:57:27.2034703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2034766Z outputs = self.model( 2025-08-14T21:57:27.2034993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2035058Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2035257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2035335Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2035557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.2035663Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.2035673Z 2025-08-14T21:57:27.2035763Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2035944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2036013Z return mod(**inputs) 2025-08-14T21:57:27.2036235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2036298Z outputs = self.model( 2025-08-14T21:57:27.2036528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2036593Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2036796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2036868Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2037089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.2037199Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.2037390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.2037945Z return self.act(input) 2025-08-14T21:57:27.2037958Z 2025-08-14T21:57:27.2038057Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2038278Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2038344Z return mod(**inputs) 2025-08-14T21:57:27.2038566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2038628Z outputs = self.model( 2025-08-14T21:57:27.2038857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2038923Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2039127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2039219Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2039447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.2039562Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.2039568Z 2025-08-14T21:57:27.2039660Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2039840Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2039908Z return mod(**inputs) 2025-08-14T21:57:27.2040130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2040198Z outputs = self.model( 2025-08-14T21:57:27.2040420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2040487Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2040693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2040763Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2040986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2041080Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2041303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.2041407Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.2041411Z 2025-08-14T21:57:27.2041503Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2041680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2041748Z return mod(**inputs) 2025-08-14T21:57:27.2041970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2042040Z outputs = self.model( 2025-08-14T21:57:27.2042260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2042325Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2042529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2042600Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2042818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2042912Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2043131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.2043213Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.2043238Z 2025-08-14T21:57:27.2043331Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2043515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2043598Z return mod(**inputs) 2025-08-14T21:57:27.2043823Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2043891Z outputs = self.model( 2025-08-14T21:57:27.2044113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2044179Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2044390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2044461Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2044703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2044814Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2045054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.2045164Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.2045168Z 2025-08-14T21:57:27.2045259Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2045484Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2045560Z return mod(**inputs) 2025-08-14T21:57:27.2045784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2045846Z outputs = self.model( 2025-08-14T21:57:27.2046083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2046152Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2046365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2046436Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2046664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2046761Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2046989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.2047118Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.2047122Z 2025-08-14T21:57:27.2047216Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2047400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2047470Z return mod(**inputs) 2025-08-14T21:57:27.2047700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2047767Z outputs = self.model( 2025-08-14T21:57:27.2048004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2048072Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2048285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2048357Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2048586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2048683Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2048915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.2049021Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.2049025Z 2025-08-14T21:57:27.2049120Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2049323Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2049391Z return mod(**inputs) 2025-08-14T21:57:27.2049618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2049681Z outputs = self.model( 2025-08-14T21:57:27.2049916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2049982Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2050207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2050281Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2050508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2050622Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2050854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.2050942Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.2050953Z 2025-08-14T21:57:27.2051047Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2051231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2051299Z return mod(**inputs) 2025-08-14T21:57:27.2051529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2051593Z outputs = self.model( 2025-08-14T21:57:27.2051830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2051897Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2052112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2052185Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2052413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2052506Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2052735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.2052850Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.2052861Z 2025-08-14T21:57:27.2052957Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2053141Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2053208Z return mod(**inputs) 2025-08-14T21:57:27.2053437Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2053501Z outputs = self.model( 2025-08-14T21:57:27.2053737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2053805Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2054020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2054091Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2054320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2054416Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2054662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.2054742Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.2054760Z 2025-08-14T21:57:27.2054863Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2055048Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2055115Z return mod(**inputs) 2025-08-14T21:57:27.2055344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2055406Z outputs = self.model( 2025-08-14T21:57:27.2055643Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2055726Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2055932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2056014Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2056256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.2056373Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.2056376Z 2025-08-14T21:57:27.2056471Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2056654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2056722Z return mod(**inputs) 2025-08-14T21:57:27.2056950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2057019Z outputs = self.model( 2025-08-14T21:57:27.2057247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2057315Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2057528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2057602Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2057827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.2057942Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.2058136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.2058208Z return self.act(input) 2025-08-14T21:57:27.2058211Z 2025-08-14T21:57:27.2058306Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2058492Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2058564Z return mod(**inputs) 2025-08-14T21:57:27.2058792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2058864Z outputs = self.model( 2025-08-14T21:57:27.2059093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2059161Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2059372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2059445Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2059669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.2059751Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.2059755Z 2025-08-14T21:57:27.2059848Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2060056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2060116Z return mod(**inputs) 2025-08-14T21:57:27.2060344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2060438Z outputs = self.model( 2025-08-14T21:57:27.2060661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2060729Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2060935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2061005Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2061257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 366, in forward 2025-08-14T21:57:27.2061331Z hidden_states = residual + hidden_states 2025-08-14T21:57:27.2061336Z 2025-08-14T21:57:27.2061428Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2061630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2061692Z return mod(**inputs) 2025-08-14T21:57:27.2061925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2061987Z outputs = self.model( 2025-08-14T21:57:27.2062210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2062283Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2062482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2062553Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2062785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2062874Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2063105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.2063207Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.2063211Z 2025-08-14T21:57:27.2063303Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2063490Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2063550Z return mod(**inputs) 2025-08-14T21:57:27.2063771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2063840Z outputs = self.model( 2025-08-14T21:57:27.2064064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2064139Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2064338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2064410Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2064639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2064724Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2064955Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 175, in forward 2025-08-14T21:57:27.2065027Z key_states = self.k_proj(current_states) 2025-08-14T21:57:27.2065031Z 2025-08-14T21:57:27.2065124Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2065310Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2065388Z return mod(**inputs) 2025-08-14T21:57:27.2065613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2065683Z outputs = self.model( 2025-08-14T21:57:27.2065922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2065994Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2066194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2066263Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2066491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2066578Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2066817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 156, in forward 2025-08-14T21:57:27.2066919Z query_states = self.q_proj(hidden_states) * self.scaling 2025-08-14T21:57:27.2066922Z 2025-08-14T21:57:27.2067028Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2067218Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2067278Z return mod(**inputs) 2025-08-14T21:57:27.2067502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2067572Z outputs = self.model( 2025-08-14T21:57:27.2067796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2067866Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2068067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2068139Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2068372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2068460Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2068685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 197, in forward 2025-08-14T21:57:27.2068813Z attn_weights = torch.bmm(query_states, key_states.transpose(1, 2)) 2025-08-14T21:57:27.2068816Z 2025-08-14T21:57:27.2068908Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2069093Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2069153Z return mod(**inputs) 2025-08-14T21:57:27.2069379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2069449Z outputs = self.model( 2025-08-14T21:57:27.2069676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2069750Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2069950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2070021Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2070250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2070338Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2070562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 176, in forward 2025-08-14T21:57:27.2070648Z value_states = self.v_proj(current_states) 2025-08-14T21:57:27.2070652Z 2025-08-14T21:57:27.2070745Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2070950Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2071011Z return mod(**inputs) 2025-08-14T21:57:27.2071235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2071321Z outputs = self.model( 2025-08-14T21:57:27.2071544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2071617Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2071818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2071887Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2072133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2072222Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2072443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 243, in forward 2025-08-14T21:57:27.2072551Z attn_output = torch.bmm(attn_probs, value_states) 2025-08-14T21:57:27.2072556Z 2025-08-14T21:57:27.2072649Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2072836Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2072895Z return mod(**inputs) 2025-08-14T21:57:27.2073115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2073185Z outputs = self.model( 2025-08-14T21:57:27.2073406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2073473Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2073682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2073751Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2073981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2074069Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2074287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 256, in forward 2025-08-14T21:57:27.2074406Z attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim) 2025-08-14T21:57:27.2074409Z 2025-08-14T21:57:27.2074500Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2074685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2074745Z return mod(**inputs) 2025-08-14T21:57:27.2074964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2075035Z outputs = self.model( 2025-08-14T21:57:27.2075256Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2075323Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2075528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2075599Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2075827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 330, in forward 2025-08-14T21:57:27.2075913Z hidden_states, self_attn_weights = self.self_attn( 2025-08-14T21:57:27.2076138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 258, in forward 2025-08-14T21:57:27.2076218Z attn_output = self.out_proj(attn_output) 2025-08-14T21:57:27.2076236Z 2025-08-14T21:57:27.2076330Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2076521Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2076597Z return mod(**inputs) 2025-08-14T21:57:27.2076821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2076892Z outputs = self.model( 2025-08-14T21:57:27.2077115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2077181Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2077390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2077477Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2077709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.2077817Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.2077835Z 2025-08-14T21:57:27.2077930Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2078118Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2078176Z return mod(**inputs) 2025-08-14T21:57:27.2078406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2078467Z outputs = self.model( 2025-08-14T21:57:27.2078690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2078761Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2078960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2079031Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2079262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 362, in forward 2025-08-14T21:57:27.2079369Z hidden_states = self.activation_fn(self.fc1(hidden_states)) 2025-08-14T21:57:27.2079565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:57:27.2079629Z return self.act(input) 2025-08-14T21:57:27.2079633Z 2025-08-14T21:57:27.2079725Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2079910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2079970Z return mod(**inputs) 2025-08-14T21:57:27.2080194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 664, in forward 2025-08-14T21:57:27.2080262Z outputs = self.model( 2025-08-14T21:57:27.2080486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 552, in forward 2025-08-14T21:57:27.2080559Z layer_outputs = decoder_layer( 2025-08-14T21:57:27.2080762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:57:27.2080832Z return super().__call__(*args, **kwargs) 2025-08-14T21:57:27.2081063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 364, in forward 2025-08-14T21:57:27.2081137Z hidden_states = self.fc2(hidden_states) 2025-08-14T21:57:27.2081140Z 2025-08-14T21:57:27.2081239Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2081417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2081480Z return mod(**inputs) 2025-08-14T21:57:27.2081707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 681, in forward 2025-08-14T21:57:27.2081800Z logits = self.lm_head(outputs[0]) 2025-08-14T21:57:27.2081804Z 2025-08-14T21:57:27.2081897Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:57:27.2082111Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:57:27.2082171Z return mod(**inputs) 2025-08-14T21:57:27.2082399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xglm/modeling_xglm.py", line 685, in forward 2025-08-14T21:57:27.2082466Z loss = self.loss_function( 2025-08-14T21:57:27.2082691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 67, in ForCausalLMLoss 2025-08-14T21:57:27.2082856Z loss = fixed_cross_entropy(logits, shift_labels, num_items_in_batch, ignore_index, **kwargs) 2025-08-14T21:57:27.2083099Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/loss/loss_utils.py", line 36, in fixed_cross_entropy 2025-08-14T21:57:27.2083303Z loss = nn.functional.cross_entropy(source, target, ignore_index=ignore_index, reduction=reduction) 2025-08-14T21:57:27.2083308Z 2025-08-14T21:57:38.3950184Z Compilation time (from dynamo_timed): 23.615562425 2025-08-14T21:57:38.4039458Z pass 2025-08-14T21:57:38.4040147Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:57:38.4041153Z TIMING: _recursive_pre_grad_passes:0.01267 _recursive_joint_graph_passes:0.7465 _recursive_post_grad_passes:0.27041 async_compile.wait:0.77323 code_gen:10.50951 inductor_compile:13.47805 backend_compile:19.16784 gc:0.00059 entire_frame_compile:23.61556 total_wall_time:23.61556 2025-08-14T21:57:38.4042245Z STATS: call_* op count: 921 | FakeTensorMode.__torch_dispatch__:29112 | FakeTensor.__torch_dispatch__:10687 | ProxyTorchDispatchMode.__torch_dispatch__:10816 2025-08-14T21:57:38.4042845Z Dynamo produced 1 graphs covering 921 ops with 0 graph breaks (0 unique) 2025-08-14T21:57:43.7545157Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:57:43.7546185Z from pkg_resources import resource_filename 2025-08-14T21:57:44.3123382Z 2025-08-14T21:57:47.4793332Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:57:47.4794032Z loading model: 0it [00:03, ?it/s] 2025-08-14T21:57:47.4826711Z cpu eval XLNetLMHeadModel 2025-08-14T21:57:50.0287839Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:57:50.9268547Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:57:51.8243529Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:58:11.5154130Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5158406Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5160438Z return mod(**inputs) 2025-08-14T21:58:11.5161085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5166641Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5167299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1307, in forward 2025-08-14T21:58:11.5167894Z word_emb_k = self.word_embedding(input_ids) 2025-08-14T21:58:11.5168063Z 2025-08-14T21:58:11.5168181Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5168703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5169511Z return mod(**inputs) 2025-08-14T21:58:11.5175333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5176033Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5176447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-08-14T21:58:11.5176895Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-08-14T21:58:11.5177379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-08-14T21:58:11.5177867Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-08-14T21:58:11.5178407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-08-14T21:58:11.5178919Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-08-14T21:58:11.5179125Z 2025-08-14T21:58:11.5179237Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5179652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5179988Z return mod(**inputs) 2025-08-14T21:58:11.5180362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5180744Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5181122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-08-14T21:58:11.5181554Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-08-14T21:58:11.5181999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-08-14T21:58:11.5182455Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-08-14T21:58:11.5182902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-08-14T21:58:11.5183377Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-08-14T21:58:11.5183574Z 2025-08-14T21:58:11.5183678Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5184034Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5184348Z return mod(**inputs) 2025-08-14T21:58:11.5184699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5185071Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5185453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5185822Z outputs = layer_module( 2025-08-14T21:58:11.5186166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5186534Z outputs = self.rel_attn( 2025-08-14T21:58:11.5186886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5187270Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5187666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5188093Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5188261Z 2025-08-14T21:58:11.5188362Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5188703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5189033Z return mod(**inputs) 2025-08-14T21:58:11.5189378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5189782Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5190165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5190545Z outputs = layer_module( 2025-08-14T21:58:11.5190905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5191286Z outputs = self.rel_attn( 2025-08-14T21:58:11.5191660Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5192113Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5192518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5192946Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5193128Z 2025-08-14T21:58:11.5193231Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5193587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5193892Z return mod(**inputs) 2025-08-14T21:58:11.5194230Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5194608Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5194987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5195360Z outputs = layer_module( 2025-08-14T21:58:11.5195708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5196078Z outputs = self.rel_attn( 2025-08-14T21:58:11.5196435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5196819Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5197223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5197650Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5197807Z 2025-08-14T21:58:11.5197916Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5198253Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5198567Z return mod(**inputs) 2025-08-14T21:58:11.5198917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5199307Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5199684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5200059Z outputs = layer_module( 2025-08-14T21:58:11.5200419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5200777Z outputs = self.rel_attn( 2025-08-14T21:58:11.5201132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5201520Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5201930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5202361Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5202553Z 2025-08-14T21:58:11.5202661Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5203023Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5203384Z return mod(**inputs) 2025-08-14T21:58:11.5203738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5204133Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5204527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5204902Z outputs = layer_module( 2025-08-14T21:58:11.5205278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5205777Z outputs = self.rel_attn( 2025-08-14T21:58:11.5206169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5206563Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5207007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5207438Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5207600Z 2025-08-14T21:58:11.5207710Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5208056Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5208375Z return mod(**inputs) 2025-08-14T21:58:11.5208730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5209111Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5209557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5209934Z outputs = layer_module( 2025-08-14T21:58:11.5210287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5210651Z outputs = self.rel_attn( 2025-08-14T21:58:11.5211005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5211388Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5211783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5212210Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5212376Z 2025-08-14T21:58:11.5212477Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5212829Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5213139Z return mod(**inputs) 2025-08-14T21:58:11.5213496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5213889Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5214270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5214635Z outputs = layer_module( 2025-08-14T21:58:11.5214996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5215358Z outputs = self.rel_attn( 2025-08-14T21:58:11.5215697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5216075Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5216469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5216903Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5217058Z 2025-08-14T21:58:11.5217190Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5217530Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5217835Z return mod(**inputs) 2025-08-14T21:58:11.5218173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5218568Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5218938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5219299Z outputs = layer_module( 2025-08-14T21:58:11.5219651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5220022Z outputs = self.rel_attn( 2025-08-14T21:58:11.5220394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5220797Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5221206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5221632Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5221793Z 2025-08-14T21:58:11.5221900Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5222251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5222567Z return mod(**inputs) 2025-08-14T21:58:11.5222933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5223305Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5223665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5224027Z outputs = layer_module( 2025-08-14T21:58:11.5224366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5224724Z outputs = self.rel_attn( 2025-08-14T21:58:11.5225071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5225681Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5226080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5226498Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5226668Z 2025-08-14T21:58:11.5226770Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5227129Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5227456Z return mod(**inputs) 2025-08-14T21:58:11.5227828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5228215Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5228587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5228965Z outputs = layer_module( 2025-08-14T21:58:11.5229305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5229659Z outputs = self.rel_attn( 2025-08-14T21:58:11.5230013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5230422Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5230831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5231266Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5231441Z 2025-08-14T21:58:11.5231538Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5231878Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5232176Z return mod(**inputs) 2025-08-14T21:58:11.5232517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5232914Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5233340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5233728Z outputs = layer_module( 2025-08-14T21:58:11.5234111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5234508Z outputs = self.rel_attn( 2025-08-14T21:58:11.5234860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5235243Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5235656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5236092Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5236253Z 2025-08-14T21:58:11.5236356Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5236725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5237057Z return mod(**inputs) 2025-08-14T21:58:11.5237425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5238009Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5238411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5238805Z outputs = layer_module( 2025-08-14T21:58:11.5239175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5239563Z outputs = self.rel_attn( 2025-08-14T21:58:11.5239938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5240346Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5240763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5241211Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5241389Z 2025-08-14T21:58:11.5241498Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5241858Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5242179Z return mod(**inputs) 2025-08-14T21:58:11.5242546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5242945Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5243336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5243729Z outputs = layer_module( 2025-08-14T21:58:11.5244114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5244568Z outputs = self.rel_attn( 2025-08-14T21:58:11.5244952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5245418Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5245892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5246346Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5246510Z 2025-08-14T21:58:11.5246615Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5246979Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5247304Z return mod(**inputs) 2025-08-14T21:58:11.5247685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5248087Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5248507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5248896Z outputs = layer_module( 2025-08-14T21:58:11.5249254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5249640Z outputs = self.rel_attn( 2025-08-14T21:58:11.5250008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5250403Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5250801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5251228Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5251387Z 2025-08-14T21:58:11.5251505Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5251873Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5252214Z return mod(**inputs) 2025-08-14T21:58:11.5252564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5252951Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5253326Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5253702Z outputs = layer_module( 2025-08-14T21:58:11.5254055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5254429Z outputs = self.rel_attn( 2025-08-14T21:58:11.5254795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5255196Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5255615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5256067Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5256234Z 2025-08-14T21:58:11.5256334Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5256682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5257001Z return mod(**inputs) 2025-08-14T21:58:11.5257348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5257748Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5258134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5258534Z outputs = layer_module( 2025-08-14T21:58:11.5258880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5259257Z outputs = self.rel_attn( 2025-08-14T21:58:11.5259601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5259972Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5260367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5260785Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5260940Z 2025-08-14T21:58:11.5261044Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5261395Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5261719Z return mod(**inputs) 2025-08-14T21:58:11.5262105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5262483Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5262865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5263234Z outputs = layer_module( 2025-08-14T21:58:11.5263591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5263939Z outputs = self.rel_attn( 2025-08-14T21:58:11.5264282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5264656Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5265040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5265454Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5265618Z 2025-08-14T21:58:11.5265718Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5266058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5266354Z return mod(**inputs) 2025-08-14T21:58:11.5266696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5267072Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5267445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5267813Z outputs = layer_module( 2025-08-14T21:58:11.5268176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5268545Z outputs = self.rel_attn( 2025-08-14T21:58:11.5268895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5269286Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5269691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5270102Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5270254Z 2025-08-14T21:58:11.5270351Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5270699Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5271014Z return mod(**inputs) 2025-08-14T21:58:11.5271367Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5271778Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5272160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5272553Z outputs = layer_module( 2025-08-14T21:58:11.5272909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5273384Z outputs = self.rel_attn( 2025-08-14T21:58:11.5273749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5274144Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5274546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5275008Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5275173Z 2025-08-14T21:58:11.5275289Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5275676Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5275989Z return mod(**inputs) 2025-08-14T21:58:11.5276348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5276741Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5277171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5277549Z outputs = layer_module( 2025-08-14T21:58:11.5277909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5278282Z outputs = self.rel_attn( 2025-08-14T21:58:11.5278637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5279030Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5279435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5279859Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5280025Z 2025-08-14T21:58:11.5280132Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5280491Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5280812Z return mod(**inputs) 2025-08-14T21:58:11.5281153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5281550Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5281945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5282337Z outputs = layer_module( 2025-08-14T21:58:11.5282697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5283078Z outputs = self.rel_attn( 2025-08-14T21:58:11.5283445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5283842Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5284260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5284699Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5284862Z 2025-08-14T21:58:11.5284977Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5285334Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5285731Z return mod(**inputs) 2025-08-14T21:58:11.5286107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5286527Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5286921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5287309Z outputs = layer_module( 2025-08-14T21:58:11.5287669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5288055Z outputs = self.rel_attn( 2025-08-14T21:58:11.5288424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5288838Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5289261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5289692Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5289883Z 2025-08-14T21:58:11.5289993Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5290351Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5290667Z return mod(**inputs) 2025-08-14T21:58:11.5291032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5291428Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5291817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5292189Z outputs = layer_module( 2025-08-14T21:58:11.5292555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5292939Z outputs = self.rel_attn( 2025-08-14T21:58:11.5293306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5293704Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5294113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5294550Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5294714Z 2025-08-14T21:58:11.5294816Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5295175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5295498Z return mod(**inputs) 2025-08-14T21:58:11.5295856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5296246Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5296637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5297015Z outputs = layer_module( 2025-08-14T21:58:11.5297369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5297756Z outputs = self.rel_attn( 2025-08-14T21:58:11.5298126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5298517Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5298920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5299359Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5299552Z 2025-08-14T21:58:11.5299658Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5300025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5300358Z return mod(**inputs) 2025-08-14T21:58:11.5300725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5301111Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5301488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5301860Z outputs = layer_module( 2025-08-14T21:58:11.5302212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5302583Z outputs = self.rel_attn( 2025-08-14T21:58:11.5302952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.5303355Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.5303502Z 2025-08-14T21:58:11.5303624Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5303978Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5304282Z return mod(**inputs) 2025-08-14T21:58:11.5304637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5305021Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5305393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5305764Z outputs = layer_module( 2025-08-14T21:58:11.5306116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5306490Z outputs = self.rel_attn( 2025-08-14T21:58:11.5306839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.5307242Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.5307388Z 2025-08-14T21:58:11.5307496Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5307835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5308151Z return mod(**inputs) 2025-08-14T21:58:11.5308549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5308934Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5309310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5309751Z outputs = layer_module( 2025-08-14T21:58:11.5310105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5310478Z outputs = self.rel_attn( 2025-08-14T21:58:11.5310831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5311207Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5311591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.5312034Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.5312224Z 2025-08-14T21:58:11.5312325Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5312681Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5312986Z return mod(**inputs) 2025-08-14T21:58:11.5313345Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5313710Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5314083Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1334, in forward 2025-08-14T21:58:11.5314479Z pos_emb = self.relative_positional_encoding(qlen, klen, bsz=bsz) 2025-08-14T21:58:11.5314928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1157, in relative_positional_encoding 2025-08-14T21:58:11.5315383Z pos_emb = self.positional_embedding(fwd_pos_seq, inv_freq, bsz) 2025-08-14T21:58:11.5315816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1115, in positional_embedding 2025-08-14T21:58:11.5316285Z pos_emb = torch.cat([torch.sin(sinusoid_inp), torch.cos(sinusoid_inp)], dim=-1) 2025-08-14T21:58:11.5316482Z 2025-08-14T21:58:11.5316578Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5316934Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5317243Z return mod(**inputs) 2025-08-14T21:58:11.5317581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5317958Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5318327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5318690Z outputs = layer_module( 2025-08-14T21:58:11.5319027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5319390Z outputs = self.rel_attn( 2025-08-14T21:58:11.5319737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.5320153Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.5320339Z 2025-08-14T21:58:11.5320441Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5320778Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5321090Z return mod(**inputs) 2025-08-14T21:58:11.5321432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5321815Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5322194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5322556Z outputs = layer_module( 2025-08-14T21:58:11.5322910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5323280Z outputs = self.rel_attn( 2025-08-14T21:58:11.5323635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5324003Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5324389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.5324831Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.5325010Z 2025-08-14T21:58:11.5325120Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5325464Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5325888Z return mod(**inputs) 2025-08-14T21:58:11.5326278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5326714Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5327131Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5327524Z outputs = layer_module( 2025-08-14T21:58:11.5327908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5328310Z outputs = self.rel_attn( 2025-08-14T21:58:11.5328700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.5329153Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.5329315Z 2025-08-14T21:58:11.5329429Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5329813Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5330158Z return mod(**inputs) 2025-08-14T21:58:11.5330540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5330975Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5331389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5331797Z outputs = layer_module( 2025-08-14T21:58:11.5332177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5332571Z outputs = self.rel_attn( 2025-08-14T21:58:11.5332954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5333372Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5333780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.5334247Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.5334419Z 2025-08-14T21:58:11.5334518Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5334861Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5335164Z return mod(**inputs) 2025-08-14T21:58:11.5335507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5335876Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5336247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5336603Z outputs = layer_module( 2025-08-14T21:58:11.5336945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5337306Z outputs = self.rel_attn( 2025-08-14T21:58:11.5337814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5338216Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5338616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5339036Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5339194Z 2025-08-14T21:58:11.5339295Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5339637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5339949Z return mod(**inputs) 2025-08-14T21:58:11.5340289Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5340726Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5341102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5341499Z outputs = layer_module( 2025-08-14T21:58:11.5341838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5342197Z outputs = self.rel_attn( 2025-08-14T21:58:11.5342547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5342926Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5343313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5343756Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5343914Z 2025-08-14T21:58:11.5344021Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5344348Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5344684Z return mod(**inputs) 2025-08-14T21:58:11.5345029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5345407Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5345771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5346131Z outputs = layer_module( 2025-08-14T21:58:11.5346471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5346965Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5347454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5347831Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5348201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5348563Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5348918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.5349280Z output = self.layer_1(output) 2025-08-14T21:58:11.5349398Z 2025-08-14T21:58:11.5349513Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5349835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5350134Z return mod(**inputs) 2025-08-14T21:58:11.5350476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5350841Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5351213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5351576Z outputs = layer_module( 2025-08-14T21:58:11.5351920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5352399Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5352892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5353271Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5353630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5354007Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5354358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.5354730Z output = self.activation_function(output) 2025-08-14T21:58:11.5355080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.5355412Z return self.act(input) 2025-08-14T21:58:11.5355523Z 2025-08-14T21:58:11.5355622Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5355971Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5356263Z return mod(**inputs) 2025-08-14T21:58:11.5356597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5356977Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5357341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5357684Z outputs = layer_module( 2025-08-14T21:58:11.5358030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5358508Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5358986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5359351Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5359709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5360065Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5360410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.5360770Z output = self.layer_2(output) 2025-08-14T21:58:11.5360889Z 2025-08-14T21:58:11.5360997Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5361338Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5361640Z return mod(**inputs) 2025-08-14T21:58:11.5361986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5362364Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5362740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5363117Z outputs = layer_module( 2025-08-14T21:58:11.5363460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5363823Z outputs = self.rel_attn( 2025-08-14T21:58:11.5364166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.5364566Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.5364714Z 2025-08-14T21:58:11.5364823Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5365164Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5365482Z return mod(**inputs) 2025-08-14T21:58:11.5365911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5366312Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5366697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5367077Z outputs = layer_module( 2025-08-14T21:58:11.5367475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5367860Z outputs = self.rel_attn( 2025-08-14T21:58:11.5368227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.5368662Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.5368819Z 2025-08-14T21:58:11.5368934Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5369291Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5369623Z return mod(**inputs) 2025-08-14T21:58:11.5369993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5370421Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5370806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5371183Z outputs = layer_module( 2025-08-14T21:58:11.5371560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5371938Z outputs = self.rel_attn( 2025-08-14T21:58:11.5372309Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5372695Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5373094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.5373548Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.5373741Z 2025-08-14T21:58:11.5373845Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5374203Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5374526Z return mod(**inputs) 2025-08-14T21:58:11.5374881Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5375281Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5375673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5376049Z outputs = layer_module( 2025-08-14T21:58:11.5376418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5376803Z outputs = self.rel_attn( 2025-08-14T21:58:11.5377170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.5377609Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.5377806Z 2025-08-14T21:58:11.5377900Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5378229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5378517Z return mod(**inputs) 2025-08-14T21:58:11.5378852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5379226Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5379597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5379951Z outputs = layer_module( 2025-08-14T21:58:11.5380296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5380661Z outputs = self.rel_attn( 2025-08-14T21:58:11.5381013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5381393Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5381773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.5382229Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.5382400Z 2025-08-14T21:58:11.5382500Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5382842Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5383153Z return mod(**inputs) 2025-08-14T21:58:11.5383497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5383872Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5384262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5384624Z outputs = layer_module( 2025-08-14T21:58:11.5384974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5385337Z outputs = self.rel_attn( 2025-08-14T21:58:11.5385683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.5386073Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.5386218Z 2025-08-14T21:58:11.5386315Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5386654Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5386962Z return mod(**inputs) 2025-08-14T21:58:11.5387304Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5387674Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5388046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5388407Z outputs = layer_module( 2025-08-14T21:58:11.5388745Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5389105Z outputs = self.rel_attn( 2025-08-14T21:58:11.5389452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5389818Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5390186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.5390614Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.5390774Z 2025-08-14T21:58:11.5390880Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5391209Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5391505Z return mod(**inputs) 2025-08-14T21:58:11.5391858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5392244Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5392619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5392987Z outputs = layer_module( 2025-08-14T21:58:11.5393351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5393711Z outputs = self.rel_attn( 2025-08-14T21:58:11.5394054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5394459Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5394855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5395289Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5395454Z 2025-08-14T21:58:11.5395552Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5395894Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5396201Z return mod(**inputs) 2025-08-14T21:58:11.5396537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5396917Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5397310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5397671Z outputs = layer_module( 2025-08-14T21:58:11.5398024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5398384Z outputs = self.rel_attn( 2025-08-14T21:58:11.5398725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5399100Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5399494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5399906Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5400060Z 2025-08-14T21:58:11.5400163Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5400493Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5400794Z return mod(**inputs) 2025-08-14T21:58:11.5401136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5401507Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5401867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5402228Z outputs = layer_module( 2025-08-14T21:58:11.5402567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5403054Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5403550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5403925Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5404296Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5404656Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5405017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.5405382Z output = self.layer_1(output) 2025-08-14T21:58:11.5405502Z 2025-08-14T21:58:11.5405669Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5406047Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5406378Z return mod(**inputs) 2025-08-14T21:58:11.5406746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5407147Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5407526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5407920Z outputs = layer_module( 2025-08-14T21:58:11.5408277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5408823Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5409325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5409703Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5410066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5410434Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5410813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.5411258Z output = self.activation_function(output) 2025-08-14T21:58:11.5411630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.5411978Z return self.act(input) 2025-08-14T21:58:11.5412088Z 2025-08-14T21:58:11.5412198Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5412597Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5412906Z return mod(**inputs) 2025-08-14T21:58:11.5413267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5413645Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5414016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5414385Z outputs = layer_module( 2025-08-14T21:58:11.5414738Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5415229Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5415724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5416087Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5416445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5416803Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5417143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.5417493Z output = self.layer_2(output) 2025-08-14T21:58:11.5417607Z 2025-08-14T21:58:11.5417710Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5418035Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5418341Z return mod(**inputs) 2025-08-14T21:58:11.5418673Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5419037Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5419390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5419740Z outputs = layer_module( 2025-08-14T21:58:11.5420077Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5420424Z outputs = self.rel_attn( 2025-08-14T21:58:11.5420766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.5421168Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.5421315Z 2025-08-14T21:58:11.5421426Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5421794Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5422109Z return mod(**inputs) 2025-08-14T21:58:11.5422460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5422854Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5423227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5423579Z outputs = layer_module( 2025-08-14T21:58:11.5423938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5424291Z outputs = self.rel_attn( 2025-08-14T21:58:11.5424638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.5425046Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.5425192Z 2025-08-14T21:58:11.5425294Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5425617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5425917Z return mod(**inputs) 2025-08-14T21:58:11.5426250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5426616Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5426987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5427353Z outputs = layer_module( 2025-08-14T21:58:11.5427702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5428057Z outputs = self.rel_attn( 2025-08-14T21:58:11.5428409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5428771Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5429150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.5429581Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.5429763Z 2025-08-14T21:58:11.5429860Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5430199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5430499Z return mod(**inputs) 2025-08-14T21:58:11.5430844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5431220Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5431593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5431949Z outputs = layer_module( 2025-08-14T21:58:11.5432290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5432650Z outputs = self.rel_attn( 2025-08-14T21:58:11.5432989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.5433408Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.5433588Z 2025-08-14T21:58:11.5433687Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5434028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5434342Z return mod(**inputs) 2025-08-14T21:58:11.5434698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5435099Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5435482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5435855Z outputs = layer_module( 2025-08-14T21:58:11.5436217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5436601Z outputs = self.rel_attn( 2025-08-14T21:58:11.5436949Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5437338Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5437832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.5438325Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.5438508Z 2025-08-14T21:58:11.5438608Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5438961Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5439281Z return mod(**inputs) 2025-08-14T21:58:11.5439632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5440024Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5440396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5440758Z outputs = layer_module( 2025-08-14T21:58:11.5441097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5441470Z outputs = self.rel_attn( 2025-08-14T21:58:11.5441824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.5442224Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.5442371Z 2025-08-14T21:58:11.5442470Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5442814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5443127Z return mod(**inputs) 2025-08-14T21:58:11.5443462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5443851Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5444216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5444577Z outputs = layer_module( 2025-08-14T21:58:11.5444911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5445275Z outputs = self.rel_attn( 2025-08-14T21:58:11.5445670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5446064Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5446455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.5446907Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.5447083Z 2025-08-14T21:58:11.5447195Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5447581Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5447924Z return mod(**inputs) 2025-08-14T21:58:11.5448279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5448688Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5449055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5449418Z outputs = layer_module( 2025-08-14T21:58:11.5449806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5450175Z outputs = self.rel_attn( 2025-08-14T21:58:11.5450541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5450963Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5451388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5451829Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5452022Z 2025-08-14T21:58:11.5452129Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5452488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5452867Z return mod(**inputs) 2025-08-14T21:58:11.5453223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5453623Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5454015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5454387Z outputs = layer_module( 2025-08-14T21:58:11.5454752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5455131Z outputs = self.rel_attn( 2025-08-14T21:58:11.5455499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5455891Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5456314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5456758Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5456922Z 2025-08-14T21:58:11.5457036Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5457384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5457706Z return mod(**inputs) 2025-08-14T21:58:11.5458066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5458454Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5458848Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5459233Z outputs = layer_module( 2025-08-14T21:58:11.5459576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5460103Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5460604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5460983Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5461355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5461735Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5462093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.5462459Z output = self.layer_1(output) 2025-08-14T21:58:11.5462597Z 2025-08-14T21:58:11.5462698Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5463049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5463366Z return mod(**inputs) 2025-08-14T21:58:11.5463715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5464158Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5464541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5464925Z outputs = layer_module( 2025-08-14T21:58:11.5465260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5465765Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5466265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5466639Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5466998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5467366Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5467722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.5468101Z output = self.activation_function(output) 2025-08-14T21:58:11.5468433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.5468760Z return self.act(input) 2025-08-14T21:58:11.5468865Z 2025-08-14T21:58:11.5468970Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5469304Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5469615Z return mod(**inputs) 2025-08-14T21:58:11.5469961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5470341Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5470711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5471075Z outputs = layer_module( 2025-08-14T21:58:11.5471423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5471922Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5472414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5472790Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5473159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5473520Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5473879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.5474249Z output = self.layer_2(output) 2025-08-14T21:58:11.5474366Z 2025-08-14T21:58:11.5474473Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5474808Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5475147Z return mod(**inputs) 2025-08-14T21:58:11.5475489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5475859Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5476254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5476610Z outputs = layer_module( 2025-08-14T21:58:11.5476950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5477299Z outputs = self.rel_attn( 2025-08-14T21:58:11.5477649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.5478040Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.5478194Z 2025-08-14T21:58:11.5478302Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5478632Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5478950Z return mod(**inputs) 2025-08-14T21:58:11.5479293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5479661Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5480035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5480392Z outputs = layer_module( 2025-08-14T21:58:11.5480737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5481089Z outputs = self.rel_attn( 2025-08-14T21:58:11.5481435Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.5481831Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.5481978Z 2025-08-14T21:58:11.5482082Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5482420Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5482732Z return mod(**inputs) 2025-08-14T21:58:11.5483076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5483450Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5483826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5484190Z outputs = layer_module( 2025-08-14T21:58:11.5484543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5484904Z outputs = self.rel_attn( 2025-08-14T21:58:11.5485257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5485686Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5486082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.5486548Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.5486756Z 2025-08-14T21:58:11.5486869Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5487260Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5487604Z return mod(**inputs) 2025-08-14T21:58:11.5487991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5488379Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5488769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5489130Z outputs = layer_module( 2025-08-14T21:58:11.5489472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5489844Z outputs = self.rel_attn( 2025-08-14T21:58:11.5490180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.5490604Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.5490787Z 2025-08-14T21:58:11.5490884Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5491230Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5491551Z return mod(**inputs) 2025-08-14T21:58:11.5491906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5492307Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5492706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5493091Z outputs = layer_module( 2025-08-14T21:58:11.5493457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5493825Z outputs = self.rel_attn( 2025-08-14T21:58:11.5494174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5494549Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5494916Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.5495371Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.5495554Z 2025-08-14T21:58:11.5495657Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5496014Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5496338Z return mod(**inputs) 2025-08-14T21:58:11.5496691Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5497092Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5497480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5497860Z outputs = layer_module( 2025-08-14T21:58:11.5498218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5498603Z outputs = self.rel_attn( 2025-08-14T21:58:11.5498965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.5499370Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.5499530Z 2025-08-14T21:58:11.5499632Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5499987Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5500307Z return mod(**inputs) 2025-08-14T21:58:11.5500656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5501046Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5501436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5501812Z outputs = layer_module( 2025-08-14T21:58:11.5502174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5502578Z outputs = self.rel_attn( 2025-08-14T21:58:11.5502943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5503340Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5503741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.5504195Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.5504369Z 2025-08-14T21:58:11.5504478Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5504831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5505156Z return mod(**inputs) 2025-08-14T21:58:11.5505516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5505896Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5506298Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5506668Z outputs = layer_module( 2025-08-14T21:58:11.5507010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5507367Z outputs = self.rel_attn( 2025-08-14T21:58:11.5507717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5508101Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5508498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5508914Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5509082Z 2025-08-14T21:58:11.5509180Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5509520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5509823Z return mod(**inputs) 2025-08-14T21:58:11.5510168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5510549Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5510920Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5511283Z outputs = layer_module( 2025-08-14T21:58:11.5511633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5511996Z outputs = self.rel_attn( 2025-08-14T21:58:11.5512336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5512724Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5513126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5513549Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5513706Z 2025-08-14T21:58:11.5513803Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5514144Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5514452Z return mod(**inputs) 2025-08-14T21:58:11.5514797Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5515167Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5515542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5515929Z outputs = layer_module( 2025-08-14T21:58:11.5516269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5516775Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5517268Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5517642Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5518003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5518369Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5518735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.5519101Z output = self.layer_1(output) 2025-08-14T21:58:11.5519216Z 2025-08-14T21:58:11.5519311Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5519685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5519988Z return mod(**inputs) 2025-08-14T21:58:11.5520316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5520695Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5521065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5521424Z outputs = layer_module( 2025-08-14T21:58:11.5521766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5522271Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5522781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5523173Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5523542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5523920Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5524285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.5524669Z output = self.activation_function(output) 2025-08-14T21:58:11.5525023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.5525361Z return self.act(input) 2025-08-14T21:58:11.5525481Z 2025-08-14T21:58:11.5525599Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5526066Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5526424Z return mod(**inputs) 2025-08-14T21:58:11.5526808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5527225Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5527649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5528064Z outputs = layer_module( 2025-08-14T21:58:11.5528434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5528949Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5529483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5529916Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5530294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5530688Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5531061Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.5531445Z output = self.layer_2(output) 2025-08-14T21:58:11.5531569Z 2025-08-14T21:58:11.5531681Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5532036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5532359Z return mod(**inputs) 2025-08-14T21:58:11.5532741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5533138Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5533560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5533933Z outputs = layer_module( 2025-08-14T21:58:11.5534287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5534653Z outputs = self.rel_attn( 2025-08-14T21:58:11.5535011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.5535418Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.5535568Z 2025-08-14T21:58:11.5535670Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5536027Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5536352Z return mod(**inputs) 2025-08-14T21:58:11.5536716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5537116Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5537505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5538001Z outputs = layer_module( 2025-08-14T21:58:11.5538354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5538726Z outputs = self.rel_attn( 2025-08-14T21:58:11.5539082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.5539487Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.5539637Z 2025-08-14T21:58:11.5539741Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5540097Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5540416Z return mod(**inputs) 2025-08-14T21:58:11.5540773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5541155Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5541540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5541912Z outputs = layer_module( 2025-08-14T21:58:11.5542258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5542629Z outputs = self.rel_attn( 2025-08-14T21:58:11.5542989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5543417Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5543793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.5544249Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.5544459Z 2025-08-14T21:58:11.5544570Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5544929Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5545244Z return mod(**inputs) 2025-08-14T21:58:11.5545606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5545999Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5546402Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5546773Z outputs = layer_module( 2025-08-14T21:58:11.5547132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5547531Z outputs = self.rel_attn( 2025-08-14T21:58:11.5547887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.5548320Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.5548490Z 2025-08-14T21:58:11.5548594Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5548923Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5549231Z return mod(**inputs) 2025-08-14T21:58:11.5549479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5549559Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5549795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5549868Z outputs = layer_module( 2025-08-14T21:58:11.5550103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5550172Z outputs = self.rel_attn( 2025-08-14T21:58:11.5550415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5550495Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5550746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.5550863Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.5550867Z 2025-08-14T21:58:11.5550964Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5551156Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5551218Z return mod(**inputs) 2025-08-14T21:58:11.5551455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5551532Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5551773Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5551847Z outputs = layer_module( 2025-08-14T21:58:11.5552086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5552152Z outputs = self.rel_attn( 2025-08-14T21:58:11.5552398Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.5552496Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.5552518Z 2025-08-14T21:58:11.5552626Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5552818Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5552898Z return mod(**inputs) 2025-08-14T21:58:11.5553150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5553230Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5553481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5553559Z outputs = layer_module( 2025-08-14T21:58:11.5553792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5553876Z outputs = self.rel_attn( 2025-08-14T21:58:11.5554108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5554176Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5554446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.5554563Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.5554567Z 2025-08-14T21:58:11.5554668Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5554848Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5554909Z return mod(**inputs) 2025-08-14T21:58:11.5555150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5555226Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5555468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5555532Z outputs = layer_module( 2025-08-14T21:58:11.5555765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5555835Z outputs = self.rel_attn( 2025-08-14T21:58:11.5556070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5556153Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5556415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5556518Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5556522Z 2025-08-14T21:58:11.5556623Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5556807Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5556869Z return mod(**inputs) 2025-08-14T21:58:11.5557114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5557191Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5557425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5557496Z outputs = layer_module( 2025-08-14T21:58:11.5557729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5557800Z outputs = self.rel_attn( 2025-08-14T21:58:11.5558031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5558116Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5558405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5558507Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5558525Z 2025-08-14T21:58:11.5558629Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5558810Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5558871Z return mod(**inputs) 2025-08-14T21:58:11.5559109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5559186Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5559417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5559530Z outputs = layer_module( 2025-08-14T21:58:11.5559762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5559976Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5560217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5560289Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5560529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5560597Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5560836Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.5560904Z output = self.layer_1(output) 2025-08-14T21:58:11.5560908Z 2025-08-14T21:58:11.5561001Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5561195Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5561256Z return mod(**inputs) 2025-08-14T21:58:11.5561489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5561575Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5561803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5561872Z outputs = layer_module( 2025-08-14T21:58:11.5562103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5562296Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5562550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5562626Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5562872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5562945Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5563183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.5563303Z output = self.activation_function(output) 2025-08-14T21:58:11.5563507Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.5563574Z return self.act(input) 2025-08-14T21:58:11.5563585Z 2025-08-14T21:58:11.5563682Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5563875Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5563974Z return mod(**inputs) 2025-08-14T21:58:11.5564219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5564300Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5564574Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5564639Z outputs = layer_module( 2025-08-14T21:58:11.5564894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5565091Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5565344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5565442Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5565750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5565830Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5566107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.5566182Z output = self.layer_2(output) 2025-08-14T21:58:11.5566186Z 2025-08-14T21:58:11.5566296Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5566520Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5566589Z return mod(**inputs) 2025-08-14T21:58:11.5566868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5566960Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5567216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5567285Z outputs = layer_module( 2025-08-14T21:58:11.5567531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5567610Z outputs = self.rel_attn( 2025-08-14T21:58:11.5567858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.5567954Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.5567966Z 2025-08-14T21:58:11.5568065Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5568258Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5568330Z return mod(**inputs) 2025-08-14T21:58:11.5568576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5568658Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5568915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5568982Z outputs = layer_module( 2025-08-14T21:58:11.5569231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5569298Z outputs = self.rel_attn( 2025-08-14T21:58:11.5569539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.5569645Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.5569650Z 2025-08-14T21:58:11.5569748Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5569985Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5570056Z return mod(**inputs) 2025-08-14T21:58:11.5570321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5570409Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5570665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5570730Z outputs = layer_module( 2025-08-14T21:58:11.5570976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5571042Z outputs = self.rel_attn( 2025-08-14T21:58:11.5571284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5571362Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5571641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.5571780Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.5571784Z 2025-08-14T21:58:11.5571898Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5572092Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5572165Z return mod(**inputs) 2025-08-14T21:58:11.5572411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5572500Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5572749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5572813Z outputs = layer_module( 2025-08-14T21:58:11.5573064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5573132Z outputs = self.rel_attn( 2025-08-14T21:58:11.5573376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.5573517Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.5573522Z 2025-08-14T21:58:11.5573621Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5573819Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5573884Z return mod(**inputs) 2025-08-14T21:58:11.5574130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5574217Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5574464Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5574535Z outputs = layer_module( 2025-08-14T21:58:11.5574780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5574849Z outputs = self.rel_attn( 2025-08-14T21:58:11.5575098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5575169Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5575426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.5575556Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.5575559Z 2025-08-14T21:58:11.5575657Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5575856Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5575922Z return mod(**inputs) 2025-08-14T21:58:11.5576192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5576280Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5576522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5576612Z outputs = layer_module( 2025-08-14T21:58:11.5576856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5576923Z outputs = self.rel_attn( 2025-08-14T21:58:11.5577176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.5577275Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.5577278Z 2025-08-14T21:58:11.5577392Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5577593Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5577659Z return mod(**inputs) 2025-08-14T21:58:11.5577924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5578006Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5578247Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5578320Z outputs = layer_module( 2025-08-14T21:58:11.5578561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5578634Z outputs = self.rel_attn( 2025-08-14T21:58:11.5578875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5578947Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5579212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.5579334Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.5579345Z 2025-08-14T21:58:11.5579438Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5579630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5579689Z return mod(**inputs) 2025-08-14T21:58:11.5579927Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5580001Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5580228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5580299Z outputs = layer_module( 2025-08-14T21:58:11.5580528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5580592Z outputs = self.rel_attn( 2025-08-14T21:58:11.5580831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5580915Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5581173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5581274Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5581277Z 2025-08-14T21:58:11.5581377Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5581570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5581632Z return mod(**inputs) 2025-08-14T21:58:11.5581875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5582007Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5582244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5582341Z outputs = layer_module( 2025-08-14T21:58:11.5582583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5582647Z outputs = self.rel_attn( 2025-08-14T21:58:11.5582893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5582988Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5583265Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5583365Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5583370Z 2025-08-14T21:58:11.5583465Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5584652Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5584724Z return mod(**inputs) 2025-08-14T21:58:11.5584975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5585052Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5585284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5585356Z outputs = layer_module( 2025-08-14T21:58:11.5585587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5585779Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5586033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5586108Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5586351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5586419Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5586649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.5586724Z output = self.layer_1(output) 2025-08-14T21:58:11.5586728Z 2025-08-14T21:58:11.5586824Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5587016Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5587079Z return mod(**inputs) 2025-08-14T21:58:11.5587314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5587400Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5587632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5587695Z outputs = layer_module( 2025-08-14T21:58:11.5587934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5588123Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5588371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5588444Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5588678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5588769Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5588998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.5589100Z output = self.activation_function(output) 2025-08-14T21:58:11.5589295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.5589362Z return self.act(input) 2025-08-14T21:58:11.5589365Z 2025-08-14T21:58:11.5589467Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5589650Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5589710Z return mod(**inputs) 2025-08-14T21:58:11.5589965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5590043Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5590293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5590359Z outputs = layer_module( 2025-08-14T21:58:11.5590591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5590792Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5591030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5591107Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5591342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5591408Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5591648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.5591720Z output = self.layer_2(output) 2025-08-14T21:58:11.5591725Z 2025-08-14T21:58:11.5591819Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5592010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5592071Z return mod(**inputs) 2025-08-14T21:58:11.5592313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5592389Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5592674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5592747Z outputs = layer_module( 2025-08-14T21:58:11.5592985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5593058Z outputs = self.rel_attn( 2025-08-14T21:58:11.5593300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.5593401Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.5593405Z 2025-08-14T21:58:11.5593510Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5593706Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5593769Z return mod(**inputs) 2025-08-14T21:58:11.5594022Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5594100Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5594355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5594444Z outputs = layer_module( 2025-08-14T21:58:11.5594688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5594774Z outputs = self.rel_attn( 2025-08-14T21:58:11.5595004Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.5595108Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.5595111Z 2025-08-14T21:58:11.5595205Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5595384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5595451Z return mod(**inputs) 2025-08-14T21:58:11.5595698Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5595777Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5596016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5596091Z outputs = layer_module( 2025-08-14T21:58:11.5596332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5596394Z outputs = self.rel_attn( 2025-08-14T21:58:11.5596629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5596704Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5596950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.5597070Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.5597081Z 2025-08-14T21:58:11.5597175Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5597357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5597424Z return mod(**inputs) 2025-08-14T21:58:11.5597658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5597732Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5597971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5598036Z outputs = layer_module( 2025-08-14T21:58:11.5598272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5598334Z outputs = self.rel_attn( 2025-08-14T21:58:11.5598566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.5598693Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.5598697Z 2025-08-14T21:58:11.5598789Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5598975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5599042Z return mod(**inputs) 2025-08-14T21:58:11.5599276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5599358Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5599589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5599651Z outputs = layer_module( 2025-08-14T21:58:11.5599897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5599981Z outputs = self.rel_attn( 2025-08-14T21:58:11.5600224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5600294Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5600565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.5600692Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.5600696Z 2025-08-14T21:58:11.5600791Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5600975Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5601045Z return mod(**inputs) 2025-08-14T21:58:11.5601302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5601391Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5601636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5601768Z outputs = layer_module( 2025-08-14T21:58:11.5602024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5602090Z outputs = self.rel_attn( 2025-08-14T21:58:11.5602338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.5602436Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.5602440Z 2025-08-14T21:58:11.5602539Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5602736Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5602801Z return mod(**inputs) 2025-08-14T21:58:11.5603048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5603148Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5603386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5603461Z outputs = layer_module( 2025-08-14T21:58:11.5603699Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5603764Z outputs = self.rel_attn( 2025-08-14T21:58:11.5604016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5604086Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5604346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.5604473Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.5604476Z 2025-08-14T21:58:11.5604573Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5604773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5604839Z return mod(**inputs) 2025-08-14T21:58:11.5605086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5605175Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5605423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5605497Z outputs = layer_module( 2025-08-14T21:58:11.5605824Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5605898Z outputs = self.rel_attn( 2025-08-14T21:58:11.5606195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5606292Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5606600Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5606731Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5606736Z 2025-08-14T21:58:11.5606845Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5607064Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5607133Z return mod(**inputs) 2025-08-14T21:58:11.5607401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5607511Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5607775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5607848Z outputs = layer_module( 2025-08-14T21:58:11.5608105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5608173Z outputs = self.rel_attn( 2025-08-14T21:58:11.5608425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5608509Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5608767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5608878Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5608881Z 2025-08-14T21:58:11.5608978Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5609175Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5609236Z return mod(**inputs) 2025-08-14T21:58:11.5609480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5609569Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5609809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5609879Z outputs = layer_module( 2025-08-14T21:58:11.5610116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5610312Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5610572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5610646Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5610893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5610967Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5611205Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.5611281Z output = self.layer_1(output) 2025-08-14T21:58:11.5611284Z 2025-08-14T21:58:11.5611378Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5611572Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5611641Z return mod(**inputs) 2025-08-14T21:58:11.5611886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5611971Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5612225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5612289Z outputs = layer_module( 2025-08-14T21:58:11.5612549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5612745Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5613475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5613550Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5613794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5613886Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5614127Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.5614213Z output = self.activation_function(output) 2025-08-14T21:58:11.5614436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.5614507Z return self.act(input) 2025-08-14T21:58:11.5614511Z 2025-08-14T21:58:11.5614619Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5614804Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5614866Z return mod(**inputs) 2025-08-14T21:58:11.5615109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5615187Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5615431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5615496Z outputs = layer_module( 2025-08-14T21:58:11.5615733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5615936Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5616179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5616253Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5616498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5616566Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5616810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.5616883Z output = self.layer_2(output) 2025-08-14T21:58:11.5616886Z 2025-08-14T21:58:11.5616984Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5617188Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5617252Z return mod(**inputs) 2025-08-14T21:58:11.5617499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5617577Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5617817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5617887Z outputs = layer_module( 2025-08-14T21:58:11.5618121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5618186Z outputs = self.rel_attn( 2025-08-14T21:58:11.5618429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.5618540Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.5618545Z 2025-08-14T21:58:11.5618665Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5618859Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5618919Z return mod(**inputs) 2025-08-14T21:58:11.5619161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5619236Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5619468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5619538Z outputs = layer_module( 2025-08-14T21:58:11.5619782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5619855Z outputs = self.rel_attn( 2025-08-14T21:58:11.5620100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.5620197Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.5620201Z 2025-08-14T21:58:11.5620301Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5620485Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5620552Z return mod(**inputs) 2025-08-14T21:58:11.5620784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5620860Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5621101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5621166Z outputs = layer_module( 2025-08-14T21:58:11.5621411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5621487Z outputs = self.rel_attn( 2025-08-14T21:58:11.5621732Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5621814Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5622074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.5622199Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.5622203Z 2025-08-14T21:58:11.5622312Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5622507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5622580Z return mod(**inputs) 2025-08-14T21:58:11.5622826Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5622908Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5623167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5623228Z outputs = layer_module( 2025-08-14T21:58:11.5623460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5623530Z outputs = self.rel_attn( 2025-08-14T21:58:11.5623761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.5623887Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.5623892Z 2025-08-14T21:58:11.5623987Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5624181Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5624252Z return mod(**inputs) 2025-08-14T21:58:11.5624499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5624580Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5624811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5624873Z outputs = layer_module( 2025-08-14T21:58:11.5625110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5625172Z outputs = self.rel_attn( 2025-08-14T21:58:11.5625415Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5625494Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5625764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.5625892Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.5625894Z 2025-08-14T21:58:11.5625986Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5626166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5626235Z return mod(**inputs) 2025-08-14T21:58:11.5626467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5626547Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5626778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5626841Z outputs = layer_module( 2025-08-14T21:58:11.5627078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5627141Z outputs = self.rel_attn( 2025-08-14T21:58:11.5627373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.5627472Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.5627475Z 2025-08-14T21:58:11.5627567Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5627751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5627811Z return mod(**inputs) 2025-08-14T21:58:11.5628041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5628122Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5628350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5628413Z outputs = layer_module( 2025-08-14T21:58:11.5628649Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5628712Z outputs = self.rel_attn( 2025-08-14T21:58:11.5628947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5629012Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5629254Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.5629372Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.5629375Z 2025-08-14T21:58:11.5629470Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5629680Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5629741Z return mod(**inputs) 2025-08-14T21:58:11.5629975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5630075Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5630305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5630369Z outputs = layer_module( 2025-08-14T21:58:11.5630607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5630669Z outputs = self.rel_attn( 2025-08-14T21:58:11.5630964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5631050Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5631307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5631432Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5631437Z 2025-08-14T21:58:11.5631536Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5631728Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5631790Z return mod(**inputs) 2025-08-14T21:58:11.5632030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5632114Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5632357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5632418Z outputs = layer_module( 2025-08-14T21:58:11.5632657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5632720Z outputs = self.rel_attn( 2025-08-14T21:58:11.5632956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5633038Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5633288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5633397Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5633400Z 2025-08-14T21:58:11.5633493Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5633682Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5633745Z return mod(**inputs) 2025-08-14T21:58:11.5633980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5634065Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5634295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5634359Z outputs = layer_module( 2025-08-14T21:58:11.5634599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5634787Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5635030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5635101Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5635334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5635427Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5635658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.5635747Z output = self.layer_1(output) 2025-08-14T21:58:11.5635750Z 2025-08-14T21:58:11.5635844Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5636026Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5636095Z return mod(**inputs) 2025-08-14T21:58:11.5636328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5636404Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5636653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5636715Z outputs = layer_module( 2025-08-14T21:58:11.5636969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5637161Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5637401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5637479Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5637832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5637913Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5638145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.5638230Z output = self.activation_function(output) 2025-08-14T21:58:11.5638434Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.5638502Z return self.act(input) 2025-08-14T21:58:11.5638507Z 2025-08-14T21:58:11.5638607Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5638796Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5638856Z return mod(**inputs) 2025-08-14T21:58:11.5639095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5639171Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5639408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5639481Z outputs = layer_module( 2025-08-14T21:58:11.5639717Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5639930Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5640170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5640241Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5640483Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5640551Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5640784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.5640863Z output = self.layer_2(output) 2025-08-14T21:58:11.5640866Z 2025-08-14T21:58:11.5640968Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5641166Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5641268Z return mod(**inputs) 2025-08-14T21:58:11.5641510Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5641618Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5641854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5641918Z outputs = layer_module( 2025-08-14T21:58:11.5642157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5642221Z outputs = self.rel_attn( 2025-08-14T21:58:11.5642466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.5642584Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.5642589Z 2025-08-14T21:58:11.5642691Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5642922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5642988Z return mod(**inputs) 2025-08-14T21:58:11.5643232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5643312Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5643547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5643619Z outputs = layer_module( 2025-08-14T21:58:11.5643854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5643918Z outputs = self.rel_attn( 2025-08-14T21:58:11.5644163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.5644261Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.5644266Z 2025-08-14T21:58:11.5644372Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5644561Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5644624Z return mod(**inputs) 2025-08-14T21:58:11.5644869Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5644945Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5645190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5645255Z outputs = layer_module( 2025-08-14T21:58:11.5645499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5645576Z outputs = self.rel_attn( 2025-08-14T21:58:11.5645884Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5645960Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5646234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.5646360Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.5646364Z 2025-08-14T21:58:11.5646468Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5646665Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5646728Z return mod(**inputs) 2025-08-14T21:58:11.5646977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5647073Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5647318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5647401Z outputs = layer_module( 2025-08-14T21:58:11.5647641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5647716Z outputs = self.rel_attn( 2025-08-14T21:58:11.5647954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.5648078Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.5648081Z 2025-08-14T21:58:11.5648187Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5648397Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5648469Z return mod(**inputs) 2025-08-14T21:58:11.5648708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5648803Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5649048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5649112Z outputs = layer_module( 2025-08-14T21:58:11.5649346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5649417Z outputs = self.rel_attn( 2025-08-14T21:58:11.5649655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5649731Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5649984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.5650108Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.5650110Z 2025-08-14T21:58:11.5650217Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5650404Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5650474Z return mod(**inputs) 2025-08-14T21:58:11.5650709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5650785Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5651030Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5651093Z outputs = layer_module( 2025-08-14T21:58:11.5651330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5651404Z outputs = self.rel_attn( 2025-08-14T21:58:11.5651638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.5651742Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.5651746Z 2025-08-14T21:58:11.5651842Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5652028Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5652097Z return mod(**inputs) 2025-08-14T21:58:11.5652332Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5652415Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5652656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5652738Z outputs = layer_module( 2025-08-14T21:58:11.5652986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5653051Z outputs = self.rel_attn( 2025-08-14T21:58:11.5653307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5653384Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5653642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.5653767Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.5653770Z 2025-08-14T21:58:11.5653880Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5654063Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5654147Z return mod(**inputs) 2025-08-14T21:58:11.5654387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5662571Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5663109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5663185Z outputs = layer_module( 2025-08-14T21:58:11.5663443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5663522Z outputs = self.rel_attn( 2025-08-14T21:58:11.5663763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5663859Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5664121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5664234Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5664240Z 2025-08-14T21:58:11.5664390Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5664587Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5664655Z return mod(**inputs) 2025-08-14T21:58:11.5664900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5664984Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5665223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5665289Z outputs = layer_module( 2025-08-14T21:58:11.5665524Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5665601Z outputs = self.rel_attn( 2025-08-14T21:58:11.5665833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5665921Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5666183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5666286Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5666290Z 2025-08-14T21:58:11.5666398Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5666585Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5666647Z return mod(**inputs) 2025-08-14T21:58:11.5666889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5666969Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5667313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5667383Z outputs = layer_module( 2025-08-14T21:58:11.5667646Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5667855Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5668110Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5668200Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5668436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5668508Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5668753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.5668838Z output = self.layer_1(output) 2025-08-14T21:58:11.5668842Z 2025-08-14T21:58:11.5669032Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5669231Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5669292Z return mod(**inputs) 2025-08-14T21:58:11.5669533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5669611Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5669839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5669910Z outputs = layer_module( 2025-08-14T21:58:11.5670139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5670336Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5670591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5670665Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5670904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5670971Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5671199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.5671293Z output = self.activation_function(output) 2025-08-14T21:58:11.5671491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.5671565Z return self.act(input) 2025-08-14T21:58:11.5671570Z 2025-08-14T21:58:11.5671670Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5671863Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5671935Z return mod(**inputs) 2025-08-14T21:58:11.5672170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5672249Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5672491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5672554Z outputs = layer_module( 2025-08-14T21:58:11.5672799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5673007Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5673260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5673342Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5673589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5673661Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5673892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.5673959Z output = self.layer_2(output) 2025-08-14T21:58:11.5673963Z 2025-08-14T21:58:11.5674066Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5674249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5674310Z return mod(**inputs) 2025-08-14T21:58:11.5674549Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5674628Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5674933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5674998Z outputs = layer_module( 2025-08-14T21:58:11.5675228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5675300Z outputs = self.rel_attn( 2025-08-14T21:58:11.5675528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.5675630Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.5675634Z 2025-08-14T21:58:11.5675726Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5675910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5675978Z return mod(**inputs) 2025-08-14T21:58:11.5676210Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5676286Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5676520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5676582Z outputs = layer_module( 2025-08-14T21:58:11.5676814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5676878Z outputs = self.rel_attn( 2025-08-14T21:58:11.5677107Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.5677208Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.5677213Z 2025-08-14T21:58:11.5677305Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5677497Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5677558Z return mod(**inputs) 2025-08-14T21:58:11.5677793Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5677880Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5678111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5678173Z outputs = layer_module( 2025-08-14T21:58:11.5678406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5678469Z outputs = self.rel_attn( 2025-08-14T21:58:11.5678703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5678787Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5679042Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.5679224Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.5679228Z 2025-08-14T21:58:11.5679322Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5679512Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5679572Z return mod(**inputs) 2025-08-14T21:58:11.5679804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5679887Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5680118Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5680181Z outputs = layer_module( 2025-08-14T21:58:11.5680452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5680517Z outputs = self.rel_attn( 2025-08-14T21:58:11.5680756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.5680879Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.5680883Z 2025-08-14T21:58:11.5680980Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5681177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5681238Z return mod(**inputs) 2025-08-14T21:58:11.5681475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5681559Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5681798Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5681871Z outputs = layer_module( 2025-08-14T21:58:11.5682108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5682173Z outputs = self.rel_attn( 2025-08-14T21:58:11.5682416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5682503Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5682764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.5682888Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.5682893Z 2025-08-14T21:58:11.5682990Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5683186Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5683248Z return mod(**inputs) 2025-08-14T21:58:11.5683489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5683572Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5683808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5683880Z outputs = layer_module( 2025-08-14T21:58:11.5684112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5684175Z outputs = self.rel_attn( 2025-08-14T21:58:11.5684419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.5684533Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.5684537Z 2025-08-14T21:58:11.5684640Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5684845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5684907Z return mod(**inputs) 2025-08-14T21:58:11.5685150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5685227Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5685466Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5685542Z outputs = layer_module( 2025-08-14T21:58:11.5685925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5686009Z outputs = self.rel_attn( 2025-08-14T21:58:11.5686276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5686370Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5686647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.5686773Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.5686779Z 2025-08-14T21:58:11.5686888Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5687086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5687150Z return mod(**inputs) 2025-08-14T21:58:11.5687406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5687484Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5687726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5687803Z outputs = layer_module( 2025-08-14T21:58:11.5688040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5688112Z outputs = self.rel_attn( 2025-08-14T21:58:11.5688347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5688432Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5688694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5688801Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5688804Z 2025-08-14T21:58:11.5688907Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5689094Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5689160Z return mod(**inputs) 2025-08-14T21:58:11.5689409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5689487Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5689722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5689793Z outputs = layer_module( 2025-08-14T21:58:11.5690028Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5690097Z outputs = self.rel_attn( 2025-08-14T21:58:11.5690333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5690432Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5690695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5690812Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5690815Z 2025-08-14T21:58:11.5690916Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5691100Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5691161Z return mod(**inputs) 2025-08-14T21:58:11.5691401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5691477Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5691713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5691785Z outputs = layer_module( 2025-08-14T21:58:11.5692017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5692248Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5692503Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5692577Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5692831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5692899Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5693135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.5693211Z output = self.layer_1(output) 2025-08-14T21:58:11.5693216Z 2025-08-14T21:58:11.5693314Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5693507Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5693571Z return mod(**inputs) 2025-08-14T21:58:11.5693811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5693895Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5694139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5694211Z outputs = layer_module( 2025-08-14T21:58:11.5694450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5694644Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5694901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5694976Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5695222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5695298Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5695536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.5695628Z output = self.activation_function(output) 2025-08-14T21:58:11.5695832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.5695899Z return self.act(input) 2025-08-14T21:58:11.5695902Z 2025-08-14T21:58:11.5696005Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5696192Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5696279Z return mod(**inputs) 2025-08-14T21:58:11.5696525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5696617Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5696860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5696924Z outputs = layer_module( 2025-08-14T21:58:11.5697156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5697356Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5697601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5697681Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5697958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5698028Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5698287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.5698354Z output = self.layer_2(output) 2025-08-14T21:58:11.5698358Z 2025-08-14T21:58:11.5698459Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5698643Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5698703Z return mod(**inputs) 2025-08-14T21:58:11.5698942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5699019Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5699251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5699322Z outputs = layer_module( 2025-08-14T21:58:11.5699557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5699629Z outputs = self.rel_attn( 2025-08-14T21:58:11.5699862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.5699953Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.5699957Z 2025-08-14T21:58:11.5700057Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5700240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5700310Z return mod(**inputs) 2025-08-14T21:58:11.5700542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5700618Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5700859Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5700922Z outputs = layer_module( 2025-08-14T21:58:11.5701154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5701225Z outputs = self.rel_attn( 2025-08-14T21:58:11.5701455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.5701556Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.5701560Z 2025-08-14T21:58:11.5701653Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5701834Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5701920Z return mod(**inputs) 2025-08-14T21:58:11.5702161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5702261Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5702494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5702558Z outputs = layer_module( 2025-08-14T21:58:11.5702796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5702860Z outputs = self.rel_attn( 2025-08-14T21:58:11.5703089Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5703163Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5703407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.5703555Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.5703573Z 2025-08-14T21:58:11.5703667Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5703850Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5703917Z return mod(**inputs) 2025-08-14T21:58:11.5704148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5704223Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5704460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5704521Z outputs = layer_module( 2025-08-14T21:58:11.5704756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5704820Z outputs = self.rel_attn( 2025-08-14T21:58:11.5705051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.5705182Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.5705186Z 2025-08-14T21:58:11.5705280Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5705467Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5705526Z return mod(**inputs) 2025-08-14T21:58:11.5705755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5705839Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5706069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5706133Z outputs = layer_module( 2025-08-14T21:58:11.5706374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5706438Z outputs = self.rel_attn( 2025-08-14T21:58:11.5706676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5706743Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5706990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.5707115Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.5707118Z 2025-08-14T21:58:11.5707210Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5707400Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5707477Z return mod(**inputs) 2025-08-14T21:58:11.5707714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5707815Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5708045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5708107Z outputs = layer_module( 2025-08-14T21:58:11.5708341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5708404Z outputs = self.rel_attn( 2025-08-14T21:58:11.5708641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.5708732Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.5708736Z 2025-08-14T21:58:11.5708831Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5709022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5709111Z return mod(**inputs) 2025-08-14T21:58:11.5709354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5709430Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5709658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5709728Z outputs = layer_module( 2025-08-14T21:58:11.5709959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5710020Z outputs = self.rel_attn( 2025-08-14T21:58:11.5710255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5710324Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5710577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.5710693Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.5710697Z 2025-08-14T21:58:11.5710791Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5710982Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5711043Z return mod(**inputs) 2025-08-14T21:58:11.5711280Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5711355Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5711584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5711654Z outputs = layer_module( 2025-08-14T21:58:11.5711882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5711948Z outputs = self.rel_attn( 2025-08-14T21:58:11.5712189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5712272Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5712528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5712631Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5712635Z 2025-08-14T21:58:11.5712729Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5712921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5712997Z return mod(**inputs) 2025-08-14T21:58:11.5713235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5713320Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5713567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5713637Z outputs = layer_module( 2025-08-14T21:58:11.5713865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5713929Z outputs = self.rel_attn( 2025-08-14T21:58:11.5714168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5714249Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5714504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5714604Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5714607Z 2025-08-14T21:58:11.5714728Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5714921Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5714982Z return mod(**inputs) 2025-08-14T21:58:11.5715215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5715301Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5715533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5715602Z outputs = layer_module( 2025-08-14T21:58:11.5715833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5716027Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5716278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5716352Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5716589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5716659Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5716888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.5716963Z output = self.layer_1(output) 2025-08-14T21:58:11.5716966Z 2025-08-14T21:58:11.5717059Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5717240Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5717309Z return mod(**inputs) 2025-08-14T21:58:11.5717543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5717626Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5717856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5717917Z outputs = layer_module( 2025-08-14T21:58:11.5718153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5718340Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5718584Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5718654Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5718904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5718981Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5719233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.5719315Z output = self.activation_function(output) 2025-08-14T21:58:11.5719515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.5719580Z return self.act(input) 2025-08-14T21:58:11.5719583Z 2025-08-14T21:58:11.5719683Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5719862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5719921Z return mod(**inputs) 2025-08-14T21:58:11.5720159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5720234Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5720500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5720564Z outputs = layer_module( 2025-08-14T21:58:11.5720794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5720987Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5721224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5721294Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5721532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5721600Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5721838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.5721906Z output = self.layer_2(output) 2025-08-14T21:58:11.5721909Z 2025-08-14T21:58:11.5722001Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5722190Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5722250Z return mod(**inputs) 2025-08-14T21:58:11.5722489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5722565Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5722795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5722866Z outputs = layer_module( 2025-08-14T21:58:11.5723093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5723159Z outputs = self.rel_attn( 2025-08-14T21:58:11.5723401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.5723495Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.5723498Z 2025-08-14T21:58:11.5723602Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5723787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5723848Z return mod(**inputs) 2025-08-14T21:58:11.5724100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5724175Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5724426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5724488Z outputs = layer_module( 2025-08-14T21:58:11.5724718Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5724811Z outputs = self.rel_attn( 2025-08-14T21:58:11.5725062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.5725157Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.5725160Z 2025-08-14T21:58:11.5725263Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5725448Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5725516Z return mod(**inputs) 2025-08-14T21:58:11.5725838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5725927Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5726209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5726278Z outputs = layer_module( 2025-08-14T21:58:11.5726546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5726629Z outputs = self.rel_attn( 2025-08-14T21:58:11.5726901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5726991Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5727274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.5727413Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.5727419Z 2025-08-14T21:58:11.5727538Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5727753Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5727826Z return mod(**inputs) 2025-08-14T21:58:11.5728066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5728143Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5728394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5728457Z outputs = layer_module( 2025-08-14T21:58:11.5728689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5728765Z outputs = self.rel_attn( 2025-08-14T21:58:11.5729027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.5729173Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.5729181Z 2025-08-14T21:58:11.5729287Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5729496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5729573Z return mod(**inputs) 2025-08-14T21:58:11.5729843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5729936Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5730202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5730272Z outputs = layer_module( 2025-08-14T21:58:11.5730542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5730631Z outputs = self.rel_attn( 2025-08-14T21:58:11.5730901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5731001Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5731286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.5731428Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.5731432Z 2025-08-14T21:58:11.5731537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5731744Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5731821Z return mod(**inputs) 2025-08-14T21:58:11.5732086Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5732180Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5732486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5732559Z outputs = layer_module( 2025-08-14T21:58:11.5732831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5732904Z outputs = self.rel_attn( 2025-08-14T21:58:11.5733162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.5733273Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.5733278Z 2025-08-14T21:58:11.5733383Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5733594Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5733664Z return mod(**inputs) 2025-08-14T21:58:11.5733942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5734037Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5734300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5734369Z outputs = layer_module( 2025-08-14T21:58:11.5734601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5734665Z outputs = self.rel_attn( 2025-08-14T21:58:11.5734906Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5734973Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5735222Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.5735348Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.5735352Z 2025-08-14T21:58:11.5735448Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5735641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5735704Z return mod(**inputs) 2025-08-14T21:58:11.5735940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5736024Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5736263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5736326Z outputs = layer_module( 2025-08-14T21:58:11.5736567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5736648Z outputs = self.rel_attn( 2025-08-14T21:58:11.5736897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5737006Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5737260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5737371Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5737374Z 2025-08-14T21:58:11.5737470Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5737824Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5737897Z return mod(**inputs) 2025-08-14T21:58:11.5738143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5738236Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5738573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5738643Z outputs = layer_module( 2025-08-14T21:58:11.5738888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5738953Z outputs = self.rel_attn( 2025-08-14T21:58:11.5739198Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5739285Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5739548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5739659Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5739662Z 2025-08-14T21:58:11.5739764Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5739960Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5740026Z return mod(**inputs) 2025-08-14T21:58:11.5740266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5740364Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5740608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5740674Z outputs = layer_module( 2025-08-14T21:58:11.5740922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5741119Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5741375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5741448Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5741687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5741766Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5742002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.5742076Z output = self.layer_1(output) 2025-08-14T21:58:11.5742079Z 2025-08-14T21:58:11.5742176Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5742362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5742427Z return mod(**inputs) 2025-08-14T21:58:11.5742664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5742782Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5743032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5743118Z outputs = layer_module( 2025-08-14T21:58:11.5743362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5743552Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5743796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5743875Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5744114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5744191Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5744441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.5744544Z output = self.activation_function(output) 2025-08-14T21:58:11.5744750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.5744816Z return self.act(input) 2025-08-14T21:58:11.5744819Z 2025-08-14T21:58:11.5744915Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5745108Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5745171Z return mod(**inputs) 2025-08-14T21:58:11.5745412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5745490Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5745728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5745800Z outputs = layer_module( 2025-08-14T21:58:11.5746037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5746238Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5746482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5746554Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5746800Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5746869Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5747103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.5747179Z output = self.layer_2(output) 2025-08-14T21:58:11.5747182Z 2025-08-14T21:58:11.5747281Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5747476Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5747536Z return mod(**inputs) 2025-08-14T21:58:11.5747772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5747857Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5748106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5748174Z outputs = layer_module( 2025-08-14T21:58:11.5748405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5748485Z outputs = self.rel_attn( 2025-08-14T21:58:11.5748722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.5748831Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.5748834Z 2025-08-14T21:58:11.5748925Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5749115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5749173Z return mod(**inputs) 2025-08-14T21:58:11.5749407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5749480Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5749707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5749777Z outputs = layer_module( 2025-08-14T21:58:11.5750006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5750099Z outputs = self.rel_attn( 2025-08-14T21:58:11.5750341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.5750434Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.5750438Z 2025-08-14T21:58:11.5750538Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5750720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5750780Z return mod(**inputs) 2025-08-14T21:58:11.5751015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5751093Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5751334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5751399Z outputs = layer_module( 2025-08-14T21:58:11.5751637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5751709Z outputs = self.rel_attn( 2025-08-14T21:58:11.5751942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5752010Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5752277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.5752395Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.5752398Z 2025-08-14T21:58:11.5752497Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5752677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5752740Z return mod(**inputs) 2025-08-14T21:58:11.5752980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5753056Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5753294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5753355Z outputs = layer_module( 2025-08-14T21:58:11.5753583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5753654Z outputs = self.rel_attn( 2025-08-14T21:58:11.5753882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.5754002Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.5754027Z 2025-08-14T21:58:11.5754121Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5754306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5754390Z return mod(**inputs) 2025-08-14T21:58:11.5754626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5754701Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5754945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5755008Z outputs = layer_module( 2025-08-14T21:58:11.5755248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5755309Z outputs = self.rel_attn( 2025-08-14T21:58:11.5755542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5755618Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5755896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.5756016Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.5756028Z 2025-08-14T21:58:11.5756120Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5756298Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5756368Z return mod(**inputs) 2025-08-14T21:58:11.5756602Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5756678Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5756919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5756983Z outputs = layer_module( 2025-08-14T21:58:11.5757234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5757297Z outputs = self.rel_attn( 2025-08-14T21:58:11.5757526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.5757625Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.5757628Z 2025-08-14T21:58:11.5757720Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5757903Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5757972Z return mod(**inputs) 2025-08-14T21:58:11.5758199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5758283Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5758515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5758578Z outputs = layer_module( 2025-08-14T21:58:11.5758814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5758879Z outputs = self.rel_attn( 2025-08-14T21:58:11.5759106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5759179Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5759423Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.5759543Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.5759562Z 2025-08-14T21:58:11.5759658Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5759843Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5759926Z return mod(**inputs) 2025-08-14T21:58:11.5760162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5760245Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5760476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5760538Z outputs = layer_module( 2025-08-14T21:58:11.5760776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5760840Z outputs = self.rel_attn( 2025-08-14T21:58:11.5761069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5761164Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5761479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5761593Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5761596Z 2025-08-14T21:58:11.5761692Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5761887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5761959Z return mod(**inputs) 2025-08-14T21:58:11.5762206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5762291Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5762540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5762606Z outputs = layer_module( 2025-08-14T21:58:11.5762863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5762931Z outputs = self.rel_attn( 2025-08-14T21:58:11.5763185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5763276Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5763540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5763650Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5763654Z 2025-08-14T21:58:11.5763753Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5763948Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5764020Z return mod(**inputs) 2025-08-14T21:58:11.5764275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5764363Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5764611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5764677Z outputs = layer_module( 2025-08-14T21:58:11.5764934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5765138Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5765394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5765480Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5765822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5765912Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5766184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.5766256Z output = self.layer_1(output) 2025-08-14T21:58:11.5766260Z 2025-08-14T21:58:11.5766370Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5766570Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5766645Z return mod(**inputs) 2025-08-14T21:58:11.5766900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5766980Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5767236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5767306Z outputs = layer_module( 2025-08-14T21:58:11.5767588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5767807Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5768074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5768163Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5768420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5768494Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5768756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.5768846Z output = self.activation_function(output) 2025-08-14T21:58:11.5769069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.5769140Z return self.act(input) 2025-08-14T21:58:11.5769143Z 2025-08-14T21:58:11.5769245Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5769452Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5769517Z return mod(**inputs) 2025-08-14T21:58:11.5769808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5769898Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5770151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5770226Z outputs = layer_module( 2025-08-14T21:58:11.5770479Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5770685Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5770953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5771028Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5771288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5771360Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5771609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.5771687Z output = self.layer_2(output) 2025-08-14T21:58:11.5771690Z 2025-08-14T21:58:11.5771838Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5772036Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5772110Z return mod(**inputs) 2025-08-14T21:58:11.5772381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5772470Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5772725Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5772789Z outputs = layer_module( 2025-08-14T21:58:11.5773032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5773097Z outputs = self.rel_attn( 2025-08-14T21:58:11.5773335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.5773436Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.5773439Z 2025-08-14T21:58:11.5773567Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5773761Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5773824Z return mod(**inputs) 2025-08-14T21:58:11.5774060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5774148Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5774382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5774453Z outputs = layer_module( 2025-08-14T21:58:11.5774688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5774753Z outputs = self.rel_attn( 2025-08-14T21:58:11.5774995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.5775092Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.5775096Z 2025-08-14T21:58:11.5775198Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5775384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5775446Z return mod(**inputs) 2025-08-14T21:58:11.5775692Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5775768Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5776005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5776075Z outputs = layer_module( 2025-08-14T21:58:11.5776310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5776381Z outputs = self.rel_attn( 2025-08-14T21:58:11.5776619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5776687Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5776945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.5777067Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.5777070Z 2025-08-14T21:58:11.5777166Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5777360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5777421Z return mod(**inputs) 2025-08-14T21:58:11.5777664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5777756Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5777997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5778081Z outputs = layer_module( 2025-08-14T21:58:11.5778316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5778386Z outputs = self.rel_attn( 2025-08-14T21:58:11.5778621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.5778743Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.5778746Z 2025-08-14T21:58:11.5778849Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5779033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5779096Z return mod(**inputs) 2025-08-14T21:58:11.5779365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5779444Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5779689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5779752Z outputs = layer_module( 2025-08-14T21:58:11.5779986Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5780057Z outputs = self.rel_attn( 2025-08-14T21:58:11.5780292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5780366Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5780620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.5780740Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.5780744Z 2025-08-14T21:58:11.5780845Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5781031Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5781093Z return mod(**inputs) 2025-08-14T21:58:11.5781366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5781443Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5781685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5781749Z outputs = layer_module( 2025-08-14T21:58:11.5781984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5782056Z outputs = self.rel_attn( 2025-08-14T21:58:11.5782291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.5782387Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.5782398Z 2025-08-14T21:58:11.5782492Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5782675Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5782745Z return mod(**inputs) 2025-08-14T21:58:11.5782981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5783057Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5783302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5783384Z outputs = layer_module( 2025-08-14T21:58:11.5783640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5783735Z outputs = self.rel_attn( 2025-08-14T21:58:11.5783975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5784051Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5784302Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.5784417Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.5784427Z 2025-08-14T21:58:11.5784522Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5784707Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5784777Z return mod(**inputs) 2025-08-14T21:58:11.5785050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5785129Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5785371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5785435Z outputs = layer_module( 2025-08-14T21:58:11.5785678Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5785742Z outputs = self.rel_attn( 2025-08-14T21:58:11.5785979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5786070Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5786330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5786447Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5786458Z 2025-08-14T21:58:11.5786556Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5786737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5786805Z return mod(**inputs) 2025-08-14T21:58:11.5787038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5787114Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5787354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5787415Z outputs = layer_module( 2025-08-14T21:58:11.5787650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5787717Z outputs = self.rel_attn( 2025-08-14T21:58:11.5787951Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5788040Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5788290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5788389Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5788400Z 2025-08-14T21:58:11.5788493Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5788677Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5788743Z return mod(**inputs) 2025-08-14T21:58:11.5788977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5789071Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5789312Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5789390Z outputs = layer_module( 2025-08-14T21:58:11.5789633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5789825Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5790067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5790147Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5790380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5790449Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5790685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.5790778Z output = self.layer_1(output) 2025-08-14T21:58:11.5790782Z 2025-08-14T21:58:11.5790884Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5791072Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5791134Z return mod(**inputs) 2025-08-14T21:58:11.5791376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5791454Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5791702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5791766Z outputs = layer_module( 2025-08-14T21:58:11.5792010Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5792218Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5792469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5792543Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5792789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5792867Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5793102Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.5793182Z output = self.activation_function(output) 2025-08-14T21:58:11.5793375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.5793449Z return self.act(input) 2025-08-14T21:58:11.5793452Z 2025-08-14T21:58:11.5793548Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5793738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5793799Z return mod(**inputs) 2025-08-14T21:58:11.5794029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5794111Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5794340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5794400Z outputs = layer_module( 2025-08-14T21:58:11.5794634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5794837Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5795082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5795170Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5795401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5795475Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5795704Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.5795777Z output = self.layer_2(output) 2025-08-14T21:58:11.5795780Z 2025-08-14T21:58:11.5795875Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5796058Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5796127Z return mod(**inputs) 2025-08-14T21:58:11.5796375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5796465Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5796703Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5796766Z outputs = layer_module( 2025-08-14T21:58:11.5797002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5797067Z outputs = self.rel_attn( 2025-08-14T21:58:11.5797297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.5797397Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.5797401Z 2025-08-14T21:58:11.5797494Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5797685Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5797758Z return mod(**inputs) 2025-08-14T21:58:11.5797993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5798075Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5798303Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5798366Z outputs = layer_module( 2025-08-14T21:58:11.5798599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5798662Z outputs = self.rel_attn( 2025-08-14T21:58:11.5798894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.5798989Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.5798992Z 2025-08-14T21:58:11.5799088Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5799273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5799335Z return mod(**inputs) 2025-08-14T21:58:11.5799564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5799645Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5799875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5799944Z outputs = layer_module( 2025-08-14T21:58:11.5800173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5800236Z outputs = self.rel_attn( 2025-08-14T21:58:11.5800494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5800564Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5800835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.5800954Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.5800958Z 2025-08-14T21:58:11.5801050Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5801239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5801300Z return mod(**inputs) 2025-08-14T21:58:11.5801531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5801614Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5801850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5801934Z outputs = layer_module( 2025-08-14T21:58:11.5802182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5802251Z outputs = self.rel_attn( 2025-08-14T21:58:11.5802497Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.5802620Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.5802623Z 2025-08-14T21:58:11.5802723Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5802907Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5802967Z return mod(**inputs) 2025-08-14T21:58:11.5803211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5803289Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5803525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5803596Z outputs = layer_module( 2025-08-14T21:58:11.5803834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5803904Z outputs = self.rel_attn( 2025-08-14T21:58:11.5804144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5804210Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5804473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.5804593Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.5804597Z 2025-08-14T21:58:11.5804702Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5804891Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5804955Z return mod(**inputs) 2025-08-14T21:58:11.5805207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5805285Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5805530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5805602Z outputs = layer_module( 2025-08-14T21:58:11.5805946Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5806023Z outputs = self.rel_attn( 2025-08-14T21:58:11.5806284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.5806381Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.5806405Z 2025-08-14T21:58:11.5806514Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5806701Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5806766Z return mod(**inputs) 2025-08-14T21:58:11.5807026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5807109Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5807378Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5807442Z outputs = layer_module( 2025-08-14T21:58:11.5807683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5807758Z outputs = self.rel_attn( 2025-08-14T21:58:11.5808029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5808110Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5808362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.5808478Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.5808481Z 2025-08-14T21:58:11.5808587Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5808770Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5808831Z return mod(**inputs) 2025-08-14T21:58:11.5809075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5809155Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5809400Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5809466Z outputs = layer_module( 2025-08-14T21:58:11.5809706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5809778Z outputs = self.rel_attn( 2025-08-14T21:58:11.5810017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5810109Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5810364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5810466Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5810471Z 2025-08-14T21:58:11.5810576Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5810762Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5810826Z return mod(**inputs) 2025-08-14T21:58:11.5811075Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5811153Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5811420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5811485Z outputs = layer_module( 2025-08-14T21:58:11.5811729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5811804Z outputs = self.rel_attn( 2025-08-14T21:58:11.5812062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5812170Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5812433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5812556Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5812559Z 2025-08-14T21:58:11.5812662Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5812845Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5812906Z return mod(**inputs) 2025-08-14T21:58:11.5813151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5813227Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5813469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5813533Z outputs = layer_module( 2025-08-14T21:58:11.5813796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5814000Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5814244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5814324Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5814564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5814633Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5814872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.5814942Z output = self.layer_1(output) 2025-08-14T21:58:11.5814945Z 2025-08-14T21:58:11.5815044Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5815239Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5815302Z return mod(**inputs) 2025-08-14T21:58:11.5815546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5815621Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5815857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5815928Z outputs = layer_module( 2025-08-14T21:58:11.5816163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5816364Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5816610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5816685Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5816934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5817003Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5817242Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.5817332Z output = self.activation_function(output) 2025-08-14T21:58:11.5817529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.5817603Z return self.act(input) 2025-08-14T21:58:11.5817606Z 2025-08-14T21:58:11.5817702Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5817906Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5817977Z return mod(**inputs) 2025-08-14T21:58:11.5818515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5818601Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5818842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5818907Z outputs = layer_module( 2025-08-14T21:58:11.5819155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5819348Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5819599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5819682Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5819959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5820040Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5820283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.5820352Z output = self.layer_2(output) 2025-08-14T21:58:11.5820355Z 2025-08-14T21:58:11.5820462Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5820651Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5820727Z return mod(**inputs) 2025-08-14T21:58:11.5820961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5821041Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5821284Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5821348Z outputs = layer_module( 2025-08-14T21:58:11.5821582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5821655Z outputs = self.rel_attn( 2025-08-14T21:58:11.5821891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.5821988Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.5821991Z 2025-08-14T21:58:11.5822086Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5822267Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5822339Z return mod(**inputs) 2025-08-14T21:58:11.5822573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5822653Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5822893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5822956Z outputs = layer_module( 2025-08-14T21:58:11.5823195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5823261Z outputs = self.rel_attn( 2025-08-14T21:58:11.5823492Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.5823590Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.5823593Z 2025-08-14T21:58:11.5823688Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5823895Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5823957Z return mod(**inputs) 2025-08-14T21:58:11.5824196Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5824296Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5824536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5824599Z outputs = layer_module( 2025-08-14T21:58:11.5824886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5824949Z outputs = self.rel_attn( 2025-08-14T21:58:11.5825185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5825256Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5825517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.5825659Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.5825663Z 2025-08-14T21:58:11.5825757Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5825945Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5826006Z return mod(**inputs) 2025-08-14T21:58:11.5826239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5826320Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5826551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5826615Z outputs = layer_module( 2025-08-14T21:58:11.5826852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5826917Z outputs = self.rel_attn( 2025-08-14T21:58:11.5827156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.5827281Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.5827284Z 2025-08-14T21:58:11.5827378Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5827568Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5827628Z return mod(**inputs) 2025-08-14T21:58:11.5827867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5827942Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5828173Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5828246Z outputs = layer_module( 2025-08-14T21:58:11.5828478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5828541Z outputs = self.rel_attn( 2025-08-14T21:58:11.5828780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5828846Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5829101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.5829217Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.5829220Z 2025-08-14T21:58:11.5829315Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5829531Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5829591Z return mod(**inputs) 2025-08-14T21:58:11.5829832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5829925Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5830155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5830225Z outputs = layer_module( 2025-08-14T21:58:11.5830455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5830517Z outputs = self.rel_attn( 2025-08-14T21:58:11.5830756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.5830847Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.5830852Z 2025-08-14T21:58:11.5830951Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5831161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5831225Z return mod(**inputs) 2025-08-14T21:58:11.5831467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5831545Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5831782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5831854Z outputs = layer_module( 2025-08-14T21:58:11.5832093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5832170Z outputs = self.rel_attn( 2025-08-14T21:58:11.5832409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5832477Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5832750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.5832862Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.5832866Z 2025-08-14T21:58:11.5832969Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5833149Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5833210Z return mod(**inputs) 2025-08-14T21:58:11.5833447Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5833522Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5833753Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5833825Z outputs = layer_module( 2025-08-14T21:58:11.5834059Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5834129Z outputs = self.rel_attn( 2025-08-14T21:58:11.5834364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5834448Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5834713Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5834820Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5834824Z 2025-08-14T21:58:11.5834926Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5835114Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5835200Z return mod(**inputs) 2025-08-14T21:58:11.5835446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5835545Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5835776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5835848Z outputs = layer_module( 2025-08-14T21:58:11.5836073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5836144Z outputs = self.rel_attn( 2025-08-14T21:58:11.5836373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5836452Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5836710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5836837Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5836841Z 2025-08-14T21:58:11.5836945Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5837128Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5837188Z return mod(**inputs) 2025-08-14T21:58:11.5837425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5837498Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5837872Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5837948Z outputs = layer_module( 2025-08-14T21:58:11.5838182Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5838389Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5838636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5838711Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5838963Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5839034Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5839278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.5839348Z output = self.layer_1(output) 2025-08-14T21:58:11.5839351Z 2025-08-14T21:58:11.5839449Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5839641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5839704Z return mod(**inputs) 2025-08-14T21:58:11.5839941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5840029Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5840263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5840335Z outputs = layer_module( 2025-08-14T21:58:11.5840568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5840761Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5841014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5841153Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5841411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5841507Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5841748Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.5841842Z output = self.activation_function(output) 2025-08-14T21:58:11.5842054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.5842124Z return self.act(input) 2025-08-14T21:58:11.5842128Z 2025-08-14T21:58:11.5842238Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5842428Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5842502Z return mod(**inputs) 2025-08-14T21:58:11.5842749Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5842878Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5843132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5843197Z outputs = layer_module( 2025-08-14T21:58:11.5843438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5843643Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5843900Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5843979Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5844218Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5844284Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5844531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.5844601Z output = self.layer_2(output) 2025-08-14T21:58:11.5844605Z 2025-08-14T21:58:11.5844708Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5844892Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5844954Z return mod(**inputs) 2025-08-14T21:58:11.5845199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5845275Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5845514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5845586Z outputs = layer_module( 2025-08-14T21:58:11.5845878Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5845961Z outputs = self.rel_attn( 2025-08-14T21:58:11.5846211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.5846310Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.5846314Z 2025-08-14T21:58:11.5846424Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5846623Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5846705Z return mod(**inputs) 2025-08-14T21:58:11.5846939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5847036Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5847281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5847358Z outputs = layer_module( 2025-08-14T21:58:11.5847592Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5847664Z outputs = self.rel_attn( 2025-08-14T21:58:11.5847898Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.5847997Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.5848001Z 2025-08-14T21:58:11.5848096Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5848280Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5848350Z return mod(**inputs) 2025-08-14T21:58:11.5848586Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5848706Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5848952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5849018Z outputs = layer_module( 2025-08-14T21:58:11.5849263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5849328Z outputs = self.rel_attn( 2025-08-14T21:58:11.5849565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5849641Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5849895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.5850025Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.5850028Z 2025-08-14T21:58:11.5850127Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5850315Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5850384Z return mod(**inputs) 2025-08-14T21:58:11.5850625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5850707Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5850944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5851007Z outputs = layer_module( 2025-08-14T21:58:11.5851253Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5851318Z outputs = self.rel_attn( 2025-08-14T21:58:11.5851554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.5851688Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.5851692Z 2025-08-14T21:58:11.5851788Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5851986Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5852047Z return mod(**inputs) 2025-08-14T21:58:11.5852285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5852370Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5852605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5852675Z outputs = layer_module( 2025-08-14T21:58:11.5852928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5852994Z outputs = self.rel_attn( 2025-08-14T21:58:11.5853255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5853326Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5853580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.5853710Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.5853714Z 2025-08-14T21:58:11.5853809Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5854001Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5854063Z return mod(**inputs) 2025-08-14T21:58:11.5854301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5854419Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5854662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5854726Z outputs = layer_module( 2025-08-14T21:58:11.5854971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5855037Z outputs = self.rel_attn( 2025-08-14T21:58:11.5855286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.5855383Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.5855387Z 2025-08-14T21:58:11.5855492Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5855689Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5855754Z return mod(**inputs) 2025-08-14T21:58:11.5856007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5856089Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5856333Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5856408Z outputs = layer_module( 2025-08-14T21:58:11.5856647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5856714Z outputs = self.rel_attn( 2025-08-14T21:58:11.5856964Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5857035Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5857301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.5857421Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.5857425Z 2025-08-14T21:58:11.5857523Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5857720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5857785Z return mod(**inputs) 2025-08-14T21:58:11.5858038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5858118Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5858364Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5858437Z outputs = layer_module( 2025-08-14T21:58:11.5858688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5858751Z outputs = self.rel_attn( 2025-08-14T21:58:11.5858998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5859097Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5859361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5859467Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5859470Z 2025-08-14T21:58:11.5859565Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5859760Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5859824Z return mod(**inputs) 2025-08-14T21:58:11.5860068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5860147Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5860448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5860531Z outputs = layer_module( 2025-08-14T21:58:11.5860765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5860828Z outputs = self.rel_attn( 2025-08-14T21:58:11.5861065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5861149Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5861413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5861518Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5861521Z 2025-08-14T21:58:11.5861618Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5861812Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5861875Z return mod(**inputs) 2025-08-14T21:58:11.5862109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5862191Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5862424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5862492Z outputs = layer_module( 2025-08-14T21:58:11.5862728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5862922Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5863179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5863252Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5863496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5863563Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5863796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.5863872Z output = self.layer_1(output) 2025-08-14T21:58:11.5863875Z 2025-08-14T21:58:11.5863969Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5864161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5864223Z return mod(**inputs) 2025-08-14T21:58:11.5864475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5864562Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5864812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5864884Z outputs = layer_module( 2025-08-14T21:58:11.5865120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5865308Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5865551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5865623Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5865853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5865928Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5866186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.5866276Z output = self.activation_function(output) 2025-08-14T21:58:11.5866471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.5866535Z return self.act(input) 2025-08-14T21:58:11.5866539Z 2025-08-14T21:58:11.5866640Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5866828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5866889Z return mod(**inputs) 2025-08-14T21:58:11.5867125Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5867202Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5867441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5867504Z outputs = layer_module( 2025-08-14T21:58:11.5867735Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5867929Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5868168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5868246Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5868478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5868545Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5868786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.5868856Z output = self.layer_2(output) 2025-08-14T21:58:11.5868860Z 2025-08-14T21:58:11.5868956Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5869147Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5869207Z return mod(**inputs) 2025-08-14T21:58:11.5869446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5869521Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5869751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5869825Z outputs = layer_module( 2025-08-14T21:58:11.5870053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5870134Z outputs = self.rel_attn( 2025-08-14T21:58:11.5870373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.5870480Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.5870483Z 2025-08-14T21:58:11.5870584Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5870765Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5870824Z return mod(**inputs) 2025-08-14T21:58:11.5871063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5871138Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5871375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5871441Z outputs = layer_module( 2025-08-14T21:58:11.5871708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5871782Z outputs = self.rel_attn( 2025-08-14T21:58:11.5872013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.5872105Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.5872116Z 2025-08-14T21:58:11.5872213Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5872394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5872463Z return mod(**inputs) 2025-08-14T21:58:11.5872694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5872771Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5873017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5873083Z outputs = layer_module( 2025-08-14T21:58:11.5873319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5873382Z outputs = self.rel_attn( 2025-08-14T21:58:11.5873611Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5873686Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5873931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.5874050Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.5874054Z 2025-08-14T21:58:11.5874158Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5874340Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5874410Z return mod(**inputs) 2025-08-14T21:58:11.5874644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5874719Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5874959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5875019Z outputs = layer_module( 2025-08-14T21:58:11.5875248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5875317Z outputs = self.rel_attn( 2025-08-14T21:58:11.5875545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.5875696Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.5875699Z 2025-08-14T21:58:11.5875796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5875994Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5876061Z return mod(**inputs) 2025-08-14T21:58:11.5876290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5876372Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5876598Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5876659Z outputs = layer_module( 2025-08-14T21:58:11.5876897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5876960Z outputs = self.rel_attn( 2025-08-14T21:58:11.5877200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5877289Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5877540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.5877663Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.5877665Z 2025-08-14T21:58:11.5877759Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5877941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5878011Z return mod(**inputs) 2025-08-14T21:58:11.5878244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5878327Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5878560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5878624Z outputs = layer_module( 2025-08-14T21:58:11.5878864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5878926Z outputs = self.rel_attn( 2025-08-14T21:58:11.5879158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.5879258Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.5879262Z 2025-08-14T21:58:11.5879356Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5879545Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5879604Z return mod(**inputs) 2025-08-14T21:58:11.5879837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5879920Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5880154Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5880224Z outputs = layer_module( 2025-08-14T21:58:11.5880452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5880515Z outputs = self.rel_attn( 2025-08-14T21:58:11.5880751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5880819Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5881068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.5881205Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.5881209Z 2025-08-14T21:58:11.5881302Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5881496Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5881574Z return mod(**inputs) 2025-08-14T21:58:11.5881815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5881898Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5882139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5882212Z outputs = layer_module( 2025-08-14T21:58:11.5882455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5882522Z outputs = self.rel_attn( 2025-08-14T21:58:11.5882786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5882900Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5883158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5883273Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5883277Z 2025-08-14T21:58:11.5883376Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5883571Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5883633Z return mod(**inputs) 2025-08-14T21:58:11.5883868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5883951Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5884189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5884253Z outputs = layer_module( 2025-08-14T21:58:11.5884496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5884562Z outputs = self.rel_attn( 2025-08-14T21:58:11.5884806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5884887Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5885141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5885250Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5885253Z 2025-08-14T21:58:11.5885348Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5885540Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5885603Z return mod(**inputs) 2025-08-14T21:58:11.5885932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5886027Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5886285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5886358Z outputs = layer_module( 2025-08-14T21:58:11.5886630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5886846Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5887116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5887211Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5887457Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5887551Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5887790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.5887868Z output = self.layer_1(output) 2025-08-14T21:58:11.5887872Z 2025-08-14T21:58:11.5887970Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5888161Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5888233Z return mod(**inputs) 2025-08-14T21:58:11.5888481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5888561Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5888811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5888905Z outputs = layer_module( 2025-08-14T21:58:11.5889159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5889353Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5889599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5889679Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5889918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5889993Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5890234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.5890317Z output = self.activation_function(output) 2025-08-14T21:58:11.5890526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.5890593Z return self.act(input) 2025-08-14T21:58:11.5890597Z 2025-08-14T21:58:11.5890701Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5890886Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5890949Z return mod(**inputs) 2025-08-14T21:58:11.5891193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5891270Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5891509Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5891579Z outputs = layer_module( 2025-08-14T21:58:11.5891816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5892014Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5892255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5892326Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5892568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5892634Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5892917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.5893009Z output = self.layer_2(output) 2025-08-14T21:58:11.5893012Z 2025-08-14T21:58:11.5893109Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5893308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5893395Z return mod(**inputs) 2025-08-14T21:58:11.5893637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5893724Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5893962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5894033Z outputs = layer_module( 2025-08-14T21:58:11.5894273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5894338Z outputs = self.rel_attn( 2025-08-14T21:58:11.5894582Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.5894691Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.5894711Z 2025-08-14T21:58:11.5894812Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5895006Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5895068Z return mod(**inputs) 2025-08-14T21:58:11.5895310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5895386Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5895627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5895699Z outputs = layer_module( 2025-08-14T21:58:11.5895940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5896015Z outputs = self.rel_attn( 2025-08-14T21:58:11.5896250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.5896346Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.5896349Z 2025-08-14T21:58:11.5896453Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5896640Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5896702Z return mod(**inputs) 2025-08-14T21:58:11.5896952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5897030Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5897278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5897344Z outputs = layer_module( 2025-08-14T21:58:11.5897589Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5897664Z outputs = self.rel_attn( 2025-08-14T21:58:11.5897909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5897978Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5898240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.5898363Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.5898366Z 2025-08-14T21:58:11.5898470Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5898656Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5898744Z return mod(**inputs) 2025-08-14T21:58:11.5898984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5899075Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5899311Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5899372Z outputs = layer_module( 2025-08-14T21:58:11.5899597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5899667Z outputs = self.rel_attn( 2025-08-14T21:58:11.5899896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.5900015Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.5900025Z 2025-08-14T21:58:11.5900119Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5900301Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5900394Z return mod(**inputs) 2025-08-14T21:58:11.5900628Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5900704Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5900942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5901004Z outputs = layer_module( 2025-08-14T21:58:11.5901238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5901301Z outputs = self.rel_attn( 2025-08-14T21:58:11.5901530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5901606Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5901855Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.5901975Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.5901986Z 2025-08-14T21:58:11.5902079Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5902259Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5902327Z return mod(**inputs) 2025-08-14T21:58:11.5902559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5902634Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5902871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5902935Z outputs = layer_module( 2025-08-14T21:58:11.5903171Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5903235Z outputs = self.rel_attn( 2025-08-14T21:58:11.5903463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.5903561Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.5903565Z 2025-08-14T21:58:11.5903659Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5903839Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5903907Z return mod(**inputs) 2025-08-14T21:58:11.5904135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5904216Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5904459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5904523Z outputs = layer_module( 2025-08-14T21:58:11.5904778Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5904842Z outputs = self.rel_attn( 2025-08-14T21:58:11.5905079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5905147Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5905392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.5905509Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.5905513Z 2025-08-14T21:58:11.5905604Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5905787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5905854Z return mod(**inputs) 2025-08-14T21:58:11.5906116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5906199Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5906428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5906490Z outputs = layer_module( 2025-08-14T21:58:11.5906726Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5906789Z outputs = self.rel_attn( 2025-08-14T21:58:11.5907017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5907104Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5907357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5907466Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5907469Z 2025-08-14T21:58:11.5907563Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5907745Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5907813Z return mod(**inputs) 2025-08-14T21:58:11.5908044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5908127Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5908358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5908420Z outputs = layer_module( 2025-08-14T21:58:11.5908657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5908723Z outputs = self.rel_attn( 2025-08-14T21:58:11.5908952Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5909039Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5909288Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5909393Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5909396Z 2025-08-14T21:58:11.5909492Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5909672Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5909742Z return mod(**inputs) 2025-08-14T21:58:11.5910001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5910084Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5910334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5910398Z outputs = layer_module( 2025-08-14T21:58:11.5910634Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5910824Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5911069Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5911142Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5911373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5911449Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5911707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.5911777Z output = self.layer_1(output) 2025-08-14T21:58:11.5911780Z 2025-08-14T21:58:11.5911887Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5912068Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5912146Z return mod(**inputs) 2025-08-14T21:58:11.5912379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5912454Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5912693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5912757Z outputs = layer_module( 2025-08-14T21:58:11.5912989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5913191Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5913432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5913511Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5913746Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5913812Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5914049Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.5914131Z output = self.activation_function(output) 2025-08-14T21:58:11.5914334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.5914400Z return self.act(input) 2025-08-14T21:58:11.5914405Z 2025-08-14T21:58:11.5914500Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5914693Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5914754Z return mod(**inputs) 2025-08-14T21:58:11.5914983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5915066Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5915297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5915368Z outputs = layer_module( 2025-08-14T21:58:11.5915599Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5915814Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5916072Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5916142Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5916381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5916448Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5916676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.5916753Z output = self.layer_2(output) 2025-08-14T21:58:11.5916756Z 2025-08-14T21:58:11.5916849Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5917033Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5917102Z return mod(**inputs) 2025-08-14T21:58:11.5917359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5917442Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5917670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5917733Z outputs = layer_module( 2025-08-14T21:58:11.5917970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5918033Z outputs = self.rel_attn( 2025-08-14T21:58:11.5918269Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.5918359Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.5918364Z 2025-08-14T21:58:11.5918457Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5918649Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5918711Z return mod(**inputs) 2025-08-14T21:58:11.5918939Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5919036Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5919266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5919335Z outputs = layer_module( 2025-08-14T21:58:11.5919567Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5919631Z outputs = self.rel_attn( 2025-08-14T21:58:11.5919867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.5919961Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.5919966Z 2025-08-14T21:58:11.5920068Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5920249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5920308Z return mod(**inputs) 2025-08-14T21:58:11.5920544Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5920620Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5920850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5920920Z outputs = layer_module( 2025-08-14T21:58:11.5921151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5921241Z outputs = self.rel_attn( 2025-08-14T21:58:11.5921481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5921568Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5921831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.5921954Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.5921957Z 2025-08-14T21:58:11.5922059Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5922244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5922307Z return mod(**inputs) 2025-08-14T21:58:11.5922551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5922629Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5922908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5922982Z outputs = layer_module( 2025-08-14T21:58:11.5923215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5923285Z outputs = self.rel_attn( 2025-08-14T21:58:11.5923517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.5923640Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.5923643Z 2025-08-14T21:58:11.5923746Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5923931Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5923995Z return mod(**inputs) 2025-08-14T21:58:11.5924240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5924318Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5924564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5924627Z outputs = layer_module( 2025-08-14T21:58:11.5924860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5924934Z outputs = self.rel_attn( 2025-08-14T21:58:11.5925168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5925246Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5925501Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.5925680Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.5925687Z 2025-08-14T21:58:11.5925807Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5926004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5926072Z return mod(**inputs) 2025-08-14T21:58:11.5926327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5926409Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5926668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5926740Z outputs = layer_module( 2025-08-14T21:58:11.5926987Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5927090Z outputs = self.rel_attn( 2025-08-14T21:58:11.5927344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.5927471Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.5927474Z 2025-08-14T21:58:11.5927578Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5927787Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5927860Z return mod(**inputs) 2025-08-14T21:58:11.5928101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5928182Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5928431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5928498Z outputs = layer_module( 2025-08-14T21:58:11.5928741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5928832Z outputs = self.rel_attn( 2025-08-14T21:58:11.5929073Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5929151Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5929410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.5929532Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.5929536Z 2025-08-14T21:58:11.5929631Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5929814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5929884Z return mod(**inputs) 2025-08-14T21:58:11.5930126Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5930205Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5930456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5930520Z outputs = layer_module( 2025-08-14T21:58:11.5930770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5930835Z outputs = self.rel_attn( 2025-08-14T21:58:11.5931076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5931168Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5931426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5931532Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5931544Z 2025-08-14T21:58:11.5931641Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5931828Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5931897Z return mod(**inputs) 2025-08-14T21:58:11.5932141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5932219Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5932463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5932527Z outputs = layer_module( 2025-08-14T21:58:11.5932780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5932863Z outputs = self.rel_attn( 2025-08-14T21:58:11.5933105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5933199Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5933472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5933574Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5933583Z 2025-08-14T21:58:11.5933678Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5933867Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5933935Z return mod(**inputs) 2025-08-14T21:58:11.5934175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5934251Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5934504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5934605Z outputs = layer_module( 2025-08-14T21:58:11.5934857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5935052Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5935301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5935381Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5935622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5935691Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5935940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.5936010Z output = self.layer_1(output) 2025-08-14T21:58:11.5936015Z 2025-08-14T21:58:11.5936120Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5936308Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5936369Z return mod(**inputs) 2025-08-14T21:58:11.5936621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5936698Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5936944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5937007Z outputs = layer_module( 2025-08-14T21:58:11.5937250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5937452Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5937818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5937906Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5938151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5938222Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5938472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.5938555Z output = self.activation_function(output) 2025-08-14T21:58:11.5938762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.5938878Z return self.act(input) 2025-08-14T21:58:11.5938882Z 2025-08-14T21:58:11.5938976Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5939168Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5939251Z return mod(**inputs) 2025-08-14T21:58:11.5939487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5939570Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5939809Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5939871Z outputs = layer_module( 2025-08-14T21:58:11.5940113Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5940306Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5940564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5940681Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5940919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5940994Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5941228Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.5941302Z output = self.layer_2(output) 2025-08-14T21:58:11.5941305Z 2025-08-14T21:58:11.5941399Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5941580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5941650Z return mod(**inputs) 2025-08-14T21:58:11.5941879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5941954Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5942194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5942257Z outputs = layer_module( 2025-08-14T21:58:11.5942494Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5942557Z outputs = self.rel_attn( 2025-08-14T21:58:11.5942789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.5942886Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.5942890Z 2025-08-14T21:58:11.5942983Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5943171Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5943233Z return mod(**inputs) 2025-08-14T21:58:11.5943467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5943550Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5943781Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5943844Z outputs = layer_module( 2025-08-14T21:58:11.5944079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5944142Z outputs = self.rel_attn( 2025-08-14T21:58:11.5944382Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.5944475Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.5944493Z 2025-08-14T21:58:11.5944588Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5944779Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5944857Z return mod(**inputs) 2025-08-14T21:58:11.5945101Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5945175Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5945408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5945476Z outputs = layer_module( 2025-08-14T21:58:11.5945708Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5945771Z outputs = self.rel_attn( 2025-08-14T21:58:11.5946012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5946081Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5946369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.5946491Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.5946495Z 2025-08-14T21:58:11.5946588Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5946775Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5946834Z return mod(**inputs) 2025-08-14T21:58:11.5947067Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5947150Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5947389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5947462Z outputs = layer_module( 2025-08-14T21:58:11.5947705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5947769Z outputs = self.rel_attn( 2025-08-14T21:58:11.5948005Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.5948128Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.5948131Z 2025-08-14T21:58:11.5948229Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5948409Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5948468Z return mod(**inputs) 2025-08-14T21:58:11.5948707Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5948784Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5949023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5949093Z outputs = layer_module( 2025-08-14T21:58:11.5949325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5949395Z outputs = self.rel_attn( 2025-08-14T21:58:11.5949625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5949692Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5949948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.5950063Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.5950082Z 2025-08-14T21:58:11.5950184Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5950367Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5950444Z return mod(**inputs) 2025-08-14T21:58:11.5950681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5950754Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5950984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5951054Z outputs = layer_module( 2025-08-14T21:58:11.5951282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5951352Z outputs = self.rel_attn( 2025-08-14T21:58:11.5951580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.5951674Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.5951677Z 2025-08-14T21:58:11.5951830Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5952017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5952085Z return mod(**inputs) 2025-08-14T21:58:11.5952317Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5952393Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5952629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5952691Z outputs = layer_module( 2025-08-14T21:58:11.5952918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5952991Z outputs = self.rel_attn( 2025-08-14T21:58:11.5953220Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5953296Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5953539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.5953653Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.5953656Z 2025-08-14T21:58:11.5953761Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5953941Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5954009Z return mod(**inputs) 2025-08-14T21:58:11.5954240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5954317Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5954553Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5954618Z outputs = layer_module( 2025-08-14T21:58:11.5954849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5954920Z outputs = self.rel_attn( 2025-08-14T21:58:11.5955149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5955237Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5955484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5955585Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5955589Z 2025-08-14T21:58:11.5955688Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5955885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5955947Z return mod(**inputs) 2025-08-14T21:58:11.5956201Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5956276Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5956518Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5956582Z outputs = layer_module( 2025-08-14T21:58:11.5956811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5956882Z outputs = self.rel_attn( 2025-08-14T21:58:11.5957109Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5957199Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5957488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5957591Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5957594Z 2025-08-14T21:58:11.5957696Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5957881Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5957943Z return mod(**inputs) 2025-08-14T21:58:11.5958183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5958259Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5958502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5958566Z outputs = layer_module( 2025-08-14T21:58:11.5958799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5959000Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5959241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5959320Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5959551Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5959617Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5959854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.5959921Z output = self.layer_1(output) 2025-08-14T21:58:11.5959926Z 2025-08-14T21:58:11.5960021Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5960214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5960277Z return mod(**inputs) 2025-08-14T21:58:11.5960522Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5960600Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5960833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5960904Z outputs = layer_module( 2025-08-14T21:58:11.5961142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5961343Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5961616Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5961692Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5961960Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5962041Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5962274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.5962366Z output = self.activation_function(output) 2025-08-14T21:58:11.5962566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.5962638Z return self.act(input) 2025-08-14T21:58:11.5962641Z 2025-08-14T21:58:11.5962737Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5962922Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5962992Z return mod(**inputs) 2025-08-14T21:58:11.5963258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5963343Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5963581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5963643Z outputs = layer_module( 2025-08-14T21:58:11.5963885Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5964074Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5964316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5964396Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5964641Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5964717Z output_x = self.ff(output_x) 2025-08-14T21:58:11.5964954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.5965021Z output = self.layer_2(output) 2025-08-14T21:58:11.5965025Z 2025-08-14T21:58:11.5965129Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5965317Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5965386Z return mod(**inputs) 2025-08-14T21:58:11.5965681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5965774Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5966036Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5966107Z outputs = layer_module( 2025-08-14T21:58:11.5966359Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5966438Z outputs = self.rel_attn( 2025-08-14T21:58:11.5966689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.5966797Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.5966801Z 2025-08-14T21:58:11.5966903Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5967104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5967174Z return mod(**inputs) 2025-08-14T21:58:11.5967413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5967521Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5967762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5967841Z outputs = layer_module( 2025-08-14T21:58:11.5968092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5968157Z outputs = self.rel_attn( 2025-08-14T21:58:11.5968397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.5968507Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.5968511Z 2025-08-14T21:58:11.5968611Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5968811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5968878Z return mod(**inputs) 2025-08-14T21:58:11.5969148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5969237Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5969488Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5969556Z outputs = layer_module( 2025-08-14T21:58:11.5969804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5969872Z outputs = self.rel_attn( 2025-08-14T21:58:11.5970122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5970194Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5970450Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.5970586Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.5970591Z 2025-08-14T21:58:11.5970690Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5970887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5970951Z return mod(**inputs) 2025-08-14T21:58:11.5971193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5971282Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5971521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5971587Z outputs = layer_module( 2025-08-14T21:58:11.5971832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5971900Z outputs = self.rel_attn( 2025-08-14T21:58:11.5972146Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.5972275Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.5972278Z 2025-08-14T21:58:11.5972377Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5972580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5972644Z return mod(**inputs) 2025-08-14T21:58:11.5972891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5972970Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5973209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5973300Z outputs = layer_module( 2025-08-14T21:58:11.5973547Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5973623Z outputs = self.rel_attn( 2025-08-14T21:58:11.5973865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5973935Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5974192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.5974311Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.5974314Z 2025-08-14T21:58:11.5974409Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5974601Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5974665Z return mod(**inputs) 2025-08-14T21:58:11.5974935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5975013Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5975251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5975321Z outputs = layer_module( 2025-08-14T21:58:11.5975557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5975621Z outputs = self.rel_attn( 2025-08-14T21:58:11.5975864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.5975956Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.5975960Z 2025-08-14T21:58:11.5976062Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5976249Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5976313Z return mod(**inputs) 2025-08-14T21:58:11.5976557Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5976634Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5976882Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5976945Z outputs = layer_module( 2025-08-14T21:58:11.5977183Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5977255Z outputs = self.rel_attn( 2025-08-14T21:58:11.5977493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.5977562Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.5977827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.5977944Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.5977947Z 2025-08-14T21:58:11.5993440Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5993756Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5993827Z return mod(**inputs) 2025-08-14T21:58:11.5994100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5994181Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5994424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5994589Z outputs = layer_module( 2025-08-14T21:58:11.5994838Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5994963Z outputs = self.rel_attn( 2025-08-14T21:58:11.5995209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5995299Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5995569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5995681Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5995687Z 2025-08-14T21:58:11.5995794Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5995995Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5996063Z return mod(**inputs) 2025-08-14T21:58:11.5996314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5996458Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5996697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5996766Z outputs = layer_module( 2025-08-14T21:58:11.5996999Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.5997071Z outputs = self.rel_attn( 2025-08-14T21:58:11.5997300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.5997383Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.5997640Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.5997744Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.5997747Z 2025-08-14T21:58:11.5997847Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.5998045Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.5998107Z return mod(**inputs) 2025-08-14T21:58:11.5998349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.5998429Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.5998661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.5998733Z outputs = layer_module( 2025-08-14T21:58:11.5998958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.5999159Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.5999401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.5999478Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.5999719Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.5999790Z output_x = self.ff(output_x) 2025-08-14T21:58:11.6000018Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.6000100Z output = self.layer_1(output) 2025-08-14T21:58:11.6000104Z 2025-08-14T21:58:11.6000201Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6000394Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6000473Z return mod(**inputs) 2025-08-14T21:58:11.6000712Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6000814Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6001048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6001119Z outputs = layer_module( 2025-08-14T21:58:11.6001346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.6001543Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.6001791Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.6001864Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.6002106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.6002217Z output_x = self.ff(output_x) 2025-08-14T21:58:11.6002458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.6002551Z output = self.activation_function(output) 2025-08-14T21:58:11.6002751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.6002817Z return self.act(input) 2025-08-14T21:58:11.6002821Z 2025-08-14T21:58:11.6002927Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6003115Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6003186Z return mod(**inputs) 2025-08-14T21:58:11.6003428Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6003511Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6003766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6003832Z outputs = layer_module( 2025-08-14T21:58:11.6004074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.6004292Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.6004543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.6004620Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.6004861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.6004932Z output_x = self.ff(output_x) 2025-08-14T21:58:11.6005181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.6005255Z output = self.layer_2(output) 2025-08-14T21:58:11.6005259Z 2025-08-14T21:58:11.6005365Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6005556Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6005691Z return mod(**inputs) 2025-08-14T21:58:11.6005956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6006038Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6006285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6006375Z outputs = layer_module( 2025-08-14T21:58:11.6006622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.6006701Z outputs = self.rel_attn( 2025-08-14T21:58:11.6006962Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.6007062Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.6007066Z 2025-08-14T21:58:11.6007172Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6007368Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6007441Z return mod(**inputs) 2025-08-14T21:58:11.6007688Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6007770Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6008027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6008107Z outputs = layer_module( 2025-08-14T21:58:11.6008370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.6008449Z outputs = self.rel_attn( 2025-08-14T21:58:11.6008694Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.6008804Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.6008808Z 2025-08-14T21:58:11.6008907Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6009103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6009172Z return mod(**inputs) 2025-08-14T21:58:11.6009419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6009501Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6009760Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6009827Z outputs = layer_module( 2025-08-14T21:58:11.6010081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.6010148Z outputs = self.rel_attn( 2025-08-14T21:58:11.6010392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.6010473Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.6010736Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.6010878Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.6010883Z 2025-08-14T21:58:11.6010982Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6011179Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6011249Z return mod(**inputs) 2025-08-14T21:58:11.6011500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6011580Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6011831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6011895Z outputs = layer_module( 2025-08-14T21:58:11.6012145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.6012211Z outputs = self.rel_attn( 2025-08-14T21:58:11.6012474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.6012615Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.6012635Z 2025-08-14T21:58:11.6012737Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6012935Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6012996Z return mod(**inputs) 2025-08-14T21:58:11.6013239Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6013326Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6013572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6013637Z outputs = layer_module( 2025-08-14T21:58:11.6013891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.6013959Z outputs = self.rel_attn( 2025-08-14T21:58:11.6014244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.6014317Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.6014579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.6014709Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.6014713Z 2025-08-14T21:58:11.6014811Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6015010Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6015073Z return mod(**inputs) 2025-08-14T21:58:11.6015318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6015407Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6015654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6015720Z outputs = layer_module( 2025-08-14T21:58:11.6015971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.6016036Z outputs = self.rel_attn( 2025-08-14T21:58:11.6016286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.6016386Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.6016389Z 2025-08-14T21:58:11.6016487Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6016692Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6016754Z return mod(**inputs) 2025-08-14T21:58:11.6016996Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6017074Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6017305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6017375Z outputs = layer_module( 2025-08-14T21:58:11.6017604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.6017666Z outputs = self.rel_attn( 2025-08-14T21:58:11.6017904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.6017971Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.6018223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.6018364Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.6018370Z 2025-08-14T21:58:11.6018486Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6018678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6018738Z return mod(**inputs) 2025-08-14T21:58:11.6018968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6019052Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6019283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6019351Z outputs = layer_module( 2025-08-14T21:58:11.6019578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.6019643Z outputs = self.rel_attn( 2025-08-14T21:58:11.6019907Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.6019991Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.6020248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.6020351Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.6020355Z 2025-08-14T21:58:11.6020447Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6020635Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6020695Z return mod(**inputs) 2025-08-14T21:58:11.6020925Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6021008Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6021241Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6021311Z outputs = layer_module( 2025-08-14T21:58:11.6021536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.6021599Z outputs = self.rel_attn( 2025-08-14T21:58:11.6021833Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.6021913Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.6022165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.6022263Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.6022268Z 2025-08-14T21:58:11.6022361Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6022550Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6022613Z return mod(**inputs) 2025-08-14T21:58:11.6022843Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6022927Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6023157Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6023224Z outputs = layer_module( 2025-08-14T21:58:11.6023452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.6023640Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.6023904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.6023977Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.6024234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.6024301Z output_x = self.ff(output_x) 2025-08-14T21:58:11.6024534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.6024611Z output = self.layer_1(output) 2025-08-14T21:58:11.6024614Z 2025-08-14T21:58:11.6024705Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6024888Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6024955Z return mod(**inputs) 2025-08-14T21:58:11.6025185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6025269Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6025527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6025590Z outputs = layer_module( 2025-08-14T21:58:11.6025830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.6026018Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.6026266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.6026336Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.6026566Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.6026640Z output_x = self.ff(output_x) 2025-08-14T21:58:11.6026875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.6026958Z output = self.activation_function(output) 2025-08-14T21:58:11.6027164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.6027229Z return self.act(input) 2025-08-14T21:58:11.6027232Z 2025-08-14T21:58:11.6027335Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6027518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6027576Z return mod(**inputs) 2025-08-14T21:58:11.6027815Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6027891Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6028133Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6028197Z outputs = layer_module( 2025-08-14T21:58:11.6028432Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.6028629Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.6028871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.6028940Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.6029180Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.6029248Z output_x = self.ff(output_x) 2025-08-14T21:58:11.6029486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.6029573Z output = self.layer_2(output) 2025-08-14T21:58:11.6029577Z 2025-08-14T21:58:11.6029674Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6029884Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6029945Z return mod(**inputs) 2025-08-14T21:58:11.6030184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6030260Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6030489Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6030558Z outputs = layer_module( 2025-08-14T21:58:11.6030785Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.6030851Z outputs = self.rel_attn( 2025-08-14T21:58:11.6031124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 416, in forward 2025-08-14T21:58:11.6031218Z q_head_h = torch.einsum("ibh,hnd->ibnd", h, self.q) 2025-08-14T21:58:11.6031222Z 2025-08-14T21:58:11.6031324Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6031508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6031571Z return mod(**inputs) 2025-08-14T21:58:11.6031814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6031891Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6032128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6032199Z outputs = layer_module( 2025-08-14T21:58:11.6032441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.6032518Z outputs = self.rel_attn( 2025-08-14T21:58:11.6032762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 417, in forward 2025-08-14T21:58:11.6032867Z k_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.k) 2025-08-14T21:58:11.6032871Z 2025-08-14T21:58:11.6032972Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6033155Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6033222Z return mod(**inputs) 2025-08-14T21:58:11.6033455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6033531Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6033769Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6033829Z outputs = layer_module( 2025-08-14T21:58:11.6034058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.6034130Z outputs = self.rel_attn( 2025-08-14T21:58:11.6034360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.6034436Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.6034682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 263, in rel_attn_core 2025-08-14T21:58:11.6034802Z ac = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_w_bias, k_head_h) 2025-08-14T21:58:11.6034805Z 2025-08-14T21:58:11.6034904Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6035104Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6035172Z return mod(**inputs) 2025-08-14T21:58:11.6035408Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6035500Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6035737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6035798Z outputs = layer_module( 2025-08-14T21:58:11.6036029Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.6036099Z outputs = self.rel_attn( 2025-08-14T21:58:11.6036330Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 422, in forward 2025-08-14T21:58:11.6036456Z k_head_r = torch.einsum("ibh,hnd->ibnd", r.type(self.r.dtype), self.r) 2025-08-14T21:58:11.6036461Z 2025-08-14T21:58:11.6036553Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6036769Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6036841Z return mod(**inputs) 2025-08-14T21:58:11.6037078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6037163Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6037395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6037458Z outputs = layer_module( 2025-08-14T21:58:11.6037915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.6037983Z outputs = self.rel_attn( 2025-08-14T21:58:11.6038215Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.6038295Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.6038548Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 266, in rel_attn_core 2025-08-14T21:58:11.6038675Z bd = torch.einsum("ibnd,jbnd->bnij", q_head + self.r_r_bias, k_head_r) 2025-08-14T21:58:11.6038679Z 2025-08-14T21:58:11.6038773Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6038967Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6039031Z return mod(**inputs) 2025-08-14T21:58:11.6039267Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6039356Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6039591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6039667Z outputs = layer_module( 2025-08-14T21:58:11.6039902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.6039970Z outputs = self.rel_attn( 2025-08-14T21:58:11.6040213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 418, in forward 2025-08-14T21:58:11.6040308Z v_head_h = torch.einsum("ibh,hnd->ibnd", cat, self.v) 2025-08-14T21:58:11.6040311Z 2025-08-14T21:58:11.6040407Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6040600Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6040661Z return mod(**inputs) 2025-08-14T21:58:11.6040901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6041044Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6041291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6041389Z outputs = layer_module( 2025-08-14T21:58:11.6041647Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.6041722Z outputs = self.rel_attn( 2025-08-14T21:58:11.6041980Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 425, in forward 2025-08-14T21:58:11.6042052Z attn_vec = self.rel_attn_core( 2025-08-14T21:58:11.6042334Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 294, in rel_attn_core 2025-08-14T21:58:11.6042458Z attn_vec = torch.einsum("bnij,jbnd->ibnd", attn_prob, v_head_h) 2025-08-14T21:58:11.6042464Z 2025-08-14T21:58:11.6042566Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6042835Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6042900Z return mod(**inputs) 2025-08-14T21:58:11.6043148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6043226Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6043461Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6043532Z outputs = layer_module( 2025-08-14T21:58:11.6043765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.6043827Z outputs = self.rel_attn( 2025-08-14T21:58:11.6044066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.6044153Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.6044414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.6044520Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.6044524Z 2025-08-14T21:58:11.6044625Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6044831Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6044895Z return mod(**inputs) 2025-08-14T21:58:11.6045148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6045229Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6045477Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6045553Z outputs = layer_module( 2025-08-14T21:58:11.6045866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 494, in forward 2025-08-14T21:58:11.6045942Z outputs = self.rel_attn( 2025-08-14T21:58:11.6046211Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 440, in forward 2025-08-14T21:58:11.6046302Z output_h = self.post_attention(h, attn_vec) 2025-08-14T21:58:11.6046601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 304, in post_attention 2025-08-14T21:58:11.6046708Z attn_out = torch.einsum("ibnd,hnd->ibh", attn_vec, self.o) 2025-08-14T21:58:11.6046711Z 2025-08-14T21:58:11.6046812Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6047017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6047100Z return mod(**inputs) 2025-08-14T21:58:11.6047362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6047463Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6047715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6047788Z outputs = layer_module( 2025-08-14T21:58:11.6048039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.6048248Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.6048521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.6048600Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.6048864Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.6048967Z output_x = self.ff(output_x) 2025-08-14T21:58:11.6049217Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 463, in forward 2025-08-14T21:58:11.6049297Z output = self.layer_1(output) 2025-08-14T21:58:11.6049301Z 2025-08-14T21:58:11.6049402Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6049604Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6049669Z return mod(**inputs) 2025-08-14T21:58:11.6049919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6050009Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6050258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6050337Z outputs = layer_module( 2025-08-14T21:58:11.6050578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.6050774Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.6051027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.6051099Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.6051337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.6051413Z output_x = self.ff(output_x) 2025-08-14T21:58:11.6051648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 464, in forward 2025-08-14T21:58:11.6051739Z output = self.activation_function(output) 2025-08-14T21:58:11.6051943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:11.6052011Z return self.act(input) 2025-08-14T21:58:11.6052015Z 2025-08-14T21:58:11.6052119Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6052305Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6052365Z return mod(**inputs) 2025-08-14T21:58:11.6052607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1607, in forward 2025-08-14T21:58:11.6052685Z transformer_outputs = self.transformer( 2025-08-14T21:58:11.6052926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1368, in forward 2025-08-14T21:58:11.6052990Z outputs = layer_module( 2025-08-14T21:58:11.6053240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 512, in forward 2025-08-14T21:58:11.6053444Z output_h = apply_chunking_to_forward(self.ff_chunk, self.chunk_size_feed_forward, self.seq_len_dim, output_h) 2025-08-14T21:58:11.6053711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:11.6053793Z return forward_fn(*input_tensors) 2025-08-14T21:58:11.6054037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 518, in ff_chunk 2025-08-14T21:58:11.6054106Z output_x = self.ff(output_x) 2025-08-14T21:58:11.6054355Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 466, in forward 2025-08-14T21:58:11.6054424Z output = self.layer_2(output) 2025-08-14T21:58:11.6054427Z 2025-08-14T21:58:11.6054528Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6054729Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6054845Z return mod(**inputs) 2025-08-14T21:58:11.6055105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1624, in forward 2025-08-14T21:58:11.6055202Z logits = self.lm_loss(transformer_outputs[0]) 2025-08-14T21:58:11.6055206Z 2025-08-14T21:58:11.6055306Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:11.6055508Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:11.6055571Z return mod(**inputs) 2025-08-14T21:58:11.6055829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/xlnet/modeling_xlnet.py", line 1630, in forward 2025-08-14T21:58:11.6055967Z loss = loss_fct(logits.view(-1, logits.size(-1)), labels.view(-1)) 2025-08-14T21:58:11.6055972Z 2025-08-14T21:58:23.8188743Z Compilation time (from dynamo_timed): 30.057544365 2025-08-14T21:58:23.8231287Z pass 2025-08-14T21:58:23.8231749Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:58:23.8232603Z TIMING: _recursive_pre_grad_passes:0.01251 _recursive_joint_graph_passes:1.28172 _recursive_post_grad_passes:0.22234 async_compile.wait:0.79203 code_gen:10.24988 inductor_compile:14.55618 backend_compile:23.80993 gc:0.00157 entire_frame_compile:30.05754 total_wall_time:30.05754 2025-08-14T21:58:23.8233605Z STATS: call_* op count: 818 | FakeTensorMode.__torch_dispatch__:56665 | FakeTensor.__torch_dispatch__:16773 | ProxyTorchDispatchMode.__torch_dispatch__:18623 2025-08-14T21:58:23.8234130Z Dynamo produced 1 graphs covering 818 ops with 0 graph breaks (0 unique) 2025-08-14T21:58:29.2606813Z /opt/conda/envs/py_3.9/lib/python3.9/site-packages/llvmlite/binding/ffi.py:175: UserWarning: pkg_resources is deprecated as an API. See https://setuptools.pypa.io/en/latest/pkg_resources.html. The pkg_resources package is slated for removal as early as 2025-11-30. Refrain from using this package or pin to Setuptools<81. 2025-08-14T21:58:29.2607888Z from pkg_resources import resource_filename 2025-08-14T21:58:29.8259130Z 2025-08-14T21:58:31.2181332Z loading model: 0it [00:00, ?it/s] 2025-08-14T21:58:31.2185478Z loading model: 0it [00:01, ?it/s] 2025-08-14T21:58:31.2201023Z cpu eval YituTechConvBert 2025-08-14T21:58:32.1050893Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:58:32.3763451Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:58:32.6544463Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:58:44.3931403Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.3935592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.3937062Z return mod(**inputs) 2025-08-14T21:58:44.3938192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.3939008Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.3939467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.3939868Z hidden_states = self.encoder( 2025-08-14T21:58:44.3940263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.3940636Z layer_outputs = layer_module( 2025-08-14T21:58:44.3940957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.3941293Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.3941676Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.3942152Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.3942531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.3942909Z self_outputs = self.self( 2025-08-14T21:58:44.3943272Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T21:58:44.3943666Z mixed_query_layer = self.query(hidden_states) 2025-08-14T21:58:44.3943808Z 2025-08-14T21:58:44.3943910Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.3944255Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.3944565Z return mod(**inputs) 2025-08-14T21:58:44.3944918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.3945304Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.3945687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.3946061Z hidden_states = self.encoder( 2025-08-14T21:58:44.3946418Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.3946798Z layer_outputs = layer_module( 2025-08-14T21:58:44.3947132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.3947486Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.3947854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.3948235Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.3948620Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.3948996Z self_outputs = self.self( 2025-08-14T21:58:44.3949365Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T21:58:44.3949761Z mixed_key_layer = self.key(hidden_states) 2025-08-14T21:58:44.3949897Z 2025-08-14T21:58:44.3950012Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.3950373Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.3950688Z return mod(**inputs) 2025-08-14T21:58:44.3951055Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.3951495Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.3951870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.3952268Z hidden_states = self.encoder( 2025-08-14T21:58:44.3952639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.3953007Z layer_outputs = layer_module( 2025-08-14T21:58:44.3953342Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.3953685Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.3954065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.3954448Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.3954835Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.3955212Z self_outputs = self.self( 2025-08-14T21:58:44.3955623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T21:58:44.3956014Z mixed_value_layer = self.value(hidden_states) 2025-08-14T21:58:44.3956160Z 2025-08-14T21:58:44.3956239Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.3956446Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.3956665Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.3957004Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.3957311Z return mod(**inputs) 2025-08-14T21:58:44.3957666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.3958039Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.3958422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.3958799Z hidden_states = self.encoder( 2025-08-14T21:58:44.3959162Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.3959540Z layer_outputs = layer_module( 2025-08-14T21:58:44.3959858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.3960200Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.3960565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.3960947Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.3961327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.3961696Z self_outputs = self.self( 2025-08-14T21:58:44.3962070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T21:58:44.3962481Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T21:58:44.3962630Z 2025-08-14T21:58:44.3962714Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.3962933Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.3963273Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.3963583Z return mod(**inputs) 2025-08-14T21:58:44.3963944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.3964334Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.3964721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.3965128Z hidden_states = self.encoder( 2025-08-14T21:58:44.3965502Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.3966051Z layer_outputs = layer_module( 2025-08-14T21:58:44.3966393Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.3966754Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.3967143Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.3967545Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.3967953Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.3968331Z self_outputs = self.self( 2025-08-14T21:58:44.3968689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.3969190Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.3969656Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T21:58:44.3970037Z x = self.depthwise(hidden_states) 2025-08-14T21:58:44.3970169Z 2025-08-14T21:58:44.3970270Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.3970609Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.3970914Z return mod(**inputs) 2025-08-14T21:58:44.3971263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.3971651Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.3972032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.3972404Z hidden_states = self.encoder( 2025-08-14T21:58:44.3972775Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.3973150Z layer_outputs = layer_module( 2025-08-14T21:58:44.3973470Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.3973802Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.3974185Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.3974557Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.3974929Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.3975289Z self_outputs = self.self( 2025-08-14T21:58:44.3975645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.3976088Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.3976531Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T21:58:44.3976908Z x = self.pointwise(x) 2025-08-14T21:58:44.3977018Z 2025-08-14T21:58:44.3977115Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.3977463Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.3977765Z return mod(**inputs) 2025-08-14T21:58:44.3978128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.3978552Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.3978944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.3979334Z hidden_states = self.encoder( 2025-08-14T21:58:44.3979705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.3980084Z layer_outputs = layer_module( 2025-08-14T21:58:44.3980403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.3980750Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.3981132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.3981521Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.3981899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.3982300Z self_outputs = self.self( 2025-08-14T21:58:44.3982664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T21:58:44.3983118Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T21:58:44.3983311Z 2025-08-14T21:58:44.3983410Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.3983751Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.3984061Z return mod(**inputs) 2025-08-14T21:58:44.3984426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.3984803Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.3985177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.3985544Z hidden_states = self.encoder( 2025-08-14T21:58:44.3985895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.3986262Z layer_outputs = layer_module( 2025-08-14T21:58:44.3986576Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.3986908Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.3987287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.3987675Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.3988057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.3988461Z self_outputs = self.self( 2025-08-14T21:58:44.3988831Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T21:58:44.3989277Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T21:58:44.3989446Z 2025-08-14T21:58:44.3989557Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.3989914Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.3990240Z return mod(**inputs) 2025-08-14T21:58:44.3990614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.3991011Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.3991460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.3991892Z hidden_states = self.encoder( 2025-08-14T21:58:44.3992283Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.3992719Z layer_outputs = layer_module( 2025-08-14T21:58:44.3993064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.3993427Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.3993828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.3994232Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.3994642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.3995037Z self_outputs = self.self( 2025-08-14T21:58:44.3995419Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T21:58:44.3995915Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T21:58:44.3996103Z 2025-08-14T21:58:44.3996185Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.3996399Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.3996629Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.3996988Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.3997317Z return mod(**inputs) 2025-08-14T21:58:44.3997683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.3998090Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.3998495Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.3998895Z hidden_states = self.encoder( 2025-08-14T21:58:44.3999277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.3999677Z layer_outputs = layer_module( 2025-08-14T21:58:44.4000015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4000373Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4000768Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4001179Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4001588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4001984Z self_outputs = self.self( 2025-08-14T21:58:44.4002373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T21:58:44.4002819Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T21:58:44.4002986Z 2025-08-14T21:58:44.4003096Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4003444Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4003766Z return mod(**inputs) 2025-08-14T21:58:44.4004144Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4004744Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4005177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4005605Z hidden_states = self.encoder( 2025-08-14T21:58:44.4006082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4006528Z layer_outputs = layer_module( 2025-08-14T21:58:44.4006890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4007289Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4007683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4008073Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4008471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T21:58:44.4008920Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:58:44.4009363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T21:58:44.4009758Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4009899Z 2025-08-14T21:58:44.4010002Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4010384Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4010696Z return mod(**inputs) 2025-08-14T21:58:44.4011060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4011453Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4011858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4012668Z hidden_states = self.encoder( 2025-08-14T21:58:44.4013050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4013433Z layer_outputs = layer_module( 2025-08-14T21:58:44.4013772Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4014140Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4014525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4014931Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4015324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4015711Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4016122Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4016591Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4017033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T21:58:44.4017438Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4017575Z 2025-08-14T21:58:44.4017679Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4018022Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4018338Z return mod(**inputs) 2025-08-14T21:58:44.4018695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4019090Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4019491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4019867Z hidden_states = self.encoder( 2025-08-14T21:58:44.4020233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4020640Z layer_outputs = layer_module( 2025-08-14T21:58:44.4020972Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4021335Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4021743Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4022146Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4022541Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4022910Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4023319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4023775Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4024233Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T21:58:44.4024643Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:58:44.4025000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:44.4025326Z return self.act(input) 2025-08-14T21:58:44.4025431Z 2025-08-14T21:58:44.4025537Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4025866Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4026168Z return mod(**inputs) 2025-08-14T21:58:44.4026523Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4026902Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4027292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4027673Z hidden_states = self.encoder( 2025-08-14T21:58:44.4028041Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4028411Z layer_outputs = layer_module( 2025-08-14T21:58:44.4028734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4029073Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4029446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4029837Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4030214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4030588Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4030985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T21:58:44.4031446Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:58:44.4031879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T21:58:44.4032273Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4032413Z 2025-08-14T21:58:44.4032513Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4032853Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4033154Z return mod(**inputs) 2025-08-14T21:58:44.4033500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4033908Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4034292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4034690Z hidden_states = self.encoder( 2025-08-14T21:58:44.4035048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4035421Z layer_outputs = layer_module( 2025-08-14T21:58:44.4035742Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4036074Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4036441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4036828Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4037206Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4037782Z self_outputs = self.self( 2025-08-14T21:58:44.4038176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T21:58:44.4038593Z mixed_query_layer = self.query(hidden_states) 2025-08-14T21:58:44.4038739Z 2025-08-14T21:58:44.4038849Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4039191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4039518Z return mod(**inputs) 2025-08-14T21:58:44.4039886Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4040279Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4040677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4041071Z hidden_states = self.encoder( 2025-08-14T21:58:44.4041453Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4041832Z layer_outputs = layer_module( 2025-08-14T21:58:44.4042170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4042518Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4042917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4043319Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4043723Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4044131Z self_outputs = self.self( 2025-08-14T21:58:44.4044500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T21:58:44.4044905Z mixed_key_layer = self.key(hidden_states) 2025-08-14T21:58:44.4045043Z 2025-08-14T21:58:44.4045142Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4045488Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4045878Z return mod(**inputs) 2025-08-14T21:58:44.4046260Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4046678Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4047078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4047461Z hidden_states = self.encoder( 2025-08-14T21:58:44.4047918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4048312Z layer_outputs = layer_module( 2025-08-14T21:58:44.4048671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4049025Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4049485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4049889Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4050274Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4050661Z self_outputs = self.self( 2025-08-14T21:58:44.4051033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T21:58:44.4051438Z mixed_value_layer = self.value(hidden_states) 2025-08-14T21:58:44.4051588Z 2025-08-14T21:58:44.4051721Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4051932Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4052161Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4052505Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4052820Z return mod(**inputs) 2025-08-14T21:58:44.4053189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4053582Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4054039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4054433Z hidden_states = self.encoder( 2025-08-14T21:58:44.4054821Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4055203Z layer_outputs = layer_module( 2025-08-14T21:58:44.4055540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4055885Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4056341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4056736Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4057136Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4057544Z self_outputs = self.self( 2025-08-14T21:58:44.4057905Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T21:58:44.4058315Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T21:58:44.4058467Z 2025-08-14T21:58:44.4058546Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4058771Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4059107Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4059417Z return mod(**inputs) 2025-08-14T21:58:44.4059771Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4060156Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4060542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4060922Z hidden_states = self.encoder( 2025-08-14T21:58:44.4061291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4061688Z layer_outputs = layer_module( 2025-08-14T21:58:44.4062017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4062374Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4062752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4063132Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4063517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4063896Z self_outputs = self.self( 2025-08-14T21:58:44.4064252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.4064711Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.4065204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T21:58:44.4065594Z x = self.depthwise(hidden_states) 2025-08-14T21:58:44.4065722Z 2025-08-14T21:58:44.4065820Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4066159Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4066520Z return mod(**inputs) 2025-08-14T21:58:44.4066877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4067255Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4067636Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4068021Z hidden_states = self.encoder( 2025-08-14T21:58:44.4068405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4068802Z layer_outputs = layer_module( 2025-08-14T21:58:44.4069134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4069482Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4069868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4070249Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4070627Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4070996Z self_outputs = self.self( 2025-08-14T21:58:44.4071362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.4071824Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.4072287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T21:58:44.4072653Z x = self.pointwise(x) 2025-08-14T21:58:44.4072767Z 2025-08-14T21:58:44.4072863Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4073201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4073507Z return mod(**inputs) 2025-08-14T21:58:44.4073857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4074245Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4074632Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4075040Z hidden_states = self.encoder( 2025-08-14T21:58:44.4075420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4075831Z layer_outputs = layer_module( 2025-08-14T21:58:44.4076159Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4076489Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4076874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4077259Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4077654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4078037Z self_outputs = self.self( 2025-08-14T21:58:44.4078411Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T21:58:44.4078912Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T21:58:44.4079129Z 2025-08-14T21:58:44.4079230Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4079580Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4079894Z return mod(**inputs) 2025-08-14T21:58:44.4080259Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4080650Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4081045Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4081434Z hidden_states = self.encoder( 2025-08-14T21:58:44.4081818Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4082196Z layer_outputs = layer_module( 2025-08-14T21:58:44.4082528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4082884Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4083264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4083661Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4084054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4084439Z self_outputs = self.self( 2025-08-14T21:58:44.4084801Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T21:58:44.4085232Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T21:58:44.4085397Z 2025-08-14T21:58:44.4085508Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4085959Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4086272Z return mod(**inputs) 2025-08-14T21:58:44.4086642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4087043Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4087425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4087812Z hidden_states = self.encoder( 2025-08-14T21:58:44.4088187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4088602Z layer_outputs = layer_module( 2025-08-14T21:58:44.4088932Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4089284Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4089695Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4090089Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4090481Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4090864Z self_outputs = self.self( 2025-08-14T21:58:44.4091234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T21:58:44.4091664Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T21:58:44.4091854Z 2025-08-14T21:58:44.4091933Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4092136Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4092394Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4092737Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4093046Z return mod(**inputs) 2025-08-14T21:58:44.4093403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4093790Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4094170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4094553Z hidden_states = self.encoder( 2025-08-14T21:58:44.4094928Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4095302Z layer_outputs = layer_module( 2025-08-14T21:58:44.4095629Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4095976Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4096356Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4096745Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4097132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4097510Z self_outputs = self.self( 2025-08-14T21:58:44.4097865Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T21:58:44.4098285Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T21:58:44.4098438Z 2025-08-14T21:58:44.4098546Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4098887Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4099195Z return mod(**inputs) 2025-08-14T21:58:44.4099552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4099945Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4100323Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4100701Z hidden_states = self.encoder( 2025-08-14T21:58:44.4101070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4101444Z layer_outputs = layer_module( 2025-08-14T21:58:44.4101766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4102139Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4102527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4102968Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4103349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T21:58:44.4103776Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:58:44.4104203Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T21:58:44.4104574Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4104707Z 2025-08-14T21:58:44.4104802Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4105130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4105430Z return mod(**inputs) 2025-08-14T21:58:44.4105804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4106191Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4106565Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4106930Z hidden_states = self.encoder( 2025-08-14T21:58:44.4107292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4107664Z layer_outputs = layer_module( 2025-08-14T21:58:44.4107979Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4108298Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4108672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4109069Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4109441Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4109805Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4110209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4110663Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4111074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T21:58:44.4111465Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4111603Z 2025-08-14T21:58:44.4111700Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4112044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4112344Z return mod(**inputs) 2025-08-14T21:58:44.4112709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4113087Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4113452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4113822Z hidden_states = self.encoder( 2025-08-14T21:58:44.4114177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4114541Z layer_outputs = layer_module( 2025-08-14T21:58:44.4114847Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4115210Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4115601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4116014Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4116392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4116759Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4117166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4117615Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4118037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T21:58:44.4118452Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:58:44.4118812Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:44.4119163Z return self.act(input) 2025-08-14T21:58:44.4119280Z 2025-08-14T21:58:44.4119377Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4119716Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4120017Z return mod(**inputs) 2025-08-14T21:58:44.4120368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4120755Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4121134Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4121505Z hidden_states = self.encoder( 2025-08-14T21:58:44.4121874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4122257Z layer_outputs = layer_module( 2025-08-14T21:58:44.4122591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4122936Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4123327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4123727Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4124115Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4124490Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4124911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T21:58:44.4125386Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:58:44.4125895Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T21:58:44.4126310Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4126455Z 2025-08-14T21:58:44.4126555Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4126908Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4127224Z return mod(**inputs) 2025-08-14T21:58:44.4127618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4128018Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4128407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4128836Z hidden_states = self.encoder( 2025-08-14T21:58:44.4129225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4129631Z layer_outputs = layer_module( 2025-08-14T21:58:44.4129957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4130318Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4130697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4131086Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4131458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4131831Z self_outputs = self.self( 2025-08-14T21:58:44.4132194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T21:58:44.4132600Z mixed_query_layer = self.query(hidden_states) 2025-08-14T21:58:44.4132760Z 2025-08-14T21:58:44.4132862Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4133199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4133508Z return mod(**inputs) 2025-08-14T21:58:44.4133857Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4134244Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4134631Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4135009Z hidden_states = self.encoder( 2025-08-14T21:58:44.4135369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4135748Z layer_outputs = layer_module( 2025-08-14T21:58:44.4136074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4136408Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4136788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4137174Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4137555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4138093Z self_outputs = self.self( 2025-08-14T21:58:44.4138462Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T21:58:44.4138852Z mixed_key_layer = self.key(hidden_states) 2025-08-14T21:58:44.4138984Z 2025-08-14T21:58:44.4139090Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4139435Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4139733Z return mod(**inputs) 2025-08-14T21:58:44.4140076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4140446Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4140817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4141184Z hidden_states = self.encoder( 2025-08-14T21:58:44.4141545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4141907Z layer_outputs = layer_module( 2025-08-14T21:58:44.4142273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4142607Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4143011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4143387Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4143761Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4144127Z self_outputs = self.self( 2025-08-14T21:58:44.4144478Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T21:58:44.4144864Z mixed_value_layer = self.value(hidden_states) 2025-08-14T21:58:44.4145000Z 2025-08-14T21:58:44.4145085Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4145286Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4145498Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4145909Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4146216Z return mod(**inputs) 2025-08-14T21:58:44.4146554Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4146933Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4147320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4147743Z hidden_states = self.encoder( 2025-08-14T21:58:44.4148108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4148493Z layer_outputs = layer_module( 2025-08-14T21:58:44.4148822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4149157Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4149550Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4149924Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4150291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4150651Z self_outputs = self.self( 2025-08-14T21:58:44.4151007Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T21:58:44.4151403Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T21:58:44.4151544Z 2025-08-14T21:58:44.4151617Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4151843Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4152184Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4152500Z return mod(**inputs) 2025-08-14T21:58:44.4152850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4153230Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4153615Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4153992Z hidden_states = self.encoder( 2025-08-14T21:58:44.4154352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4154727Z layer_outputs = layer_module( 2025-08-14T21:58:44.4155050Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4155403Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4155783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4156201Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4156590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4156968Z self_outputs = self.self( 2025-08-14T21:58:44.4157339Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.4157812Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.4158285Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T21:58:44.4158690Z x = self.depthwise(hidden_states) 2025-08-14T21:58:44.4158827Z 2025-08-14T21:58:44.4158931Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4159322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4159635Z return mod(**inputs) 2025-08-14T21:58:44.4160000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4160393Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4160790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4161178Z hidden_states = self.encoder( 2025-08-14T21:58:44.4161563Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4161936Z layer_outputs = layer_module( 2025-08-14T21:58:44.4162255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4162602Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4162995Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4163395Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4163782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4164167Z self_outputs = self.self( 2025-08-14T21:58:44.4164539Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.4165001Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.4165465Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T21:58:44.4165963Z x = self.pointwise(x) 2025-08-14T21:58:44.4166085Z 2025-08-14T21:58:44.4166206Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4166583Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4166919Z return mod(**inputs) 2025-08-14T21:58:44.4167287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4167692Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4168098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4168484Z hidden_states = self.encoder( 2025-08-14T21:58:44.4168862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4169273Z layer_outputs = layer_module( 2025-08-14T21:58:44.4169593Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4169954Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4170331Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4170710Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4171094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4171466Z self_outputs = self.self( 2025-08-14T21:58:44.4171827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T21:58:44.4172263Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T21:58:44.4172468Z 2025-08-14T21:58:44.4172565Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4172939Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4173248Z return mod(**inputs) 2025-08-14T21:58:44.4173601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4173985Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4174361Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4174729Z hidden_states = self.encoder( 2025-08-14T21:58:44.4175092Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4175462Z layer_outputs = layer_module( 2025-08-14T21:58:44.4175784Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4176116Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4176496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4176881Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4177250Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4177630Z self_outputs = self.self( 2025-08-14T21:58:44.4177981Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T21:58:44.4178386Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T21:58:44.4178540Z 2025-08-14T21:58:44.4178637Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4178970Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4179271Z return mod(**inputs) 2025-08-14T21:58:44.4179630Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4180014Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4180410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4180807Z hidden_states = self.encoder( 2025-08-14T21:58:44.4181168Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4181547Z layer_outputs = layer_module( 2025-08-14T21:58:44.4181879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4182212Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4182607Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4182994Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4183392Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4183764Z self_outputs = self.self( 2025-08-14T21:58:44.4184112Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T21:58:44.4184537Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T21:58:44.4184709Z 2025-08-14T21:58:44.4184844Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4185068Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4185282Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4185615Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4185926Z return mod(**inputs) 2025-08-14T21:58:44.4186297Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4186673Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4187057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4187438Z hidden_states = self.encoder( 2025-08-14T21:58:44.4187808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4188190Z layer_outputs = layer_module( 2025-08-14T21:58:44.4188521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4188874Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4189251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4189643Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4190037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4190403Z self_outputs = self.self( 2025-08-14T21:58:44.4190763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T21:58:44.4191181Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T21:58:44.4191336Z 2025-08-14T21:58:44.4191441Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4191773Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4192077Z return mod(**inputs) 2025-08-14T21:58:44.4192445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4192822Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4193187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4193553Z hidden_states = self.encoder( 2025-08-14T21:58:44.4193918Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4194283Z layer_outputs = layer_module( 2025-08-14T21:58:44.4194594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4194934Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4195314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4195726Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4196114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T21:58:44.4196606Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:58:44.4197023Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T21:58:44.4197407Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4197546Z 2025-08-14T21:58:44.4197645Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4197991Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4198312Z return mod(**inputs) 2025-08-14T21:58:44.4198669Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4199074Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4199491Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4199864Z hidden_states = self.encoder( 2025-08-14T21:58:44.4200236Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4200612Z layer_outputs = layer_module( 2025-08-14T21:58:44.4200938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4201273Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4201654Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4202047Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4202430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4202818Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4203238Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4203705Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4204137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T21:58:44.4204540Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4204679Z 2025-08-14T21:58:44.4204780Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4205130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4205474Z return mod(**inputs) 2025-08-14T21:58:44.4205935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4206384Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4206808Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4207201Z hidden_states = self.encoder( 2025-08-14T21:58:44.4207588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4207969Z layer_outputs = layer_module( 2025-08-14T21:58:44.4208292Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4208636Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4209020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4209431Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4209804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4210203Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4210623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4211083Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4211527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T21:58:44.4211947Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:58:44.4212305Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:44.4212645Z return self.act(input) 2025-08-14T21:58:44.4212764Z 2025-08-14T21:58:44.4212865Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4213251Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4213571Z return mod(**inputs) 2025-08-14T21:58:44.4213933Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4214338Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4214720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4215090Z hidden_states = self.encoder( 2025-08-14T21:58:44.4215455Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4215833Z layer_outputs = layer_module( 2025-08-14T21:58:44.4216153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4216485Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4216861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4217255Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4217633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4218021Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4218444Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T21:58:44.4218929Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:58:44.4219370Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T21:58:44.4219783Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4219922Z 2025-08-14T21:58:44.4220024Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4220377Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4220695Z return mod(**inputs) 2025-08-14T21:58:44.4221065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4221470Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4221866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4222266Z hidden_states = self.encoder( 2025-08-14T21:58:44.4222652Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4223069Z layer_outputs = layer_module( 2025-08-14T21:58:44.4223407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4223768Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4224187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4224595Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4224991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4225402Z self_outputs = self.self( 2025-08-14T21:58:44.4225788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T21:58:44.4226200Z mixed_query_layer = self.query(hidden_states) 2025-08-14T21:58:44.4226354Z 2025-08-14T21:58:44.4226457Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4226814Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4227182Z return mod(**inputs) 2025-08-14T21:58:44.4227555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4227965Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4228372Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4228759Z hidden_states = self.encoder( 2025-08-14T21:58:44.4229151Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4229555Z layer_outputs = layer_module( 2025-08-14T21:58:44.4229896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4230247Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4230650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4231061Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4231468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4231854Z self_outputs = self.self( 2025-08-14T21:58:44.4232237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T21:58:44.4232648Z mixed_key_layer = self.key(hidden_states) 2025-08-14T21:58:44.4232795Z 2025-08-14T21:58:44.4232893Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4233233Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4233553Z return mod(**inputs) 2025-08-14T21:58:44.4233910Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4234288Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4234672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4235050Z hidden_states = self.encoder( 2025-08-14T21:58:44.4235416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4235783Z layer_outputs = layer_module( 2025-08-14T21:58:44.4236108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4236445Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4236811Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4237231Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4237727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4238175Z self_outputs = self.self( 2025-08-14T21:58:44.4238546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T21:58:44.4238960Z mixed_value_layer = self.value(hidden_states) 2025-08-14T21:58:44.4239104Z 2025-08-14T21:58:44.4239195Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4239396Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4239632Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4239983Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4240306Z return mod(**inputs) 2025-08-14T21:58:44.4240706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4241180Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4241601Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4242007Z hidden_states = self.encoder( 2025-08-14T21:58:44.4242404Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4242805Z layer_outputs = layer_module( 2025-08-14T21:58:44.4243153Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4243511Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4243915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4244332Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4244739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4245139Z self_outputs = self.self( 2025-08-14T21:58:44.4245527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T21:58:44.4246030Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T21:58:44.4246200Z 2025-08-14T21:58:44.4246283Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4246534Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4246910Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4247233Z return mod(**inputs) 2025-08-14T21:58:44.4247588Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4247973Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4248362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4248809Z hidden_states = self.encoder( 2025-08-14T21:58:44.4249179Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4249559Z layer_outputs = layer_module( 2025-08-14T21:58:44.4249891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4250231Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4250625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4251063Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4251458Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4251857Z self_outputs = self.self( 2025-08-14T21:58:44.4252232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.4252708Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.4253174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T21:58:44.4253627Z x = self.depthwise(hidden_states) 2025-08-14T21:58:44.4253762Z 2025-08-14T21:58:44.4253862Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4254211Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4254519Z return mod(**inputs) 2025-08-14T21:58:44.4254926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4255325Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4255714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4256092Z hidden_states = self.encoder( 2025-08-14T21:58:44.4256467Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4256850Z layer_outputs = layer_module( 2025-08-14T21:58:44.4257175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4257526Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4257911Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4258307Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4258690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4259073Z self_outputs = self.self( 2025-08-14T21:58:44.4259442Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.4259907Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.4260368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T21:58:44.4260751Z x = self.pointwise(x) 2025-08-14T21:58:44.4260858Z 2025-08-14T21:58:44.4260966Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4261306Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4261623Z return mod(**inputs) 2025-08-14T21:58:44.4261966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4262344Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4262716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4263095Z hidden_states = self.encoder( 2025-08-14T21:58:44.4263460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4263826Z layer_outputs = layer_module( 2025-08-14T21:58:44.4264149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4264515Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4264901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4265271Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4265663Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4266028Z self_outputs = self.self( 2025-08-14T21:58:44.4266389Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T21:58:44.4266831Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T21:58:44.4267037Z 2025-08-14T21:58:44.4267130Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4267462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4267763Z return mod(**inputs) 2025-08-14T21:58:44.4268148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4268578Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4268982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4269355Z hidden_states = self.encoder( 2025-08-14T21:58:44.4269727Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4270109Z layer_outputs = layer_module( 2025-08-14T21:58:44.4270427Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4270750Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4271116Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4271488Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4271851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4272217Z self_outputs = self.self( 2025-08-14T21:58:44.4272570Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T21:58:44.4272978Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T21:58:44.4273135Z 2025-08-14T21:58:44.4273231Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4273565Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4273867Z return mod(**inputs) 2025-08-14T21:58:44.4274213Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4274608Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4275032Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4275417Z hidden_states = self.encoder( 2025-08-14T21:58:44.4275786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4276161Z layer_outputs = layer_module( 2025-08-14T21:58:44.4276482Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4276824Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4277186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4277562Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4277967Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4278338Z self_outputs = self.self( 2025-08-14T21:58:44.4278729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T21:58:44.4279156Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T21:58:44.4279322Z 2025-08-14T21:58:44.4279414Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4279606Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4279826Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4280162Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4280456Z return mod(**inputs) 2025-08-14T21:58:44.4280810Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4281196Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4281610Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4281984Z hidden_states = self.encoder( 2025-08-14T21:58:44.4282357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4282740Z layer_outputs = layer_module( 2025-08-14T21:58:44.4283066Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4283400Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4283780Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4284166Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4284545Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4284927Z self_outputs = self.self( 2025-08-14T21:58:44.4285293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T21:58:44.4285834Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T21:58:44.4286017Z 2025-08-14T21:58:44.4286127Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4286509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4286851Z return mod(**inputs) 2025-08-14T21:58:44.4287219Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4287623Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4288016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4288397Z hidden_states = self.encoder( 2025-08-14T21:58:44.4288763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4289141Z layer_outputs = layer_module( 2025-08-14T21:58:44.4289463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4289804Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4290177Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4290593Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4290974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T21:58:44.4291433Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:58:44.4291868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T21:58:44.4292278Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4292410Z 2025-08-14T21:58:44.4292515Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4292846Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4293158Z return mod(**inputs) 2025-08-14T21:58:44.4293517Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4293907Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4294287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4294670Z hidden_states = self.encoder( 2025-08-14T21:58:44.4295081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4295462Z layer_outputs = layer_module( 2025-08-14T21:58:44.4295789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4296128Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4296512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4296900Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4297291Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4297663Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4298071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4298530Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4298968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T21:58:44.4299368Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4299509Z 2025-08-14T21:58:44.4299606Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4299949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4300258Z return mod(**inputs) 2025-08-14T21:58:44.4300619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4301005Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4301386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4301770Z hidden_states = self.encoder( 2025-08-14T21:58:44.4302135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4302522Z layer_outputs = layer_module( 2025-08-14T21:58:44.4302844Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4303173Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4303537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4303923Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4304301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4304702Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4305103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4305584Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4306011Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T21:58:44.4306426Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:58:44.4306795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:44.4307116Z return self.act(input) 2025-08-14T21:58:44.4307221Z 2025-08-14T21:58:44.4307326Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4307655Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4307970Z return mod(**inputs) 2025-08-14T21:58:44.4308344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4308793Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4309186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4309573Z hidden_states = self.encoder( 2025-08-14T21:58:44.4309958Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4310325Z layer_outputs = layer_module( 2025-08-14T21:58:44.4310651Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4310993Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4311369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4311749Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4312124Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4312498Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4312892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T21:58:44.4313352Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:58:44.4313803Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T21:58:44.4314188Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4314318Z 2025-08-14T21:58:44.4314414Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4314752Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4315055Z return mod(**inputs) 2025-08-14T21:58:44.4315412Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4315792Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4316172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4316549Z hidden_states = self.encoder( 2025-08-14T21:58:44.4316908Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4317284Z layer_outputs = layer_module( 2025-08-14T21:58:44.4317608Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4317946Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4318373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4318762Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4319161Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4319533Z self_outputs = self.self( 2025-08-14T21:58:44.4319888Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T21:58:44.4320281Z mixed_query_layer = self.query(hidden_states) 2025-08-14T21:58:44.4320418Z 2025-08-14T21:58:44.4320524Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4320865Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4321177Z return mod(**inputs) 2025-08-14T21:58:44.4321543Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4321978Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4322354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4322733Z hidden_states = self.encoder( 2025-08-14T21:58:44.4323096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4323472Z layer_outputs = layer_module( 2025-08-14T21:58:44.4323789Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4324123Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4324499Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4324885Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4325278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4325732Z self_outputs = self.self( 2025-08-14T21:58:44.4326167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T21:58:44.4326608Z mixed_key_layer = self.key(hidden_states) 2025-08-14T21:58:44.4326761Z 2025-08-14T21:58:44.4326871Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4327244Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4327544Z return mod(**inputs) 2025-08-14T21:58:44.4327901Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4328292Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4328675Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4329046Z hidden_states = self.encoder( 2025-08-14T21:58:44.4329416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4329796Z layer_outputs = layer_module( 2025-08-14T21:58:44.4330129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4330473Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4330851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4331234Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4331605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4332033Z self_outputs = self.self( 2025-08-14T21:58:44.4332397Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T21:58:44.4332817Z mixed_value_layer = self.value(hidden_states) 2025-08-14T21:58:44.4332958Z 2025-08-14T21:58:44.4333037Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4333248Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4333478Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4333817Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4334137Z return mod(**inputs) 2025-08-14T21:58:44.4334498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4334892Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4335278Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4335700Z hidden_states = self.encoder( 2025-08-14T21:58:44.4336080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4336461Z layer_outputs = layer_module( 2025-08-14T21:58:44.4336787Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4337128Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4337512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4338049Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4338446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4338832Z self_outputs = self.self( 2025-08-14T21:58:44.4339209Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T21:58:44.4339618Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T21:58:44.4339774Z 2025-08-14T21:58:44.4339851Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4340081Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4340417Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4340729Z return mod(**inputs) 2025-08-14T21:58:44.4341094Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4341485Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4341871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4342257Z hidden_states = self.encoder( 2025-08-14T21:58:44.4342633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4343006Z layer_outputs = layer_module( 2025-08-14T21:58:44.4343315Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4343648Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4344027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4344402Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4344786Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4345212Z self_outputs = self.self( 2025-08-14T21:58:44.4345580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.4346023Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.4346515Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T21:58:44.4346896Z x = self.depthwise(hidden_states) 2025-08-14T21:58:44.4347014Z 2025-08-14T21:58:44.4347129Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4347462Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4347773Z return mod(**inputs) 2025-08-14T21:58:44.4348138Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4348527Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4348941Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4349360Z hidden_states = self.encoder( 2025-08-14T21:58:44.4349750Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4350118Z layer_outputs = layer_module( 2025-08-14T21:58:44.4350443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4350785Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4351170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4351570Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4351950Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4352326Z self_outputs = self.self( 2025-08-14T21:58:44.4352681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.4353133Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.4353590Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T21:58:44.4353966Z x = self.pointwise(x) 2025-08-14T21:58:44.4354071Z 2025-08-14T21:58:44.4354167Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4354509Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4354818Z return mod(**inputs) 2025-08-14T21:58:44.4355169Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4355548Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4355922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4356287Z hidden_states = self.encoder( 2025-08-14T21:58:44.4356637Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4357000Z layer_outputs = layer_module( 2025-08-14T21:58:44.4357314Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4357634Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4358008Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4358395Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4358804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4359169Z self_outputs = self.self( 2025-08-14T21:58:44.4359555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T21:58:44.4360009Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T21:58:44.4360204Z 2025-08-14T21:58:44.4360308Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4360641Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4360949Z return mod(**inputs) 2025-08-14T21:58:44.4361307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4361687Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4362074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4362508Z hidden_states = self.encoder( 2025-08-14T21:58:44.4362896Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4363333Z layer_outputs = layer_module( 2025-08-14T21:58:44.4363677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4364036Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4364433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4364837Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4365231Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4365613Z self_outputs = self.self( 2025-08-14T21:58:44.4366053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T21:58:44.4366515Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T21:58:44.4366705Z 2025-08-14T21:58:44.4366813Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4367177Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4367483Z return mod(**inputs) 2025-08-14T21:58:44.4367849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4368247Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4368639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4369013Z hidden_states = self.encoder( 2025-08-14T21:58:44.4369387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4369764Z layer_outputs = layer_module( 2025-08-14T21:58:44.4370081Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4370419Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4370792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4371179Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4371556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4371928Z self_outputs = self.self( 2025-08-14T21:58:44.4372320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T21:58:44.4372745Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T21:58:44.4372938Z 2025-08-14T21:58:44.4373014Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4373215Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4373434Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4373766Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4374070Z return mod(**inputs) 2025-08-14T21:58:44.4374425Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4374801Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4375189Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4375569Z hidden_states = self.encoder( 2025-08-14T21:58:44.4375966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4376397Z layer_outputs = layer_module( 2025-08-14T21:58:44.4376724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4377065Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4377449Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4377830Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4378214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4378604Z self_outputs = self.self( 2025-08-14T21:58:44.4378974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T21:58:44.4379404Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T21:58:44.4379572Z 2025-08-14T21:58:44.4379672Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4380029Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4380324Z return mod(**inputs) 2025-08-14T21:58:44.4380683Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4381072Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4381459Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4381831Z hidden_states = self.encoder( 2025-08-14T21:58:44.4382202Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4382584Z layer_outputs = layer_module( 2025-08-14T21:58:44.4382894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4383228Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4383597Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4383976Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4384337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T21:58:44.4384754Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:58:44.4385167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T21:58:44.4385572Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4385700Z 2025-08-14T21:58:44.4385796Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4386130Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4386447Z return mod(**inputs) 2025-08-14T21:58:44.4386790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4387165Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4387536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4387906Z hidden_states = self.encoder( 2025-08-14T21:58:44.4388266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4388639Z layer_outputs = layer_module( 2025-08-14T21:58:44.4388959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4389323Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4389716Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4390097Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4390469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4390826Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4391221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4391670Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4392091Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T21:58:44.4392472Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4392611Z 2025-08-14T21:58:44.4392709Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4393044Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4393345Z return mod(**inputs) 2025-08-14T21:58:44.4393702Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4394088Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4394474Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4394863Z hidden_states = self.encoder( 2025-08-14T21:58:44.4395237Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4395621Z layer_outputs = layer_module( 2025-08-14T21:58:44.4395948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4396284Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4396662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4397055Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4397431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4397805Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4398207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4398695Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4399114Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T21:58:44.4399553Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:58:44.4399921Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:44.4400254Z return self.act(input) 2025-08-14T21:58:44.4400363Z 2025-08-14T21:58:44.4400464Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4400811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4401122Z return mod(**inputs) 2025-08-14T21:58:44.4401476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4401875Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4402271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4402691Z hidden_states = self.encoder( 2025-08-14T21:58:44.4403056Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4403432Z layer_outputs = layer_module( 2025-08-14T21:58:44.4403790Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4404285Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4404690Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4405095Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4405485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4405958Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4406431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T21:58:44.4406960Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:58:44.4407410Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T21:58:44.4407788Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4407928Z 2025-08-14T21:58:44.4408025Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4408362Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4408663Z return mod(**inputs) 2025-08-14T21:58:44.4409033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4409445Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4409852Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4410235Z hidden_states = self.encoder( 2025-08-14T21:58:44.4410657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4411037Z layer_outputs = layer_module( 2025-08-14T21:58:44.4411366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4411700Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4412079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4412468Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4412938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4413321Z self_outputs = self.self( 2025-08-14T21:58:44.4413715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T21:58:44.4414145Z mixed_query_layer = self.query(hidden_states) 2025-08-14T21:58:44.4414286Z 2025-08-14T21:58:44.4414385Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4414725Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4415038Z return mod(**inputs) 2025-08-14T21:58:44.4415388Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4415773Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4416155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4416540Z hidden_states = self.encoder( 2025-08-14T21:58:44.4416997Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4417378Z layer_outputs = layer_module( 2025-08-14T21:58:44.4417715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4418078Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4418475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4418889Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4419281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4419665Z self_outputs = self.self( 2025-08-14T21:58:44.4420020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T21:58:44.4420398Z mixed_key_layer = self.key(hidden_states) 2025-08-14T21:58:44.4420524Z 2025-08-14T21:58:44.4420626Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4420946Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4421245Z return mod(**inputs) 2025-08-14T21:58:44.4421587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4421960Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4422319Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4422681Z hidden_states = self.encoder( 2025-08-14T21:58:44.4423039Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4423398Z layer_outputs = layer_module( 2025-08-14T21:58:44.4423715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4424041Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4424406Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4424768Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4425139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4425501Z self_outputs = self.self( 2025-08-14T21:58:44.4425842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T21:58:44.4427080Z mixed_value_layer = self.value(hidden_states) 2025-08-14T21:58:44.4427221Z 2025-08-14T21:58:44.4427302Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4427530Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4427748Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4428090Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4428396Z return mod(**inputs) 2025-08-14T21:58:44.4428744Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4429131Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4429520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4429888Z hidden_states = self.encoder( 2025-08-14T21:58:44.4430243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4430642Z layer_outputs = layer_module( 2025-08-14T21:58:44.4430961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4431291Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4431650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4432024Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4432407Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4432770Z self_outputs = self.self( 2025-08-14T21:58:44.4433135Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T21:58:44.4433541Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T21:58:44.4433684Z 2025-08-14T21:58:44.4433768Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4433988Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4434322Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4434626Z return mod(**inputs) 2025-08-14T21:58:44.4434968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4435352Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4435733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4436104Z hidden_states = self.encoder( 2025-08-14T21:58:44.4436460Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4436836Z layer_outputs = layer_module( 2025-08-14T21:58:44.4437163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4437501Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4437993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4438383Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4438766Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4439134Z self_outputs = self.self( 2025-08-14T21:58:44.4439500Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.4439964Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.4440475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T21:58:44.4440873Z x = self.depthwise(hidden_states) 2025-08-14T21:58:44.4441002Z 2025-08-14T21:58:44.4441102Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4441439Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4441743Z return mod(**inputs) 2025-08-14T21:58:44.4442095Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4442483Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4442870Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4443238Z hidden_states = self.encoder( 2025-08-14T21:58:44.4443609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4444039Z layer_outputs = layer_module( 2025-08-14T21:58:44.4444373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4444705Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4445106Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4445519Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4446074Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4446509Z self_outputs = self.self( 2025-08-14T21:58:44.4446913Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.4447377Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.4447832Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T21:58:44.4448217Z x = self.pointwise(x) 2025-08-14T21:58:44.4448329Z 2025-08-14T21:58:44.4448428Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4448776Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4449077Z return mod(**inputs) 2025-08-14T21:58:44.4449436Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4449821Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4450193Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4450574Z hidden_states = self.encoder( 2025-08-14T21:58:44.4450947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4451326Z layer_outputs = layer_module( 2025-08-14T21:58:44.4451642Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4451981Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4452360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4452757Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4453120Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4453485Z self_outputs = self.self( 2025-08-14T21:58:44.4453867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T21:58:44.4454307Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T21:58:44.4454518Z 2025-08-14T21:58:44.4454613Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4454944Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4455248Z return mod(**inputs) 2025-08-14T21:58:44.4455587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4455966Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4456338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4456702Z hidden_states = self.encoder( 2025-08-14T21:58:44.4457051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4457491Z layer_outputs = layer_module( 2025-08-14T21:58:44.4457834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4458170Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4458585Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4459005Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4459414Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4459786Z self_outputs = self.self( 2025-08-14T21:58:44.4460158Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T21:58:44.4460600Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T21:58:44.4460760Z 2025-08-14T21:58:44.4460867Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4461199Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4461502Z return mod(**inputs) 2025-08-14T21:58:44.4461854Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4462232Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4462613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4462985Z hidden_states = self.encoder( 2025-08-14T21:58:44.4463350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4463727Z layer_outputs = layer_module( 2025-08-14T21:58:44.4464053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4464392Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4464765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4465152Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4465536Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4465909Z self_outputs = self.self( 2025-08-14T21:58:44.4466263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T21:58:44.4466684Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T21:58:44.4466870Z 2025-08-14T21:58:44.4466952Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4467150Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4467366Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4467720Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4468024Z return mod(**inputs) 2025-08-14T21:58:44.4468371Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4468755Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4469132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4469507Z hidden_states = self.encoder( 2025-08-14T21:58:44.4469874Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4470251Z layer_outputs = layer_module( 2025-08-14T21:58:44.4470606Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4470955Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4471338Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4471725Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4472100Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4472472Z self_outputs = self.self( 2025-08-14T21:58:44.4472830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T21:58:44.4473241Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T21:58:44.4473399Z 2025-08-14T21:58:44.4473507Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4473855Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4474156Z return mod(**inputs) 2025-08-14T21:58:44.4474505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4474880Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4475258Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4475630Z hidden_states = self.encoder( 2025-08-14T21:58:44.4475993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4476362Z layer_outputs = layer_module( 2025-08-14T21:58:44.4476685Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4477021Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4477396Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4477775Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4478155Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T21:58:44.4478585Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:58:44.4479012Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T21:58:44.4479395Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4479523Z 2025-08-14T21:58:44.4479626Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4479958Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4480283Z return mod(**inputs) 2025-08-14T21:58:44.4480639Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4481041Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4481416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4481791Z hidden_states = self.encoder( 2025-08-14T21:58:44.4482156Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4482535Z layer_outputs = layer_module( 2025-08-14T21:58:44.4482851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4483185Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4483562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4483979Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4484357Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4484739Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4485149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4485603Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4486132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T21:58:44.4486563Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4486710Z 2025-08-14T21:58:44.4486834Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4487191Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4487509Z return mod(**inputs) 2025-08-14T21:58:44.4487880Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4488280Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4488674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4489103Z hidden_states = self.encoder( 2025-08-14T21:58:44.4489520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4489945Z layer_outputs = layer_module( 2025-08-14T21:58:44.4490310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4490703Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4491129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4491550Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4491971Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4492388Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4492828Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4493328Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4493813Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T21:58:44.4494306Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:58:44.4494697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:44.4495076Z return self.act(input) 2025-08-14T21:58:44.4495218Z 2025-08-14T21:58:44.4495343Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4495703Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4496020Z return mod(**inputs) 2025-08-14T21:58:44.4496429Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4496873Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4497273Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4497673Z hidden_states = self.encoder( 2025-08-14T21:58:44.4498068Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4498523Z layer_outputs = layer_module( 2025-08-14T21:58:44.4498894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4499276Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4499700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4500139Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4500559Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4500969Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4501255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T21:58:44.4501404Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:58:44.4501655Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T21:58:44.4501741Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4501745Z 2025-08-14T21:58:44.4501844Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4502030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4502099Z return mod(**inputs) 2025-08-14T21:58:44.4502351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4502429Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4502684Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4502754Z hidden_states = self.encoder( 2025-08-14T21:58:44.4503013Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4503082Z layer_outputs = layer_module( 2025-08-14T21:58:44.4503290Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4503370Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4503618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4503695Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4503947Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4504014Z self_outputs = self.self( 2025-08-14T21:58:44.4504294Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T21:58:44.4504387Z mixed_query_layer = self.query(hidden_states) 2025-08-14T21:58:44.4504407Z 2025-08-14T21:58:44.4504511Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4504715Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4504779Z return mod(**inputs) 2025-08-14T21:58:44.4505058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4505142Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4505416Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4505499Z hidden_states = self.encoder( 2025-08-14T21:58:44.4505774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4505849Z layer_outputs = layer_module( 2025-08-14T21:58:44.4506119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4506203Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4506493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4506572Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4506837Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4506915Z self_outputs = self.self( 2025-08-14T21:58:44.4507175Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T21:58:44.4507265Z mixed_key_layer = self.key(hidden_states) 2025-08-14T21:58:44.4507269Z 2025-08-14T21:58:44.4507372Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4507576Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4507654Z return mod(**inputs) 2025-08-14T21:58:44.4507935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4508019Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4508299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4508375Z hidden_states = self.encoder( 2025-08-14T21:58:44.4508658Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4508734Z layer_outputs = layer_module( 2025-08-14T21:58:44.4508969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4509059Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4509336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4509433Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4509687Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4509754Z self_outputs = self.self( 2025-08-14T21:58:44.4510021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T21:58:44.4510107Z mixed_value_layer = self.value(hidden_states) 2025-08-14T21:58:44.4510111Z 2025-08-14T21:58:44.4510188Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4510291Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4510390Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4510595Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4510678Z return mod(**inputs) 2025-08-14T21:58:44.4510940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4511026Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4511287Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4511358Z hidden_states = self.encoder( 2025-08-14T21:58:44.4511623Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4511692Z layer_outputs = layer_module( 2025-08-14T21:58:44.4511919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4511993Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4512310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4512400Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4512667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4512738Z self_outputs = self.self( 2025-08-14T21:58:44.4512990Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T21:58:44.4513092Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T21:58:44.4513095Z 2025-08-14T21:58:44.4513179Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4513282Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4513477Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4513552Z return mod(**inputs) 2025-08-14T21:58:44.4513816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4513904Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4514164Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4514235Z hidden_states = self.encoder( 2025-08-14T21:58:44.4514505Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4514574Z layer_outputs = layer_module( 2025-08-14T21:58:44.4514796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4514881Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4515148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4515236Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4515498Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4515566Z self_outputs = self.self( 2025-08-14T21:58:44.4515834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.4515994Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.4516266Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T21:58:44.4516361Z x = self.depthwise(hidden_states) 2025-08-14T21:58:44.4516365Z 2025-08-14T21:58:44.4516467Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4516673Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4516756Z return mod(**inputs) 2025-08-14T21:58:44.4517025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4517103Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4517363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4517440Z hidden_states = self.encoder( 2025-08-14T21:58:44.4517706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4517776Z layer_outputs = layer_module( 2025-08-14T21:58:44.4518003Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4518096Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4518380Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4518461Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4518721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4518796Z self_outputs = self.self( 2025-08-14T21:58:44.4519053Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.4519208Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.4519473Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T21:58:44.4519544Z x = self.pointwise(x) 2025-08-14T21:58:44.4519547Z 2025-08-14T21:58:44.4519657Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4519854Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4519920Z return mod(**inputs) 2025-08-14T21:58:44.4520190Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4520269Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4520537Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4520608Z hidden_states = self.encoder( 2025-08-14T21:58:44.4520867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4520947Z layer_outputs = layer_module( 2025-08-14T21:58:44.4521167Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4521244Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4521516Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4521594Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4521861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4521931Z self_outputs = self.self( 2025-08-14T21:58:44.4522188Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T21:58:44.4522349Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T21:58:44.4522370Z 2025-08-14T21:58:44.4522473Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4522678Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4522768Z return mod(**inputs) 2025-08-14T21:58:44.4523037Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4523124Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4523387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4523458Z hidden_states = self.encoder( 2025-08-14T21:58:44.4523729Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4523798Z layer_outputs = layer_module( 2025-08-14T21:58:44.4524026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4524103Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4524399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4524489Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4524751Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4524827Z self_outputs = self.self( 2025-08-14T21:58:44.4525088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T21:58:44.4525206Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T21:58:44.4525210Z 2025-08-14T21:58:44.4525318Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4525515Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4525579Z return mod(**inputs) 2025-08-14T21:58:44.4525930Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4526022Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4526313Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4526390Z hidden_states = self.encoder( 2025-08-14T21:58:44.4526670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4526755Z layer_outputs = layer_module( 2025-08-14T21:58:44.4526991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4527083Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4527368Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4527450Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4527720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4527794Z self_outputs = self.self( 2025-08-14T21:58:44.4528054Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T21:58:44.4528182Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T21:58:44.4528186Z 2025-08-14T21:58:44.4528264Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4528347Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4528449Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4528659Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4528730Z return mod(**inputs) 2025-08-14T21:58:44.4528985Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4529076Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4529350Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4529424Z hidden_states = self.encoder( 2025-08-14T21:58:44.4529724Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4529798Z layer_outputs = layer_module( 2025-08-14T21:58:44.4530031Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4530119Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4530401Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4530522Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4530802Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4530875Z self_outputs = self.self( 2025-08-14T21:58:44.4531165Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T21:58:44.4531275Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T21:58:44.4531279Z 2025-08-14T21:58:44.4531378Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4531574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4531639Z return mod(**inputs) 2025-08-14T21:58:44.4531903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4531983Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4532244Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4532320Z hidden_states = self.encoder( 2025-08-14T21:58:44.4532573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4532652Z layer_outputs = layer_module( 2025-08-14T21:58:44.4532860Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4532935Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4533199Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4533282Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4533546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T21:58:44.4533681Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:58:44.4533943Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T21:58:44.4534031Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4534036Z 2025-08-14T21:58:44.4534136Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4534330Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4534403Z return mod(**inputs) 2025-08-14T21:58:44.4534662Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4534765Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4535026Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4535112Z hidden_states = self.encoder( 2025-08-14T21:58:44.4535383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4535455Z layer_outputs = layer_module( 2025-08-14T21:58:44.4535672Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4535755Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4536015Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4536105Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4536363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4536457Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4536788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4536909Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4537178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T21:58:44.4537260Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4537264Z 2025-08-14T21:58:44.4537366Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4537573Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4537803Z return mod(**inputs) 2025-08-14T21:58:44.4538096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4538192Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4538485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4538570Z hidden_states = self.encoder( 2025-08-14T21:58:44.4538858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4538934Z layer_outputs = layer_module( 2025-08-14T21:58:44.4539181Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4539257Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4539521Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4539603Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4539856Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4539941Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4540227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4540345Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4540609Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T21:58:44.4540717Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:58:44.4540931Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:44.4540998Z return self.act(input) 2025-08-14T21:58:44.4541048Z 2025-08-14T21:58:44.4541148Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4541350Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4541438Z return mod(**inputs) 2025-08-14T21:58:44.4541706Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4541782Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4542038Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4542114Z hidden_states = self.encoder( 2025-08-14T21:58:44.4542375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4542445Z layer_outputs = layer_module( 2025-08-14T21:58:44.4542665Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4542741Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4543048Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4543131Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4543384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4543467Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4543755Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T21:58:44.4543885Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:58:44.4544129Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T21:58:44.4544206Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4544209Z 2025-08-14T21:58:44.4544313Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4544494Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4544555Z return mod(**inputs) 2025-08-14T21:58:44.4544807Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4544879Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4545130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4545196Z hidden_states = self.encoder( 2025-08-14T21:58:44.4545443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4545520Z layer_outputs = layer_module( 2025-08-14T21:58:44.4545721Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4545795Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4546051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4546125Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4546375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4546439Z self_outputs = self.self( 2025-08-14T21:58:44.4546682Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T21:58:44.4546775Z mixed_query_layer = self.query(hidden_states) 2025-08-14T21:58:44.4546778Z 2025-08-14T21:58:44.4546897Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4547086Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4547149Z return mod(**inputs) 2025-08-14T21:58:44.4547409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4547490Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4547740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4547808Z hidden_states = self.encoder( 2025-08-14T21:58:44.4548070Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4548136Z layer_outputs = layer_module( 2025-08-14T21:58:44.4548352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4548427Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4548709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4548803Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4549051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4549126Z self_outputs = self.self( 2025-08-14T21:58:44.4549373Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T21:58:44.4549448Z mixed_key_layer = self.key(hidden_states) 2025-08-14T21:58:44.4549451Z 2025-08-14T21:58:44.4549554Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4549738Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4549801Z return mod(**inputs) 2025-08-14T21:58:44.4550057Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4550134Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4550394Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4550461Z hidden_states = self.encoder( 2025-08-14T21:58:44.4550714Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4550787Z layer_outputs = layer_module( 2025-08-14T21:58:44.4551001Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4551081Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4551344Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4551418Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4551677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4551744Z self_outputs = self.self( 2025-08-14T21:58:44.4551993Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T21:58:44.4552085Z mixed_value_layer = self.value(hidden_states) 2025-08-14T21:58:44.4552088Z 2025-08-14T21:58:44.4552162Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4552243Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4552336Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4552518Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4552605Z return mod(**inputs) 2025-08-14T21:58:44.4552853Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4552946Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4553195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4553261Z hidden_states = self.encoder( 2025-08-14T21:58:44.4553511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4553577Z layer_outputs = layer_module( 2025-08-14T21:58:44.4553779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4553856Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4554097Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4554175Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4554446Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4554513Z self_outputs = self.self( 2025-08-14T21:58:44.4554765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T21:58:44.4554860Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T21:58:44.4554864Z 2025-08-14T21:58:44.4554936Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4555035Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4555214Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4555282Z return mod(**inputs) 2025-08-14T21:58:44.4555527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4555601Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4555849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4555914Z hidden_states = self.encoder( 2025-08-14T21:58:44.4556160Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4556225Z layer_outputs = layer_module( 2025-08-14T21:58:44.4556424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4556498Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4556739Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4556814Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4557064Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4557129Z self_outputs = self.self( 2025-08-14T21:58:44.4557376Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.4557521Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.4557764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T21:58:44.4557842Z x = self.depthwise(hidden_states) 2025-08-14T21:58:44.4557845Z 2025-08-14T21:58:44.4557938Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4558127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4558210Z return mod(**inputs) 2025-08-14T21:58:44.4558452Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4558550Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4558792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4558859Z hidden_states = self.encoder( 2025-08-14T21:58:44.4559108Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4559173Z layer_outputs = layer_module( 2025-08-14T21:58:44.4559383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4559454Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4559696Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4559782Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4560058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4560126Z self_outputs = self.self( 2025-08-14T21:58:44.4560383Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.4560523Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.4560779Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T21:58:44.4560844Z x = self.pointwise(x) 2025-08-14T21:58:44.4560848Z 2025-08-14T21:58:44.4560942Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4561131Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4561191Z return mod(**inputs) 2025-08-14T21:58:44.4561448Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4561524Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4561822Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4561897Z hidden_states = self.encoder( 2025-08-14T21:58:44.4562150Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4562215Z layer_outputs = layer_module( 2025-08-14T21:58:44.4562431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4562505Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4562767Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4562846Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4563098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4563171Z self_outputs = self.self( 2025-08-14T21:58:44.4563426Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T21:58:44.4563576Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T21:58:44.4563580Z 2025-08-14T21:58:44.4563677Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4563862Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4563949Z return mod(**inputs) 2025-08-14T21:58:44.4564200Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4564277Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4564552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4564618Z hidden_states = self.encoder( 2025-08-14T21:58:44.4564877Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4564942Z layer_outputs = layer_module( 2025-08-14T21:58:44.4565149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4565229Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4565542Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4565698Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4566046Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4566124Z self_outputs = self.self( 2025-08-14T21:58:44.4566421Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T21:58:44.4566546Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T21:58:44.4566550Z 2025-08-14T21:58:44.4566657Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4566871Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4566934Z return mod(**inputs) 2025-08-14T21:58:44.4567197Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4567275Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4567530Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4567606Z hidden_states = self.encoder( 2025-08-14T21:58:44.4567858Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4567933Z layer_outputs = layer_module( 2025-08-14T21:58:44.4568139Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4568211Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4568469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4568544Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4568794Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4568868Z self_outputs = self.self( 2025-08-14T21:58:44.4569119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T21:58:44.4569244Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T21:58:44.4569248Z 2025-08-14T21:58:44.4569322Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4569396Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4569500Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4569683Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4569744Z return mod(**inputs) 2025-08-14T21:58:44.4570002Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4570094Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4570349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4570432Z hidden_states = self.encoder( 2025-08-14T21:58:44.4570677Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4570751Z layer_outputs = layer_module( 2025-08-14T21:58:44.4570957Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4571034Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4571282Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4571355Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4571612Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4571717Z self_outputs = self.self( 2025-08-14T21:58:44.4571970Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T21:58:44.4572080Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T21:58:44.4572084Z 2025-08-14T21:58:44.4572177Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4572369Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4572431Z return mod(**inputs) 2025-08-14T21:58:44.4572681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4572764Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4573014Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4573088Z hidden_states = self.encoder( 2025-08-14T21:58:44.4573340Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4573409Z layer_outputs = layer_module( 2025-08-14T21:58:44.4573626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4573697Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4573945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4574029Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4574279Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T21:58:44.4574407Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:58:44.4574661Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T21:58:44.4574745Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4574748Z 2025-08-14T21:58:44.4574852Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4575041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4575110Z return mod(**inputs) 2025-08-14T21:58:44.4575358Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4575434Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4575689Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4575772Z hidden_states = self.encoder( 2025-08-14T21:58:44.4576024Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4576114Z layer_outputs = layer_module( 2025-08-14T21:58:44.4576320Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4576402Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4576648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4576726Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4576977Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4577049Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4577335Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4577466Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4577731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T21:58:44.4577818Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4577821Z 2025-08-14T21:58:44.4577917Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4578103Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4578174Z return mod(**inputs) 2025-08-14T21:58:44.4578424Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4578506Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4578754Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4578830Z hidden_states = self.encoder( 2025-08-14T21:58:44.4579079Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4579145Z layer_outputs = layer_module( 2025-08-14T21:58:44.4579354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4579424Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4579667Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4579748Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4579984Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4580055Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4580337Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4580448Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4580700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T21:58:44.4580805Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:58:44.4581000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:44.4581074Z return self.act(input) 2025-08-14T21:58:44.4581077Z 2025-08-14T21:58:44.4581170Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4581357Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4581438Z return mod(**inputs) 2025-08-14T21:58:44.4581680Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4581765Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4582025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4582092Z hidden_states = self.encoder( 2025-08-14T21:58:44.4582343Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4582408Z layer_outputs = layer_module( 2025-08-14T21:58:44.4582613Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4582683Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4582923Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4583007Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4583270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4583345Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4583625Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T21:58:44.4583746Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:58:44.4583998Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T21:58:44.4584074Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4584077Z 2025-08-14T21:58:44.4584172Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4584360Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4584418Z return mod(**inputs) 2025-08-14T21:58:44.4584670Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4584744Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4584989Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4585061Z hidden_states = self.encoder( 2025-08-14T21:58:44.4585300Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4585370Z layer_outputs = layer_module( 2025-08-14T21:58:44.4585573Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4585642Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4585890Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4585967Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4586212Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4586285Z self_outputs = self.self( 2025-08-14T21:58:44.4586526Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T21:58:44.4586617Z mixed_query_layer = self.query(hidden_states) 2025-08-14T21:58:44.4586620Z 2025-08-14T21:58:44.4586714Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4586898Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4586967Z return mod(**inputs) 2025-08-14T21:58:44.4587224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4587299Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4587560Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4587625Z hidden_states = self.encoder( 2025-08-14T21:58:44.4587873Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4587938Z layer_outputs = layer_module( 2025-08-14T21:58:44.4588137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4588214Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4588454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4588537Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4588816Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4588884Z self_outputs = self.self( 2025-08-14T21:58:44.4589141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T21:58:44.4589218Z mixed_key_layer = self.key(hidden_states) 2025-08-14T21:58:44.4589221Z 2025-08-14T21:58:44.4589317Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4589506Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4589568Z return mod(**inputs) 2025-08-14T21:58:44.4589830Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4589906Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4590148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4590222Z hidden_states = self.encoder( 2025-08-14T21:58:44.4590463Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4590536Z layer_outputs = layer_module( 2025-08-14T21:58:44.4590737Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4590805Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4591051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4591124Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4591363Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4591438Z self_outputs = self.self( 2025-08-14T21:58:44.4591681Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T21:58:44.4591772Z mixed_value_layer = self.value(hidden_states) 2025-08-14T21:58:44.4591776Z 2025-08-14T21:58:44.4591851Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4591922Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4592022Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4592200Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4592262Z return mod(**inputs) 2025-08-14T21:58:44.4592508Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4592581Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4592849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4592916Z hidden_states = self.encoder( 2025-08-14T21:58:44.4593174Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4593246Z layer_outputs = layer_module( 2025-08-14T21:58:44.4593445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4593521Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4593764Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4593837Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4594084Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4594150Z self_outputs = self.self( 2025-08-14T21:58:44.4594422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T21:58:44.4594528Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T21:58:44.4594531Z 2025-08-14T21:58:44.4594606Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4594704Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4594885Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4594947Z return mod(**inputs) 2025-08-14T21:58:44.4595194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4595267Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4595513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4595579Z hidden_states = self.encoder( 2025-08-14T21:58:44.4595819Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4595892Z layer_outputs = layer_module( 2025-08-14T21:58:44.4596093Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4596163Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4596409Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4596482Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4596730Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4596795Z self_outputs = self.self( 2025-08-14T21:58:44.4597035Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.4597190Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.4597431Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T21:58:44.4597508Z x = self.depthwise(hidden_states) 2025-08-14T21:58:44.4597511Z 2025-08-14T21:58:44.4597604Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4597781Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4597849Z return mod(**inputs) 2025-08-14T21:58:44.4598090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4598179Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4598438Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4598519Z hidden_states = self.encoder( 2025-08-14T21:58:44.4598776Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4598844Z layer_outputs = layer_module( 2025-08-14T21:58:44.4599058Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4599137Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4599390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4599466Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4599728Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4599796Z self_outputs = self.self( 2025-08-14T21:58:44.4600088Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.4600240Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.4600496Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T21:58:44.4600571Z x = self.pointwise(x) 2025-08-14T21:58:44.4600575Z 2025-08-14T21:58:44.4600673Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4600868Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4600931Z return mod(**inputs) 2025-08-14T21:58:44.4601187Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4601273Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4601528Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4601597Z hidden_states = self.encoder( 2025-08-14T21:58:44.4601861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4601930Z layer_outputs = layer_module( 2025-08-14T21:58:44.4602149Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4602226Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4602485Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4602568Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4602829Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4602907Z self_outputs = self.self( 2025-08-14T21:58:44.4603170Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T21:58:44.4603319Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T21:58:44.4603323Z 2025-08-14T21:58:44.4603430Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4603622Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4603686Z return mod(**inputs) 2025-08-14T21:58:44.4603954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4604031Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4604316Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4604386Z hidden_states = self.encoder( 2025-08-14T21:58:44.4604666Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4604741Z layer_outputs = layer_module( 2025-08-14T21:58:44.4604961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4605045Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4605308Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4605388Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4605722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4605802Z self_outputs = self.self( 2025-08-14T21:58:44.4606142Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T21:58:44.4606282Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T21:58:44.4606287Z 2025-08-14T21:58:44.4606402Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4606619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4606686Z return mod(**inputs) 2025-08-14T21:58:44.4606965Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4607055Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4607346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4607430Z hidden_states = self.encoder( 2025-08-14T21:58:44.4607701Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4607770Z layer_outputs = layer_module( 2025-08-14T21:58:44.4607982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4608054Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4608306Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4608391Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4608648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4608725Z self_outputs = self.self( 2025-08-14T21:58:44.4608976Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T21:58:44.4609099Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T21:58:44.4609103Z 2025-08-14T21:58:44.4609187Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4609261Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4609362Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4609546Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4609608Z return mod(**inputs) 2025-08-14T21:58:44.4609868Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4609942Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4610192Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4610286Z hidden_states = self.encoder( 2025-08-14T21:58:44.4610540Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4610636Z layer_outputs = layer_module( 2025-08-14T21:58:44.4610849Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4610922Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4611184Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4611258Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4611512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4611586Z self_outputs = self.self( 2025-08-14T21:58:44.4611842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T21:58:44.4611986Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T21:58:44.4611991Z 2025-08-14T21:58:44.4612090Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4612276Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4612344Z return mod(**inputs) 2025-08-14T21:58:44.4612596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4612678Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4612926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4612994Z hidden_states = self.encoder( 2025-08-14T21:58:44.4613251Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4613320Z layer_outputs = layer_module( 2025-08-14T21:58:44.4613529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4613610Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4613861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4613943Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4614194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T21:58:44.4614320Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:58:44.4614577Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T21:58:44.4614655Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4614658Z 2025-08-14T21:58:44.4614762Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4614951Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4615014Z return mod(**inputs) 2025-08-14T21:58:44.4615271Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4615346Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4615595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4615668Z hidden_states = self.encoder( 2025-08-14T21:58:44.4615919Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4616012Z layer_outputs = layer_module( 2025-08-14T21:58:44.4616221Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4616294Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4616569Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4616647Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4616899Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4616970Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4617249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4617368Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4617619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T21:58:44.4617697Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4617738Z 2025-08-14T21:58:44.4617838Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4618025Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4618095Z return mod(**inputs) 2025-08-14T21:58:44.4618352Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4618430Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4618693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4618763Z hidden_states = self.encoder( 2025-08-14T21:58:44.4619025Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4619095Z layer_outputs = layer_module( 2025-08-14T21:58:44.4619321Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4619405Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4619653Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4619732Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4619983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4620056Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4620346Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4620461Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4620711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T21:58:44.4620826Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:58:44.4621027Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:44.4621099Z return self.act(input) 2025-08-14T21:58:44.4621103Z 2025-08-14T21:58:44.4621198Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4621383Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4621450Z return mod(**inputs) 2025-08-14T21:58:44.4621710Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4621783Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4622052Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4622120Z hidden_states = self.encoder( 2025-08-14T21:58:44.4622387Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4622452Z layer_outputs = layer_module( 2025-08-14T21:58:44.4622657Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4622736Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4622978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4623060Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4623295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4623365Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4623671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T21:58:44.4623795Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:58:44.4624043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T21:58:44.4624128Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4624131Z 2025-08-14T21:58:44.4624225Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4624416Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4624477Z return mod(**inputs) 2025-08-14T21:58:44.4624720Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4624803Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4625047Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4625121Z hidden_states = self.encoder( 2025-08-14T21:58:44.4625362Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4625426Z layer_outputs = layer_module( 2025-08-14T21:58:44.4625633Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4625703Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4625944Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4626025Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4626270Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4626343Z self_outputs = self.self( 2025-08-14T21:58:44.4626587Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T21:58:44.4626670Z mixed_query_layer = self.query(hidden_states) 2025-08-14T21:58:44.4626673Z 2025-08-14T21:58:44.4626773Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4626955Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4627021Z return mod(**inputs) 2025-08-14T21:58:44.4627263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4627336Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4627605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4627671Z hidden_states = self.encoder( 2025-08-14T21:58:44.4627924Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4628014Z layer_outputs = layer_module( 2025-08-14T21:58:44.4628224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4628303Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4628556Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4628631Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4628889Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4628957Z self_outputs = self.self( 2025-08-14T21:58:44.4629263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T21:58:44.4629350Z mixed_key_layer = self.key(hidden_states) 2025-08-14T21:58:44.4629353Z 2025-08-14T21:58:44.4629448Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4629637Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4629699Z return mod(**inputs) 2025-08-14T21:58:44.4629948Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4630032Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4630281Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4630357Z hidden_states = self.encoder( 2025-08-14T21:58:44.4630605Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4630685Z layer_outputs = layer_module( 2025-08-14T21:58:44.4630894Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4630963Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4631207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4631287Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4631525Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4631594Z self_outputs = self.self( 2025-08-14T21:58:44.4631834Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T21:58:44.4631919Z mixed_value_layer = self.value(hidden_states) 2025-08-14T21:58:44.4631922Z 2025-08-14T21:58:44.4632006Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4632081Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4632183Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4632363Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4632424Z return mod(**inputs) 2025-08-14T21:58:44.4632671Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4632743Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4633006Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4633078Z hidden_states = self.encoder( 2025-08-14T21:58:44.4633347Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4633421Z layer_outputs = layer_module( 2025-08-14T21:58:44.4633638Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4633708Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4633959Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4634032Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4634275Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4634346Z self_outputs = self.self( 2025-08-14T21:58:44.4634591Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T21:58:44.4634695Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T21:58:44.4634698Z 2025-08-14T21:58:44.4634798Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4634894Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4635085Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4635144Z return mod(**inputs) 2025-08-14T21:58:44.4635395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4635467Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4635709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4635780Z hidden_states = self.encoder( 2025-08-14T21:58:44.4636021Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4636087Z layer_outputs = layer_module( 2025-08-14T21:58:44.4636299Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4636370Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4636618Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4636692Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4636934Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4637007Z self_outputs = self.self( 2025-08-14T21:58:44.4637246Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.4637395Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.4637817Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T21:58:44.4637901Z x = self.depthwise(hidden_states) 2025-08-14T21:58:44.4637905Z 2025-08-14T21:58:44.4638021Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4638229Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4638298Z return mod(**inputs) 2025-08-14T21:58:44.4638595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4638681Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4638968Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4639043Z hidden_states = self.encoder( 2025-08-14T21:58:44.4639379Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4639458Z layer_outputs = layer_module( 2025-08-14T21:58:44.4639700Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4639775Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4640043Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4640123Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4640395Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4640465Z self_outputs = self.self( 2025-08-14T21:58:44.4640734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.4640895Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.4641191Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T21:58:44.4641271Z x = self.pointwise(x) 2025-08-14T21:58:44.4641275Z 2025-08-14T21:58:44.4641378Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4641574Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4641646Z return mod(**inputs) 2025-08-14T21:58:44.4641912Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4641990Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4642261Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4642332Z hidden_states = self.encoder( 2025-08-14T21:58:44.4642604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4642674Z layer_outputs = layer_module( 2025-08-14T21:58:44.4642891Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4642972Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4643235Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4643321Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4643581Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4643650Z self_outputs = self.self( 2025-08-14T21:58:44.4643922Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T21:58:44.4644075Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T21:58:44.4644080Z 2025-08-14T21:58:44.4644190Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4644385Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4644451Z return mod(**inputs) 2025-08-14T21:58:44.4644722Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4644802Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4645063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4645140Z hidden_states = self.encoder( 2025-08-14T21:58:44.4645420Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4645499Z layer_outputs = layer_module( 2025-08-14T21:58:44.4645792Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4645873Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4646145Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4646224Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4646487Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4646562Z self_outputs = self.self( 2025-08-14T21:58:44.4646827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T21:58:44.4646953Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T21:58:44.4646957Z 2025-08-14T21:58:44.4647092Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4647294Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4647369Z return mod(**inputs) 2025-08-14T21:58:44.4647635Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4647722Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4647983Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4648055Z hidden_states = self.encoder( 2025-08-14T21:58:44.4648327Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4648399Z layer_outputs = layer_module( 2025-08-14T21:58:44.4648621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4648707Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4648974Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4649058Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4649325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4649394Z self_outputs = self.self( 2025-08-14T21:58:44.4649664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T21:58:44.4649788Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T21:58:44.4649793Z 2025-08-14T21:58:44.4649879Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4649958Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4650063Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4650266Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4650332Z return mod(**inputs) 2025-08-14T21:58:44.4650595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4650681Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4650938Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4651014Z hidden_states = self.encoder( 2025-08-14T21:58:44.4651276Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4651365Z layer_outputs = layer_module( 2025-08-14T21:58:44.4651594Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4651695Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4651940Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4652020Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4652263Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4652334Z self_outputs = self.self( 2025-08-14T21:58:44.4652578Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T21:58:44.4652679Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T21:58:44.4652682Z 2025-08-14T21:58:44.4652784Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4652980Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4653078Z return mod(**inputs) 2025-08-14T21:58:44.4653324Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4653397Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4653645Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4653712Z hidden_states = self.encoder( 2025-08-14T21:58:44.4653954Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4654027Z layer_outputs = layer_module( 2025-08-14T21:58:44.4654229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4654307Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4654552Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4654626Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4654875Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T21:58:44.4654993Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:58:44.4655248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T21:58:44.4655324Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4655328Z 2025-08-14T21:58:44.4655424Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4655617Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4655681Z return mod(**inputs) 2025-08-14T21:58:44.4655935Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4656017Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4656264Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4656338Z hidden_states = self.encoder( 2025-08-14T21:58:44.4656595Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4656660Z layer_outputs = layer_module( 2025-08-14T21:58:44.4656866Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4656936Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4657195Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4657274Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4657527Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4657606Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4657887Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4657997Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4658255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T21:58:44.4658330Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4658334Z 2025-08-14T21:58:44.4658437Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4658619Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4658711Z return mod(**inputs) 2025-08-14T21:58:44.4658969Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4659044Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4659301Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4659379Z hidden_states = self.encoder( 2025-08-14T21:58:44.4659624Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4659694Z layer_outputs = layer_module( 2025-08-14T21:58:44.4659897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4659971Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4660229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4660308Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4660555Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4660628Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4660909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4661026Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4661277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T21:58:44.4661402Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:58:44.4661596Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:44.4661665Z return self.act(input) 2025-08-14T21:58:44.4661669Z 2025-08-14T21:58:44.4661770Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4661949Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4662011Z return mod(**inputs) 2025-08-14T21:58:44.4662257Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4662331Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4662580Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4662646Z hidden_states = self.encoder( 2025-08-14T21:58:44.4662904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4662977Z layer_outputs = layer_module( 2025-08-14T21:58:44.4663204Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4663277Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4663532Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4663610Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4663862Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4663935Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4664214Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T21:58:44.4664347Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:58:44.4664626Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T21:58:44.4664714Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4664717Z 2025-08-14T21:58:44.4664813Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4664998Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4665077Z return mod(**inputs) 2025-08-14T21:58:44.4665328Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4665421Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4665664Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4665733Z hidden_states = self.encoder( 2025-08-14T21:58:44.4665982Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4666048Z layer_outputs = layer_module( 2025-08-14T21:58:44.4666249Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4666327Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4666575Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4666655Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4666904Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4666971Z self_outputs = self.self( 2025-08-14T21:58:44.4667227Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T21:58:44.4667317Z mixed_query_layer = self.query(hidden_states) 2025-08-14T21:58:44.4667323Z 2025-08-14T21:58:44.4667425Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4667610Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4667672Z return mod(**inputs) 2025-08-14T21:58:44.4667926Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4668001Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4668252Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4668329Z hidden_states = self.encoder( 2025-08-14T21:58:44.4668583Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4668677Z layer_outputs = layer_module( 2025-08-14T21:58:44.4668902Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4668993Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4669248Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4669324Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4669572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4669645Z self_outputs = self.self( 2025-08-14T21:58:44.4669893Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T21:58:44.4669976Z mixed_key_layer = self.key(hidden_states) 2025-08-14T21:58:44.4669980Z 2025-08-14T21:58:44.4670074Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4670285Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4670355Z return mod(**inputs) 2025-08-14T21:58:44.4670604Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4670686Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4670936Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4671003Z hidden_states = self.encoder( 2025-08-14T21:58:44.4671262Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4671327Z layer_outputs = layer_module( 2025-08-14T21:58:44.4671533Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4671614Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4671863Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4671947Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4672194Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4672257Z self_outputs = self.self( 2025-08-14T21:58:44.4672513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T21:58:44.4672598Z mixed_value_layer = self.value(hidden_states) 2025-08-14T21:58:44.4672601Z 2025-08-14T21:58:44.4672683Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4672759Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4672855Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4673049Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4673112Z return mod(**inputs) 2025-08-14T21:58:44.4673360Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4673446Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4673693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4673769Z hidden_states = self.encoder( 2025-08-14T21:58:44.4674017Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4674084Z layer_outputs = layer_module( 2025-08-14T21:58:44.4674318Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4674389Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4674644Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4674743Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4674991Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4675066Z self_outputs = self.self( 2025-08-14T21:58:44.4675310Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T21:58:44.4675407Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T21:58:44.4675410Z 2025-08-14T21:58:44.4675493Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4675592Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4675783Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4675885Z return mod(**inputs) 2025-08-14T21:58:44.4676137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4676225Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4676472Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4676540Z hidden_states = self.encoder( 2025-08-14T21:58:44.4676799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4676866Z layer_outputs = layer_module( 2025-08-14T21:58:44.4677078Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4677151Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4677399Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4677485Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4677731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4677797Z self_outputs = self.self( 2025-08-14T21:58:44.4678051Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.4678199Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.4678454Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T21:58:44.4678527Z x = self.depthwise(hidden_states) 2025-08-14T21:58:44.4678531Z 2025-08-14T21:58:44.4678627Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4678889Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4678976Z return mod(**inputs) 2025-08-14T21:58:44.4679277Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4679565Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4679851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4680017Z hidden_states = self.encoder( 2025-08-14T21:58:44.4680295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4680411Z layer_outputs = layer_module( 2025-08-14T21:58:44.4680650Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4680773Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4681105Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4681200Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4681471Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4681586Z self_outputs = self.self( 2025-08-14T21:58:44.4681846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.4682074Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.4682349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T21:58:44.4682437Z x = self.pointwise(x) 2025-08-14T21:58:44.4682440Z 2025-08-14T21:58:44.4682612Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4682821Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4682940Z return mod(**inputs) 2025-08-14T21:58:44.4683225Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4683319Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4683621Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4683710Z hidden_states = self.encoder( 2025-08-14T21:58:44.4684000Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4684131Z layer_outputs = layer_module( 2025-08-14T21:58:44.4684374Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4684497Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4684765Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4684866Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4685163Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4685265Z self_outputs = self.self( 2025-08-14T21:58:44.4685572Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T21:58:44.4685802Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T21:58:44.4685811Z 2025-08-14T21:58:44.4701813Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4702201Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4702285Z return mod(**inputs) 2025-08-14T21:58:44.4702568Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4702663Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4702917Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4702989Z hidden_states = self.encoder( 2025-08-14T21:58:44.4703243Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4703311Z layer_outputs = layer_module( 2025-08-14T21:58:44.4703520Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4703691Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4703945Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4704079Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4704325Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4704396Z self_outputs = self.self( 2025-08-14T21:58:44.4704648Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T21:58:44.4704761Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T21:58:44.4704769Z 2025-08-14T21:58:44.4704882Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4705075Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4705138Z return mod(**inputs) 2025-08-14T21:58:44.4705456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4705536Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4705783Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4705859Z hidden_states = self.encoder( 2025-08-14T21:58:44.4706104Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4706177Z layer_outputs = layer_module( 2025-08-14T21:58:44.4706386Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4706458Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4706711Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4706789Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4707040Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4707106Z self_outputs = self.self( 2025-08-14T21:58:44.4707349Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T21:58:44.4707475Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T21:58:44.4707479Z 2025-08-14T21:58:44.4707560Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4707635Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4707739Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4707924Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4707993Z return mod(**inputs) 2025-08-14T21:58:44.4708240Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4708315Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4708564Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4708630Z hidden_states = self.encoder( 2025-08-14T21:58:44.4708871Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4708944Z layer_outputs = layer_module( 2025-08-14T21:58:44.4709148Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4709227Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4709484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4709561Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4709827Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4709984Z self_outputs = self.self( 2025-08-14T21:58:44.4710234Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T21:58:44.4710337Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T21:58:44.4710341Z 2025-08-14T21:58:44.4710438Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4710630Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4710693Z return mod(**inputs) 2025-08-14T21:58:44.4710942Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4711862Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4712137Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4712212Z hidden_states = self.encoder( 2025-08-14T21:58:44.4712456Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4712522Z layer_outputs = layer_module( 2025-08-14T21:58:44.4712734Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4712807Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4713062Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4713137Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4713384Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T21:58:44.4713512Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:58:44.4713756Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T21:58:44.4713841Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4713845Z 2025-08-14T21:58:44.4713941Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4714127Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4714198Z return mod(**inputs) 2025-08-14T21:58:44.4714443Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4714518Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4714770Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4714837Z hidden_states = self.encoder( 2025-08-14T21:58:44.4715085Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4715149Z layer_outputs = layer_module( 2025-08-14T21:58:44.4715351Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4715429Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4715674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4715760Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4716020Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4716094Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4716403Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4716518Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4716762Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T21:58:44.4716848Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4716851Z 2025-08-14T21:58:44.4716945Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4717136Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4717196Z return mod(**inputs) 2025-08-14T21:58:44.4717439Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4717537Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4717799Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4717874Z hidden_states = self.encoder( 2025-08-14T21:58:44.4718130Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4718195Z layer_outputs = layer_module( 2025-08-14T21:58:44.4718413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4718485Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4718741Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4718827Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4719119Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4719200Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4719476Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4719587Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4719850Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T21:58:44.4719957Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:58:44.4720166Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:44.4720232Z return self.act(input) 2025-08-14T21:58:44.4720237Z 2025-08-14T21:58:44.4720335Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4720534Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4720596Z return mod(**inputs) 2025-08-14T21:58:44.4720851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4720933Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4721186Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4721260Z hidden_states = self.encoder( 2025-08-14T21:58:44.4721513Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4721579Z layer_outputs = layer_module( 2025-08-14T21:58:44.4721795Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4721887Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4722141Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4722240Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4722486Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4722565Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4722846Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T21:58:44.4722972Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:58:44.4723232Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T21:58:44.4723311Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4723314Z 2025-08-14T21:58:44.4723450Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4723642Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4723705Z return mod(**inputs) 2025-08-14T21:58:44.4723961Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4724037Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4724295Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4724363Z hidden_states = self.encoder( 2025-08-14T21:58:44.4724614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4724691Z layer_outputs = layer_module( 2025-08-14T21:58:44.4724897Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4724971Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4725226Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4725303Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4725562Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4725712Z self_outputs = self.self( 2025-08-14T21:58:44.4725975Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 350, in forward 2025-08-14T21:58:44.4726073Z mixed_query_layer = self.query(hidden_states) 2025-08-14T21:58:44.4726077Z 2025-08-14T21:58:44.4726178Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4726364Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4726442Z return mod(**inputs) 2025-08-14T21:58:44.4726709Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4726798Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4727071Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4727142Z hidden_states = self.encoder( 2025-08-14T21:58:44.4727405Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4727474Z layer_outputs = layer_module( 2025-08-14T21:58:44.4727697Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4727806Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4728076Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4728183Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4728445Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4728514Z self_outputs = self.self( 2025-08-14T21:58:44.4728782Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 344, in forward 2025-08-14T21:58:44.4728862Z mixed_key_layer = self.key(hidden_states) 2025-08-14T21:58:44.4728865Z 2025-08-14T21:58:44.4728975Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4729172Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4729238Z return mod(**inputs) 2025-08-14T21:58:44.4729538Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4729620Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4729892Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4729961Z hidden_states = self.encoder( 2025-08-14T21:58:44.4730229Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4730302Z layer_outputs = layer_module( 2025-08-14T21:58:44.4730511Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4730582Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4730839Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4730916Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4731176Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4731243Z self_outputs = self.self( 2025-08-14T21:58:44.4731493Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 345, in forward 2025-08-14T21:58:44.4731585Z mixed_value_layer = self.value(hidden_states) 2025-08-14T21:58:44.4731588Z 2025-08-14T21:58:44.4731665Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4731745Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4731842Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4732030Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4732100Z return mod(**inputs) 2025-08-14T21:58:44.4732354Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4732432Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4732693Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4732758Z hidden_states = self.encoder( 2025-08-14T21:58:44.4733016Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4733082Z layer_outputs = layer_module( 2025-08-14T21:58:44.4733293Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4733370Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4733619Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4733710Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4733966Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4734048Z self_outputs = self.self( 2025-08-14T21:58:44.4734307Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 366, in forward 2025-08-14T21:58:44.4734407Z conv_out_layer = self.conv_out_layer(hidden_states) 2025-08-14T21:58:44.4734410Z 2025-08-14T21:58:44.4734485Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4734589Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4734777Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4734846Z return mod(**inputs) 2025-08-14T21:58:44.4735103Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4735196Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4735480Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4735549Z hidden_states = self.encoder( 2025-08-14T21:58:44.4735804Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4735878Z layer_outputs = layer_module( 2025-08-14T21:58:44.4736090Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4736174Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4736430Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4736508Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4736774Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4736841Z self_outputs = self.self( 2025-08-14T21:58:44.4737096Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.4737258Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.4737514Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 282, in forward 2025-08-14T21:58:44.4737596Z x = self.depthwise(hidden_states) 2025-08-14T21:58:44.4737735Z 2025-08-14T21:58:44.4737849Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4738041Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4738117Z return mod(**inputs) 2025-08-14T21:58:44.4738381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4738469Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4738733Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4738804Z hidden_states = self.encoder( 2025-08-14T21:58:44.4739080Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4739152Z layer_outputs = layer_module( 2025-08-14T21:58:44.4739366Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4739450Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4739705Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4739849Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4740111Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4740209Z self_outputs = self.self( 2025-08-14T21:58:44.4740484Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 347, in forward 2025-08-14T21:58:44.4740637Z mixed_key_conv_attn_layer = self.key_conv_attn_layer(hidden_states.transpose(1, 2)) 2025-08-14T21:58:44.4740909Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 283, in forward 2025-08-14T21:58:44.4740979Z x = self.pointwise(x) 2025-08-14T21:58:44.4740983Z 2025-08-14T21:58:44.4741084Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4741289Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4741353Z return mod(**inputs) 2025-08-14T21:58:44.4741668Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4741756Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4742009Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4742085Z hidden_states = self.encoder( 2025-08-14T21:58:44.4742336Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4742403Z layer_outputs = layer_module( 2025-08-14T21:58:44.4742622Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4742697Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4742956Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4743034Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4743286Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4743361Z self_outputs = self.self( 2025-08-14T21:58:44.4743614Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 360, in forward 2025-08-14T21:58:44.4743761Z conv_attn_layer = torch.multiply(mixed_key_conv_attn_layer, mixed_query_layer) 2025-08-14T21:58:44.4743774Z 2025-08-14T21:58:44.4743873Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4744060Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4744135Z return mod(**inputs) 2025-08-14T21:58:44.4744390Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4744469Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4744731Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4744800Z hidden_states = self.encoder( 2025-08-14T21:58:44.4745063Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4745131Z layer_outputs = layer_module( 2025-08-14T21:58:44.4745341Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4745420Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4745674Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4745769Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4746033Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4746116Z self_outputs = self.self( 2025-08-14T21:58:44.4746381Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 362, in forward 2025-08-14T21:58:44.4746499Z conv_kernel_layer = self.conv_kernel_layer(conv_attn_layer) 2025-08-14T21:58:44.4746502Z 2025-08-14T21:58:44.4746602Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4746799Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4746862Z return mod(**inputs) 2025-08-14T21:58:44.4747128Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4747207Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4747504Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4747580Z hidden_states = self.encoder( 2025-08-14T21:58:44.4747851Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4747918Z layer_outputs = layer_module( 2025-08-14T21:58:44.4748132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4748213Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4748469Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4748565Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4748814Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4748884Z self_outputs = self.self( 2025-08-14T21:58:44.4749216Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 380, in forward 2025-08-14T21:58:44.4749336Z conv_out_layer = torch.matmul(conv_out_layer, conv_kernel_layer) 2025-08-14T21:58:44.4749339Z 2025-08-14T21:58:44.4749421Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4749496Z cudagraph partition due to non gpu ops 2025-08-14T21:58:44.4749593Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4749784Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4749847Z return mod(**inputs) 2025-08-14T21:58:44.4750098Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4750182Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4750433Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4750509Z hidden_states = self.encoder( 2025-08-14T21:58:44.4750763Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4750829Z layer_outputs = layer_module( 2025-08-14T21:58:44.4751044Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4751118Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4751369Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4751452Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4751715Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 464, in forward 2025-08-14T21:58:44.4751795Z self_outputs = self.self( 2025-08-14T21:58:44.4752065Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 405, in forward 2025-08-14T21:58:44.4752171Z context_layer = torch.cat([context_layer, conv_out], 2) 2025-08-14T21:58:44.4752175Z 2025-08-14T21:58:44.4752280Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4752468Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4752537Z return mod(**inputs) 2025-08-14T21:58:44.4752788Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4752865Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4753121Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4753202Z hidden_states = self.encoder( 2025-08-14T21:58:44.4753468Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4753544Z layer_outputs = layer_module( 2025-08-14T21:58:44.4753752Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4753833Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4754082Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 561, in forward 2025-08-14T21:58:44.4754156Z self_attention_outputs = self.attention( 2025-08-14T21:58:44.4754413Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 471, in forward 2025-08-14T21:58:44.4754538Z attention_output = self.output(self_outputs[0], hidden_states) 2025-08-14T21:58:44.4754796Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 425, in forward 2025-08-14T21:58:44.4754876Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4754880Z 2025-08-14T21:58:44.4754977Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4755169Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4755230Z return mod(**inputs) 2025-08-14T21:58:44.4755475Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4755556Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4755806Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4755880Z hidden_states = self.encoder( 2025-08-14T21:58:44.4756132Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4756199Z layer_outputs = layer_module( 2025-08-14T21:58:44.4756417Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4756488Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4756740Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4756817Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4757060Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4757140Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4757422Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4757554Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4757861Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 513, in forward 2025-08-14T21:58:44.4757936Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4757939Z 2025-08-14T21:58:44.4758042Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4758228Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4758289Z return mod(**inputs) 2025-08-14T21:58:44.4758546Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4758622Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4758879Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4758948Z hidden_states = self.encoder( 2025-08-14T21:58:44.4759224Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4759302Z layer_outputs = layer_module( 2025-08-14T21:58:44.4759512Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4759584Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4759842Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4759920Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4760172Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4760245Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4760529Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 593, in feed_forward_chunk 2025-08-14T21:58:44.4760652Z intermediate_output = self.intermediate(attention_output) 2025-08-14T21:58:44.4760903Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 514, in forward 2025-08-14T21:58:44.4761020Z hidden_states = self.intermediate_act_fn(hidden_states) 2025-08-14T21:58:44.4761223Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/activations.py", line 69, in forward 2025-08-14T21:58:44.4761291Z return self.act(input) 2025-08-14T21:58:44.4761295Z 2025-08-14T21:58:44.4761403Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4761592Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4761656Z return mod(**inputs) 2025-08-14T21:58:44.4761915Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 925, in forward 2025-08-14T21:58:44.4761994Z generator_hidden_states = self.convbert( 2025-08-14T21:58:44.4762255Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 853, in forward 2025-08-14T21:58:44.4762324Z hidden_states = self.encoder( 2025-08-14T21:58:44.4762579Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 625, in forward 2025-08-14T21:58:44.4762654Z layer_outputs = layer_module( 2025-08-14T21:58:44.4762867Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/modeling_layers.py", line 94, in __call__ 2025-08-14T21:58:44.4762948Z return super().__call__(*args, **kwargs) 2025-08-14T21:58:44.4763207Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 586, in forward 2025-08-14T21:58:44.4763304Z layer_output = apply_chunking_to_forward( 2025-08-14T21:58:44.4763561Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/pytorch_utils.py", line 251, in apply_chunking_to_forward 2025-08-14T21:58:44.4763690Z return forward_fn(*input_tensors) 2025-08-14T21:58:44.4763978Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 594, in feed_forward_chunk 2025-08-14T21:58:44.4764116Z layer_output = self.output(intermediate_output, attention_output) 2025-08-14T21:58:44.4764375Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 531, in forward 2025-08-14T21:58:44.4764461Z hidden_states = self.dense(hidden_states) 2025-08-14T21:58:44.4764464Z 2025-08-14T21:58:44.4764565Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4764759Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4764830Z return mod(**inputs) 2025-08-14T21:58:44.4765123Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 938, in forward 2025-08-14T21:58:44.4765279Z prediction_scores = self.generator_predictions(generator_sequence_output) 2025-08-14T21:58:44.4765534Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 876, in forward 2025-08-14T21:58:44.4765706Z hidden_states = self.dense(generator_hidden_states) 2025-08-14T21:58:44.4765712Z 2025-08-14T21:58:44.4765827Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4766017Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4766082Z return mod(**inputs) 2025-08-14T21:58:44.4766348Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 939, in forward 2025-08-14T21:58:44.4766483Z prediction_scores = self.generator_lm_head(prediction_scores) 2025-08-14T21:58:44.4766490Z 2025-08-14T21:58:44.4766605Z cudagraph partition due to non gpu ops. Found from : 2025-08-14T21:58:44.4766811Z File "/var/lib/jenkins/workspace/benchmarks/dynamo/huggingface.py", line 532, in forward_pass 2025-08-14T21:58:44.4766879Z return mod(**inputs) 2025-08-14T21:58:44.4767178Z File "/opt/conda/envs/py_3.9/lib/python3.9/site-packages/transformers/models/convbert/modeling_convbert.py", line 945, in forward 2025-08-14T21:58:44.4767360Z loss = loss_fct(prediction_scores.view(-1, self.config.vocab_size), labels.view(-1)) 2025-08-14T21:58:44.4767364Z 2025-08-14T21:58:53.3793869Z Compilation time (from dynamo_timed): 19.361040156 2025-08-14T21:58:53.3846184Z pass 2025-08-14T21:58:53.3847054Z WARNING:common:Trying to call the empty_gpu_cache for device: cpu, which is not in list [cuda, xpu] 2025-08-14T21:58:53.3848049Z TIMING: _recursive_pre_grad_passes:0.00955 _recursive_joint_graph_passes:0.58923 _recursive_post_grad_passes:0.17717 async_compile.wait:0.5799 code_gen:8.33475 inductor_compile:10.62431 backend_compile:15.34089 gc:0.00172 entire_frame_compile:19.36104 total_wall_time:19.36104 2025-08-14T21:58:53.3849070Z STATS: call_* op count: 634 | FakeTensorMode.__torch_dispatch__:23085 | FakeTensor.__torch_dispatch__:7564 | ProxyTorchDispatchMode.__torch_dispatch__:8630 2025-08-14T21:58:53.3849609Z Dynamo produced 1 graphs covering 634 ops with 0 graph breaks (0 unique) 2025-08-14T21:58:55.2826156Z accuracy pass_rate=95.35% 2025-08-14T21:58:55.2826497Z calls_captured gmean=0.00x mean=609.233x 2025-08-14T21:58:55.2827912Z unique_graphs gmean=0.00x mean=1.093x 2025-08-14T21:58:55.2828163Z graph_breaks gmean=0.00x mean=0.140x 2025-08-14T21:58:55.2828399Z unique_graph_breaks gmean=0.00x mean=0.047x 2025-08-14T21:58:55.2828629Z autograd_captures gmean=0.00x mean=0.000x 2025-08-14T21:58:55.2833805Z autograd_compiles gmean=0.00x mean=0.000x 2025-08-14T21:58:55.2834725Z cudagraph_skips gmean=0.00x mean=1.093x 2025-08-14T21:58:55.2835061Z compilation_latency mean=19.276 seconds 2025-08-14T21:58:56.1920222Z + python benchmarks/dynamo/check_accuracy.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_cpu_inductor_huggingface_inference.csv 2025-08-14T21:58:56.4549166Z AlbertForMaskedLM PASS 2025-08-14T21:58:56.4549755Z AlbertForQuestionAnswering PASS 2025-08-14T21:58:56.4550537Z AllenaiLongformerBase PASS 2025-08-14T21:58:56.4550810Z BartForCausalLM PASS 2025-08-14T21:58:56.4555945Z BartForConditionalGeneration PASS 2025-08-14T21:58:56.4560732Z BertForMaskedLM PASS 2025-08-14T21:58:56.4561144Z BertForQuestionAnswering PASS 2025-08-14T21:58:56.4561464Z BlenderbotForCausalLM XFAIL 2025-08-14T21:58:56.4561836Z BlenderbotSmallForCausalLM PASS 2025-08-14T21:58:56.4569615Z BlenderbotSmallForConditionalGeneration PASS 2025-08-14T21:58:56.4570409Z CamemBert PASS 2025-08-14T21:58:56.4577470Z DebertaV2ForMaskedLM XFAIL 2025-08-14T21:58:56.4577979Z DebertaV2ForQuestionAnswering PASS 2025-08-14T21:58:56.4588650Z DistilBertForMaskedLM PASS 2025-08-14T21:58:56.4589083Z DistilBertForQuestionAnswering PASS 2025-08-14T21:58:56.4589412Z DistillGPT2 PASS 2025-08-14T21:58:56.4592322Z ElectraForCausalLM PASS 2025-08-14T21:58:56.4598994Z ElectraForQuestionAnswering PASS 2025-08-14T21:58:56.4600779Z GPT2ForSequenceClassification PASS 2025-08-14T21:58:56.4601018Z GoogleFnet PASS 2025-08-14T21:58:56.4601235Z LayoutLMForMaskedLM PASS 2025-08-14T21:58:56.4601466Z LayoutLMForSequenceClassification PASS 2025-08-14T21:58:56.4603917Z M2M100ForConditionalGeneration PASS 2025-08-14T21:58:56.4605350Z MBartForCausalLM PASS 2025-08-14T21:58:56.4614254Z MBartForConditionalGeneration PASS 2025-08-14T21:58:56.4614729Z MT5ForConditionalGeneration PASS 2025-08-14T21:58:56.4618467Z MegatronBertForCausalLM PASS 2025-08-14T21:58:56.4618918Z MegatronBertForQuestionAnswering PASS 2025-08-14T21:58:56.4626529Z MobileBertForMaskedLM PASS 2025-08-14T21:58:56.4626943Z MobileBertForQuestionAnswering PASS 2025-08-14T21:58:56.4635667Z OPTForCausalLM PASS 2025-08-14T21:58:56.4636086Z PLBartForCausalLM PASS 2025-08-14T21:58:56.4640565Z PLBartForConditionalGeneration PASS 2025-08-14T21:58:56.4641038Z PegasusForCausalLM PASS 2025-08-14T21:58:56.4644915Z PegasusForConditionalGeneration PASS 2025-08-14T21:58:56.4645219Z RobertaForCausalLM PASS 2025-08-14T21:58:56.4652397Z RobertaForQuestionAnswering PASS 2025-08-14T21:58:56.4657635Z T5ForConditionalGeneration PASS 2025-08-14T21:58:56.4658080Z T5Small PASS 2025-08-14T21:58:56.4659657Z TrOCRForCausalLM PASS 2025-08-14T21:58:56.4660120Z XGLMForCausalLM PASS 2025-08-14T21:58:56.4666214Z XLNetLMHeadModel PASS 2025-08-14T21:58:56.4669317Z YituTechConvBert PASS 2025-08-14T21:58:56.5185557Z + python benchmarks/dynamo/check_graph_breaks.py --actual /var/lib/jenkins/workspace/test/test-reports/inference_huggingface.csv --expected benchmarks/dynamo/ci_expected_accuracy/dynamic_cpu_inductor_huggingface_inference.csv 2025-08-14T21:58:56.7836318Z AlbertForMaskedLM PASS 2025-08-14T21:58:56.7841803Z AlbertForQuestionAnswering PASS 2025-08-14T21:58:56.7842084Z AllenaiLongformerBase PASS 2025-08-14T21:58:56.7842307Z BartForCausalLM PASS 2025-08-14T21:58:56.7842527Z BartForConditionalGeneration PASS 2025-08-14T21:58:56.7843088Z BertForMaskedLM PASS 2025-08-14T21:58:56.7847428Z BertForQuestionAnswering PASS 2025-08-14T21:58:56.7851123Z BlenderbotForCausalLM PASS 2025-08-14T21:58:56.7861808Z BlenderbotSmallForCausalLM PASS 2025-08-14T21:58:56.7867418Z BlenderbotSmallForConditionalGeneration PASS 2025-08-14T21:58:56.7871795Z CamemBert PASS 2025-08-14T21:58:56.7876586Z DebertaV2ForMaskedLM PASS 2025-08-14T21:58:56.7878786Z DebertaV2ForQuestionAnswering PASS 2025-08-14T21:58:56.7879258Z DistilBertForMaskedLM PASS 2025-08-14T21:58:56.7879624Z DistilBertForQuestionAnswering PASS 2025-08-14T21:58:56.7880016Z DistillGPT2 PASS 2025-08-14T21:58:56.7880390Z ElectraForCausalLM PASS 2025-08-14T21:58:56.7882349Z ElectraForQuestionAnswering PASS 2025-08-14T21:58:56.7890827Z GPT2ForSequenceClassification PASS 2025-08-14T21:58:56.7893271Z GoogleFnet PASS 2025-08-14T21:58:56.7893874Z LayoutLMForMaskedLM PASS 2025-08-14T21:58:56.7894145Z LayoutLMForSequenceClassification PASS 2025-08-14T21:58:56.7903658Z M2M100ForConditionalGeneration PASS 2025-08-14T21:58:56.7907744Z MBartForCausalLM PASS 2025-08-14T21:58:56.7912074Z MBartForConditionalGeneration PASS 2025-08-14T21:58:56.7914108Z MT5ForConditionalGeneration PASS 2025-08-14T21:58:56.7914475Z MegatronBertForCausalLM PASS 2025-08-14T21:58:56.7919352Z MegatronBertForQuestionAnswering PASS 2025-08-14T21:58:56.7921267Z MobileBertForMaskedLM PASS 2025-08-14T21:58:56.7921702Z MobileBertForQuestionAnswering PASS 2025-08-14T21:58:56.7922015Z OPTForCausalLM PASS 2025-08-14T21:58:56.7922314Z PLBartForCausalLM PASS 2025-08-14T21:58:56.7926729Z PLBartForConditionalGeneration PASS 2025-08-14T21:58:56.7931709Z PegasusForCausalLM PASS 2025-08-14T21:58:56.7940331Z PegasusForConditionalGeneration PASS 2025-08-14T21:58:56.7944287Z RobertaForCausalLM PASS 2025-08-14T21:58:56.7944566Z RobertaForQuestionAnswering PASS 2025-08-14T21:58:56.7949754Z T5ForConditionalGeneration PASS 2025-08-14T21:58:56.7954684Z T5Small PASS 2025-08-14T21:58:56.7956370Z TrOCRForCausalLM PASS 2025-08-14T21:58:56.7956697Z XGLMForCausalLM PASS_BUT_FLAKY 2025-08-14T21:58:56.7956938Z XLNetLMHeadModel PASS 2025-08-14T21:58:56.7957206Z YituTechConvBert PASS 2025-08-14T21:58:56.8472712Z + sccache_epilogue 2025-08-14T21:58:56.8474438Z + echo '::group::Sccache Compilation Log' 2025-08-14T21:58:56.8475189Z ##[group]Sccache Compilation Log 2025-08-14T21:58:56.8479998Z + echo '=================== sccache compilation log ===================' 2025-08-14T21:58:56.8480320Z =================== sccache compilation log =================== 2025-08-14T21:58:56.8480712Z + python /var/lib/jenkins/workspace/.ci/pytorch/print_sccache_log.py /var/lib/jenkins/sccache_error.log 2025-08-14T21:58:56.8679994Z + echo '=========== If your build fails, please take a look at the log above for possible reasons ===========' 2025-08-14T21:58:56.8680610Z =========== If your build fails, please take a look at the log above for possible reasons =========== 2025-08-14T21:58:56.8681549Z + sccache --show-stats 2025-08-14T21:58:56.8726044Z Compile requests 381 2025-08-14T21:58:56.8726334Z Compile requests executed 0 2025-08-14T21:58:56.8726551Z Cache hits 0 2025-08-14T21:58:56.8726775Z Cache misses 0 2025-08-14T21:58:56.8726976Z Cache hits rate - 2025-08-14T21:58:56.8727172Z Cache timeouts 0 2025-08-14T21:58:56.8727379Z Cache read errors 0 2025-08-14T21:58:56.8727559Z Forced recaches 0 2025-08-14T21:58:56.8728024Z Cache write errors 0 2025-08-14T21:58:56.8728220Z Cache errors 0 2025-08-14T21:58:56.8728412Z Compilations 0 2025-08-14T21:58:56.8728623Z Compilation failures 0 2025-08-14T21:58:56.8728893Z Non-cacheable compilations 0 2025-08-14T21:58:56.8729086Z Non-cacheable calls 41 2025-08-14T21:58:56.8729288Z Non-compilation calls 340 2025-08-14T21:58:56.8729492Z Unsupported compiler calls 0 2025-08-14T21:58:56.8729691Z Average cache write 0.000 s 2025-08-14T21:58:56.8729915Z Average compiler 0.000 s 2025-08-14T21:58:56.8730121Z Average cache read hit 0.000 s 2025-08-14T21:58:56.8730329Z Failed distributed compilations 0 2025-08-14T21:58:56.8730460Z 2025-08-14T21:58:56.8730530Z Non-cacheable reasons: 2025-08-14T21:58:56.8730705Z -E 41 2025-08-14T21:58:56.8730833Z 2025-08-14T21:58:56.8730993Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-08-14T21:58:56.8731266Z Version (client) 0.10.0 2025-08-14T21:58:56.8731465Z + sccache --stop-server 2025-08-14T21:58:56.8736202Z Stopping sccache server... 2025-08-14T21:58:56.8742190Z Compile requests 381 2025-08-14T21:58:56.8742495Z Compile requests executed 0 2025-08-14T21:58:56.8745637Z Cache hits 0 2025-08-14T21:58:56.8745994Z Cache misses 0 2025-08-14T21:58:56.8750557Z Cache hits rate - 2025-08-14T21:58:56.8753096Z Cache timeouts 0 2025-08-14T21:58:56.8753336Z Cache read errors 0 2025-08-14T21:58:56.8753565Z Forced recaches 0 2025-08-14T21:58:56.8753755Z Cache write errors 0 2025-08-14T21:58:56.8753939Z Cache errors 0 2025-08-14T21:58:56.8754116Z Compilations 0 2025-08-14T21:58:56.8754319Z Compilation failures 0 2025-08-14T21:58:56.8754513Z Non-cacheable compilations 0 2025-08-14T21:58:56.8754695Z Non-cacheable calls 41 2025-08-14T21:58:56.8754892Z Non-compilation calls 340 2025-08-14T21:58:56.8755088Z Unsupported compiler calls 0 2025-08-14T21:58:56.8755284Z Average cache write 0.000 s 2025-08-14T21:58:56.8755488Z Average compiler 0.000 s 2025-08-14T21:58:56.8755695Z Average cache read hit 0.000 s 2025-08-14T21:58:56.8755906Z Failed distributed compilations 0 2025-08-14T21:58:56.8756038Z 2025-08-14T21:58:56.8756109Z Non-cacheable reasons: 2025-08-14T21:58:56.8756292Z -E 41 2025-08-14T21:58:56.8756417Z 2025-08-14T21:58:56.8756582Z Cache location s3, name: ossci-compiler-cache-circleci-v2, prefix: / 2025-08-14T21:58:56.8756862Z Version (client) 0.10.0 2025-08-14T21:58:56.8757102Z + echo ::endgroup:: 2025-08-14T21:58:56.8757500Z ##[endgroup] 2025-08-14T21:58:56.8757656Z + cleanup_workspace 2025-08-14T21:58:56.8757958Z + echo 'sudo may print the following warning message that can be ignored. The chown command will still run.' 2025-08-14T21:58:56.8758374Z sudo may print the following warning message that can be ignored. The chown command will still run. 2025-08-14T21:58:56.8758734Z + echo ' sudo: setrlimit(RLIMIT_STACK): Operation not permitted' 2025-08-14T21:58:56.8759010Z sudo: setrlimit(RLIMIT_STACK): Operation not permitted 2025-08-14T21:58:56.8759330Z + echo 'For more details refer to https://github.com/sudo-project/sudo/issues/42' 2025-08-14T21:58:56.8759664Z For more details refer to https://github.com/sudo-project/sudo/issues/42 2025-08-14T21:58:56.8759937Z + sudo chown -R 1000 /var/lib/jenkins/workspace 2025-08-14T21:58:57.2862344Z ##[group]Run pytorch/test-infra/.github/actions/upload-benchmark-results@main 2025-08-14T21:58:57.2862667Z with: 2025-08-14T21:58:57.2862866Z benchmark-results-dir: test/test-reports 2025-08-14T21:58:57.2863126Z dry-run: false 2025-08-14T21:58:57.2863283Z schema-version: v3 2025-08-14T21:58:57.2863640Z github-token: *** 2025-08-14T21:58:57.2863788Z env: 2025-08-14T21:58:57.2863981Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:58:57.2864264Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:58:57.2864557Z ##[endgroup] 2025-08-14T21:58:57.2875620Z ##[group]Run set -eux 2025-08-14T21:58:57.2875824Z set -eux 2025-08-14T21:58:57.2876051Z python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0 2025-08-14T21:58:57.2876300Z  2025-08-14T21:58:57.2876453Z DEVICE_NAME="" 2025-08-14T21:58:57.2876612Z DEVICE_TYPE="" 2025-08-14T21:58:57.2876767Z  2025-08-14T21:58:57.2876952Z if command -v nvidia-smi; then 2025-08-14T21:58:57.2877214Z  # NB: I'm using PyTorch here to get the device name, however, it needs to 2025-08-14T21:58:57.2877530Z  # install the correct version of PyTorch manually for now. Any PyTorch 2025-08-14T21:58:57.2877831Z  # version is fine, I just use 2.7.1 to satify PYPIDEP linter 2025-08-14T21:58:57.2878107Z  python3 -mpip install torch==2.7.1 2025-08-14T21:58:57.2878352Z elif command -v rocminfo; then 2025-08-14T21:58:57.2878611Z  # NB: Installing torch on ROCm runner with pip here causes CI to fail 2025-08-14T21:58:57.2878931Z  # with a memoryview is too large error only on MI300 runners. Is pip 2025-08-14T21:58:57.2879252Z  # version on ROCm runner there too old? As a workaround, let's use the 2025-08-14T21:58:57.2879535Z  # GPU device name coming from rocminfo instead 2025-08-14T21:58:57.2879756Z  DEVICE_NAME=rocm 2025-08-14T21:58:57.2880052Z  DEVICE_TYPE=$(rocminfo | grep "Marketing Name" | tail -n1 | awk -F':' '{print $2}' | xargs) 2025-08-14T21:58:57.2880345Z fi 2025-08-14T21:58:57.2880485Z  2025-08-14T21:58:57.2880668Z echo "DEVICE_NAME=$DEVICE_NAME" >> $GITHUB_ENV 2025-08-14T21:58:57.2880916Z echo "DEVICE_TYPE=$DEVICE_TYPE" >> $GITHUB_ENV 2025-08-14T21:58:57.2888868Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:58:57.2889121Z env: 2025-08-14T21:58:57.2889286Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:58:57.2889580Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:58:57.2889886Z ##[endgroup] 2025-08-14T21:58:57.2920679Z + python3 -mpip install boto3==1.35.33 psutil==7.0.0 pynvml==12.0.0 2025-08-14T21:58:57.4606927Z Defaulting to user installation because normal site-packages is not writeable 2025-08-14T21:58:58.1947118Z Collecting boto3==1.35.33 2025-08-14T21:58:58.2120024Z Downloading boto3-1.35.33-py3-none-any.whl (139 kB) 2025-08-14T21:58:58.4293602Z Collecting psutil==7.0.0 2025-08-14T21:58:58.4331692Z Downloading psutil-7.0.0-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (277 kB) 2025-08-14T21:58:58.4611429Z Collecting pynvml==12.0.0 2025-08-14T21:58:58.4646910Z Downloading pynvml-12.0.0-py3-none-any.whl (26 kB) 2025-08-14T21:58:58.5017055Z Collecting s3transfer<0.11.0,>=0.10.0 2025-08-14T21:58:58.5050490Z Downloading s3transfer-0.10.4-py3-none-any.whl (83 kB) 2025-08-14T21:58:59.2692302Z Collecting botocore<1.36.0,>=1.35.33 2025-08-14T21:58:59.2736455Z Downloading botocore-1.35.99-py3-none-any.whl (13.3 MB) 2025-08-14T21:58:59.3635470Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.33) (0.10.0) 2025-08-14T21:58:59.3964167Z Collecting nvidia-ml-py<13.0.0a0,>=12.0.0 2025-08-14T21:58:59.4001471Z Downloading nvidia_ml_py-12.575.51-py3-none-any.whl (47 kB) 2025-08-14T21:58:59.4082625Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (2.8.1) 2025-08-14T21:58:59.4090273Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.25.10) 2025-08-14T21:58:59.5123539Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.36.0,>=1.35.33->boto3==1.35.33) (1.15.0) 2025-08-14T21:58:59.6115598Z Installing collected packages: botocore, s3transfer, nvidia-ml-py, pynvml, psutil, boto3 2025-08-14T21:58:59.9445883Z Attempting uninstall: nvidia-ml-py 2025-08-14T21:58:59.9450349Z Found existing installation: nvidia-ml-py 11.525.84 2025-08-14T21:58:59.9455854Z Uninstalling nvidia-ml-py-11.525.84: 2025-08-14T21:58:59.9579249Z Successfully uninstalled nvidia-ml-py-11.525.84 2025-08-14T21:59:00.0071733Z Attempting uninstall: psutil 2025-08-14T21:59:00.0074381Z Found existing installation: psutil 5.9.8 2025-08-14T21:59:00.0117155Z Uninstalling psutil-5.9.8: 2025-08-14T21:59:00.0119171Z Successfully uninstalled psutil-5.9.8 2025-08-14T21:59:00.1446637Z Successfully installed boto3-1.35.33 botocore-1.35.99 nvidia-ml-py-12.575.51 psutil-7.0.0 pynvml-12.0.0 s3transfer-0.10.4 2025-08-14T21:59:00.2442687Z + DEVICE_NAME= 2025-08-14T21:59:00.2446341Z + DEVICE_TYPE= 2025-08-14T21:59:00.2446601Z + command -v nvidia-smi 2025-08-14T21:59:00.2451734Z + command -v rocminfo 2025-08-14T21:59:00.2454048Z + echo DEVICE_NAME= 2025-08-14T21:59:00.2454249Z + echo DEVICE_TYPE= 2025-08-14T21:59:00.2467986Z ##[group]Run set -eux 2025-08-14T21:59:00.2468172Z set -eux 2025-08-14T21:59:00.2468316Z  2025-08-14T21:59:00.2468482Z if [[ -z "${GITHUB_TOKEN}" ]]; then 2025-08-14T21:59:00.2468705Z  echo "Missing github-token input" 2025-08-14T21:59:00.2468896Z  exit 1 2025-08-14T21:59:00.2469064Z fi 2025-08-14T21:59:00.2473565Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:59:00.2473779Z env: 2025-08-14T21:59:00.2473922Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:00.2474213Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:00.2474503Z DEVICE_NAME: 2025-08-14T21:59:00.2474641Z DEVICE_TYPE: 2025-08-14T21:59:00.2475009Z GITHUB_TOKEN: *** 2025-08-14T21:59:00.2475163Z ##[endgroup] 2025-08-14T21:59:00.2499177Z + [[ -z *** ]] 2025-08-14T21:59:00.2528433Z ##[group]Run pytorch/test-infra/.github/actions/get-workflow-job-id@main 2025-08-14T21:59:00.2528692Z with: 2025-08-14T21:59:00.2528978Z github-token: *** 2025-08-14T21:59:00.2529149Z env: 2025-08-14T21:59:00.2529311Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:00.2529631Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:00.2529963Z DEVICE_NAME: 2025-08-14T21:59:00.2530128Z DEVICE_TYPE: 2025-08-14T21:59:00.2530310Z ##[endgroup] 2025-08-14T21:59:00.2539865Z ##[group]Run set -eux 2025-08-14T21:59:00.2540056Z set -eux 2025-08-14T21:59:00.2540214Z  2025-08-14T21:59:00.2540536Z python3 "${GITHUB_ACTION_PATH}/../../scripts/get_workflow_job_id.py" "${GITHUB_RUN_ID}" "${RUNNER_NAME}" 2025-08-14T21:59:00.2544613Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:59:00.2544865Z env: 2025-08-14T21:59:00.2545026Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:00.2545323Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:00.2545641Z DEVICE_NAME: 2025-08-14T21:59:00.2545785Z DEVICE_TYPE: 2025-08-14T21:59:00.2546040Z GITHUB_TOKEN: *** 2025-08-14T21:59:00.2546192Z ##[endgroup] 2025-08-14T21:59:00.2566268Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/get-workflow-job-id/../../scripts/get_workflow_job_id.py 16976338999 i-0851ccaad4f014969 2025-08-14T21:59:01.8438109Z setting job-id=48128301909 2025-08-14T21:59:01.8439151Z setting job-name=linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T21:59:01.8533811Z ##[group]Run set -eux 2025-08-14T21:59:01.8534020Z set -eux 2025-08-14T21:59:01.8534181Z  2025-08-14T21:59:01.8534442Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_metadata.py" \ 2025-08-14T21:59:01.8534847Z  --schema-version "${SCHEMA_VERSION}" \ 2025-08-14T21:59:01.8535073Z  --repo "${REPO}" \ 2025-08-14T21:59:01.8535278Z  --head-branch "${HEAD_BRANCH}" \ 2025-08-14T21:59:01.8535488Z  --head-sha "${HEAD_SHA}" \ 2025-08-14T21:59:01.8535711Z  --workflow-id "${WORKFLOW_RUN_ID}" \ 2025-08-14T21:59:01.8535936Z  --run-attempt "${RUN_ATTEMPT}" \ 2025-08-14T21:59:01.8536145Z  --job-id "${JOB_ID}" \ 2025-08-14T21:59:01.8536353Z  --job-name "${JOB_NAME}" 2025-08-14T21:59:01.8541749Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:59:01.8542002Z env: 2025-08-14T21:59:01.8542156Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:01.8542478Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:01.8542795Z DEVICE_NAME: 2025-08-14T21:59:01.8542948Z DEVICE_TYPE: 2025-08-14T21:59:01.8543116Z SCHEMA_VERSION: v3 2025-08-14T21:59:01.8543294Z REPO: pytorch/pytorch 2025-08-14T21:59:01.8543471Z HEAD_BRANCH: refs/heads/main 2025-08-14T21:59:01.8543690Z HEAD_SHA: 1fc683cf17c8c673044538d10266c00f92987be2 2025-08-14T21:59:01.8543910Z WORKFLOW_RUN_ID: 16976338999 2025-08-14T21:59:01.8544085Z RUN_ATTEMPT: 1 2025-08-14T21:59:01.8544239Z JOB_ID: 48128301909 2025-08-14T21:59:01.8544649Z JOB_NAME: linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T21:59:01.8545074Z ##[endgroup] 2025-08-14T21:59:01.8577809Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_metadata.py --schema-version v3 --repo pytorch/pytorch --head-branch refs/heads/main --head-sha 1fc683cf17c8c673044538d10266c00f92987be2 --workflow-id 16976338999 --run-attempt 1 --job-id 48128301909 --job-name 'linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)' 2025-08-14T21:59:01.8846193Z ##[group]Run set -eux 2025-08-14T21:59:01.8846401Z set -eux 2025-08-14T21:59:01.8846559Z  2025-08-14T21:59:01.8846815Z python3 "${GITHUB_ACTION_PATH}/../../scripts/benchmarks/gather_runners_info.py" 2025-08-14T21:59:01.8851541Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:59:01.8851777Z env: 2025-08-14T21:59:01.8851934Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:01.8852241Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:01.8852548Z DEVICE_NAME: 2025-08-14T21:59:01.8852703Z DEVICE_TYPE: 2025-08-14T21:59:01.8852858Z ##[endgroup] 2025-08-14T21:59:01.8874433Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/benchmarks/gather_runners_info.py 2025-08-14T21:59:01.9197904Z INFO:root:Fail to import torch to get the device name 2025-08-14T21:59:01.9289967Z ##[group]Run set -eux 2025-08-14T21:59:01.9290171Z set -eux 2025-08-14T21:59:01.9290361Z  2025-08-14T21:59:01.9290535Z # TODO (huydhn): Implement this part 2025-08-14T21:59:01.9290777Z echo "dependencies={}" >> "${GITHUB_OUTPUT}" 2025-08-14T21:59:01.9295173Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:59:01.9295397Z env: 2025-08-14T21:59:01.9295553Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:01.9295978Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:01.9296280Z DEVICE_NAME: 2025-08-14T21:59:01.9296427Z DEVICE_TYPE: 2025-08-14T21:59:01.9296579Z ##[endgroup] 2025-08-14T21:59:01.9320935Z + echo 'dependencies={}' 2025-08-14T21:59:01.9339727Z ##[group]Run set -eux 2025-08-14T21:59:01.9339931Z set -eux 2025-08-14T21:59:01.9340089Z  2025-08-14T21:59:01.9340276Z if [[ ! -d "${BENCHMARK_RESULTS_DIR}" ]]; then 2025-08-14T21:59:01.9340645Z  echo "${BENCHMARK_RESULTS_DIR} does not exist, skipping" 2025-08-14T21:59:01.9340942Z  # We don't want the job to fail if the directory doesn't exist 2025-08-14T21:59:01.9341190Z  exit 0 2025-08-14T21:59:01.9341343Z fi 2025-08-14T21:59:01.9341484Z  2025-08-14T21:59:01.9341653Z if [[ "${DRY_RUN}" == "true" ]]; then 2025-08-14T21:59:01.9341957Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-08-14T21:59:01.9342313Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-08-14T21:59:01.9342585Z  --metadata "${BENCHMARK_METADATA}" \ 2025-08-14T21:59:01.9342814Z  --runners "${RUNNER_INFO}" \ 2025-08-14T21:59:01.9343040Z  --dependencies "${DEPENDENCIES}" \ 2025-08-14T21:59:01.9343247Z  --dry-run 2025-08-14T21:59:01.9343416Z else 2025-08-14T21:59:01.9343659Z  python3 "${GITHUB_ACTION_PATH}/../../scripts/upload_benchmark_results.py" \ 2025-08-14T21:59:01.9343990Z  --benchmark-results-dir "${BENCHMARK_RESULTS_DIR}" \ 2025-08-14T21:59:01.9344252Z  --metadata "${BENCHMARK_METADATA}" \ 2025-08-14T21:59:01.9344472Z  --runners "${RUNNER_INFO}" \ 2025-08-14T21:59:01.9344692Z  --dependencies "${DEPENDENCIES}" 2025-08-14T21:59:01.9344886Z fi 2025-08-14T21:59:01.9348977Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:59:01.9349212Z env: 2025-08-14T21:59:01.9349369Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:01.9349677Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:01.9349993Z DEVICE_NAME: 2025-08-14T21:59:01.9350141Z DEVICE_TYPE: 2025-08-14T21:59:01.9350315Z BENCHMARK_RESULTS_DIR: test/test-reports 2025-08-14T21:59:01.9350514Z DRY_RUN: false 2025-08-14T21:59:01.9351368Z BENCHMARK_METADATA: {"timestamp": 1755208741, "schema_version": "v3", "name": "linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "1fc683cf17c8c673044538d10266c00f92987be2", "workflow_id": 16976338999, "run_attempt": 1, "job_id": 48128301909} 2025-08-14T21:59:01.9352432Z RUNNER_INFO: [{"cpu_info": "x86_64", "cpu_count": 32, "avail_mem_in_gb": 123, "extra_info": {"hostname": "ip-10-0-8-108.ec2.internal"}, "name": "", "type": ""}] 2025-08-14T21:59:01.9352792Z DEPENDENCIES: {} 2025-08-14T21:59:01.9352959Z ##[endgroup] 2025-08-14T21:59:01.9373428Z + [[ ! -d test/test-reports ]] 2025-08-14T21:59:01.9373733Z + [[ false == \t\r\u\e ]] 2025-08-14T21:59:01.9375538Z + python3 /home/ec2-user/actions-runner/_work/_actions/pytorch/test-infra/main/.github/actions/upload-benchmark-results/../../scripts/upload_benchmark_results.py --benchmark-results-dir test/test-reports --metadata '{"timestamp": 1755208741, "schema_version": "v3", "name": "linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)", "repo": "pytorch/pytorch", "head_branch": "refs/heads/main", "head_sha": "1fc683cf17c8c673044538d10266c00f92987be2", "workflow_id": 16976338999, "run_attempt": 1, "job_id": 48128301909}' --runners '[{"cpu_info": "x86_64", "cpu_count": 32, "avail_mem_in_gb": 123, "extra_info": {"hostname": "ip-10-0-8-108.ec2.internal"}, "name": "", "type": ""}]' --dependencies '{}' 2025-08-14T21:59:02.0566052Z INFO:root:Upload test/test-reports/inference_huggingface.json to s3://ossci-benchmarks/v3/pytorch/pytorch/16976338999/48128301909/inference_huggingface.json 2025-08-14T21:59:02.0855756Z INFO:botocore.credentials:Found credentials from IAM Role: gh-ci-github-action-runners-runner-role 2025-08-14T21:59:02.2983277Z ##[group]Run cat test/**/*_toprint.log || true 2025-08-14T21:59:02.2983548Z cat test/**/*_toprint.log || true 2025-08-14T21:59:02.2988796Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:59:02.2989104Z env: 2025-08-14T21:59:02.2989257Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:02.2989538Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:02.2989844Z DEVICE_NAME: 2025-08-14T21:59:02.2990000Z DEVICE_TYPE: 2025-08-14T21:59:02.2990148Z ##[endgroup] 2025-08-14T21:59:02.3064420Z cat: 'test/**/*_toprint.log': No such file or directory 2025-08-14T21:59:02.3111256Z ##[group]Run kill "$MONITOR_SCRIPT_PID" 2025-08-14T21:59:02.3111518Z kill "$MONITOR_SCRIPT_PID" 2025-08-14T21:59:02.3116060Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:59:02.3116301Z env: 2025-08-14T21:59:02.3116463Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:02.3116767Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:02.3117078Z DEVICE_NAME: 2025-08-14T21:59:02.3117234Z DEVICE_TYPE: 2025-08-14T21:59:02.3117396Z MONITOR_SCRIPT_PID: 47819 2025-08-14T21:59:02.3117573Z ##[endgroup] 2025-08-14T21:59:02.3216069Z Prepare all required actions 2025-08-14T21:59:02.3216414Z Getting action download info 2025-08-14T21:59:02.4664455Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a) 2025-08-14T21:59:02.7652776Z Download action repository 'actions/upload-artifact@v4' (SHA:ea165f8d65b6e75b540449e92b4886f43607fa02) 2025-08-14T21:59:03.1879618Z ##[group]Run ./.github/actions/upload-test-artifacts 2025-08-14T21:59:03.1879901Z with: 2025-08-14T21:59:03.1880218Z file-suffix: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301909 2025-08-14T21:59:03.1880580Z s3-bucket: gha-artifacts 2025-08-14T21:59:03.1880781Z env: 2025-08-14T21:59:03.1880966Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:03.1881326Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:03.1881696Z DEVICE_NAME: 2025-08-14T21:59:03.1881868Z DEVICE_TYPE: 2025-08-14T21:59:03.1882034Z ##[endgroup] 2025-08-14T21:59:03.1900327Z ##[group]Run # Remove any previous test jsons if they exist 2025-08-14T21:59:03.1900659Z # Remove any previous test jsons if they exist 2025-08-14T21:59:03.1900904Z rm -f test-jsons-*.zip 2025-08-14T21:59:03.1901194Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test/test-reports -i '*.json' 2025-08-14T21:59:03.1905779Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:59:03.1906012Z env: 2025-08-14T21:59:03.1906169Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:03.1906470Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:03.1906783Z DEVICE_NAME: 2025-08-14T21:59:03.1906951Z DEVICE_TYPE: 2025-08-14T21:59:03.1907242Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301909 2025-08-14T21:59:03.1907557Z ##[endgroup] 2025-08-14T21:59:03.2095098Z adding: test/test-reports/inference_huggingface.json (deflated 99%) 2025-08-14T21:59:03.2115854Z ##[group]Run # Remove any previous test reports if they exist 2025-08-14T21:59:03.2116156Z # Remove any previous test reports if they exist 2025-08-14T21:59:03.2116392Z rm -f test-reports-*.zip 2025-08-14T21:59:03.2116677Z zip -r "test-reports-${FILE_SUFFIX}.zip" test/test-reports -i '*.xml' -i '*.csv' 2025-08-14T21:59:03.2120864Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:59:03.2121110Z env: 2025-08-14T21:59:03.2121270Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:03.2121570Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:03.2121878Z DEVICE_NAME: 2025-08-14T21:59:03.2122036Z DEVICE_TYPE: 2025-08-14T21:59:03.2122300Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301909 2025-08-14T21:59:03.2122678Z ##[endgroup] 2025-08-14T21:59:03.2162988Z adding: test/test-reports/inference_huggingface.csv (deflated 69%) 2025-08-14T21:59:03.2163637Z adding: test/test-reports/inference_huggingface_graph_breaks.csv (deflated 85%) 2025-08-14T21:59:03.2164043Z adding: test/test-reports/inference_huggingface_graph_break_deduped.csv (deflated 64%) 2025-08-14T21:59:03.2183637Z ##[group]Run # Remove any previous usage logs if they exist 2025-08-14T21:59:03.2183960Z # Remove any previous usage logs if they exist 2025-08-14T21:59:03.2184194Z rm -f logs-*.zip 2025-08-14T21:59:03.2184410Z zip "logs-${FILE_SUFFIX}.zip" 'usage_log.txt' || true 2025-08-14T21:59:03.2184710Z zip -r "logs-${FILE_SUFFIX}.zip" test/test-reports -i '*.log' || true 2025-08-14T21:59:03.2188693Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:59:03.2188918Z env: 2025-08-14T21:59:03.2189082Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:03.2189385Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:03.2189697Z DEVICE_NAME: 2025-08-14T21:59:03.2189859Z DEVICE_TYPE: 2025-08-14T21:59:03.2190274Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301909 2025-08-14T21:59:03.2190571Z ##[endgroup] 2025-08-14T21:59:03.2290174Z adding: usage_log.txt (deflated 96%) 2025-08-14T21:59:03.2298663Z 2025-08-14T21:59:03.2299012Z zip error: Nothing to do! (logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301909.zip) 2025-08-14T21:59:03.2318204Z ##[group]Run # Remove any previous debugging artifacts if they exist 2025-08-14T21:59:03.2318545Z # Remove any previous debugging artifacts if they exist 2025-08-14T21:59:03.2318820Z rm -f debug-*.zip 2025-08-14T21:59:03.2319005Z if [ -d 'test/debug' ]; then 2025-08-14T21:59:03.2319232Z  zip -r "debug-${FILE_SUFFIX}.zip" test/debug 2025-08-14T21:59:03.2319458Z fi 2025-08-14T21:59:03.2323698Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:59:03.2323928Z env: 2025-08-14T21:59:03.2324087Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:03.2324427Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:03.2324763Z DEVICE_NAME: 2025-08-14T21:59:03.2324922Z DEVICE_TYPE: 2025-08-14T21:59:03.2325193Z FILE_SUFFIX: test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301909 2025-08-14T21:59:03.2325478Z ##[endgroup] 2025-08-14T21:59:03.2387583Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-08-14T21:59:03.2387789Z with: 2025-08-14T21:59:03.2387941Z s3-bucket: gha-artifacts 2025-08-14T21:59:03.2388147Z s3-prefix: pytorch/pytorch/16976338999/1/artifact 2025-08-14T21:59:03.2388354Z retention-days: 14 2025-08-14T21:59:03.2388516Z if-no-files-found: warn 2025-08-14T21:59:03.2388691Z path: test-jsons-*.zip 2025-08-14T21:59:03.2388854Z name: artifact 2025-08-14T21:59:03.2389006Z region: us-east-1 2025-08-14T21:59:03.2389154Z env: 2025-08-14T21:59:03.2389289Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:03.2389580Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:03.2389876Z DEVICE_NAME: 2025-08-14T21:59:03.2390026Z DEVICE_TYPE: 2025-08-14T21:59:03.2390163Z ##[endgroup] 2025-08-14T21:59:03.5073995Z NOTE: s3-prefix specified, ignoring name parameter 2025-08-14T21:59:03.5074487Z With the provided path, there will be 1 file uploaded 2025-08-14T21:59:03.5074775Z Uploading to s3 prefix: pytorch/pytorch/16976338999/1/artifact 2025-08-14T21:59:03.5102405Z Starting upload of test-jsons-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301909.zip 2025-08-14T21:59:03.6269960Z Finished upload of test-jsons-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301909.zip 2025-08-14T21:59:03.6420302Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-08-14T21:59:03.6420600Z with: 2025-08-14T21:59:03.6420759Z s3-bucket: gha-artifacts 2025-08-14T21:59:03.6420965Z s3-prefix: pytorch/pytorch/16976338999/1/artifact 2025-08-14T21:59:03.6421173Z retention-days: 14 2025-08-14T21:59:03.6421413Z if-no-files-found: error 2025-08-14T21:59:03.6421593Z path: test-reports-*.zip 2025-08-14T21:59:03.6421749Z name: artifact 2025-08-14T21:59:03.6421902Z region: us-east-1 2025-08-14T21:59:03.6422050Z env: 2025-08-14T21:59:03.6422183Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:03.6422469Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:03.6422762Z DEVICE_NAME: 2025-08-14T21:59:03.6422908Z DEVICE_TYPE: 2025-08-14T21:59:03.6423048Z ##[endgroup] 2025-08-14T21:59:03.8835775Z NOTE: s3-prefix specified, ignoring name parameter 2025-08-14T21:59:03.8838161Z With the provided path, there will be 1 file uploaded 2025-08-14T21:59:03.8838448Z Uploading to s3 prefix: pytorch/pytorch/16976338999/1/artifact 2025-08-14T21:59:03.8865137Z Starting upload of test-reports-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301909.zip 2025-08-14T21:59:04.0041470Z Finished upload of test-reports-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301909.zip 2025-08-14T21:59:04.0182466Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-08-14T21:59:04.0182740Z with: 2025-08-14T21:59:04.0182998Z s3-bucket: gha-artifacts 2025-08-14T21:59:04.0200371Z s3-prefix: pytorch/pytorch/16976338999/1/artifact 2025-08-14T21:59:04.0200726Z retention-days: 14 2025-08-14T21:59:04.0200899Z if-no-files-found: ignore 2025-08-14T21:59:04.0201081Z path: logs-*.zip 2025-08-14T21:59:04.0201247Z name: artifact 2025-08-14T21:59:04.0201408Z region: us-east-1 2025-08-14T21:59:04.0201580Z env: 2025-08-14T21:59:04.0201740Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:04.0202051Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:04.0202381Z DEVICE_NAME: 2025-08-14T21:59:04.0202556Z DEVICE_TYPE: 2025-08-14T21:59:04.0202714Z ##[endgroup] 2025-08-14T21:59:04.2667054Z NOTE: s3-prefix specified, ignoring name parameter 2025-08-14T21:59:04.2667561Z With the provided path, there will be 1 file uploaded 2025-08-14T21:59:04.2668023Z Uploading to s3 prefix: pytorch/pytorch/16976338999/1/artifact 2025-08-14T21:59:04.2696758Z Starting upload of logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301909.zip 2025-08-14T21:59:04.3818022Z Finished upload of logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301909.zip 2025-08-14T21:59:04.4019163Z ##[group]Run seemethere/upload-artifact-s3@v5 2025-08-14T21:59:04.4019379Z with: 2025-08-14T21:59:04.4019537Z s3-bucket: gha-artifacts 2025-08-14T21:59:04.4019748Z s3-prefix: pytorch/pytorch/16976338999/1/artifact 2025-08-14T21:59:04.4019963Z retention-days: 14 2025-08-14T21:59:04.4020125Z if-no-files-found: ignore 2025-08-14T21:59:04.4020302Z path: debug-*.zip 2025-08-14T21:59:04.4020467Z name: artifact 2025-08-14T21:59:04.4020611Z region: us-east-1 2025-08-14T21:59:04.4020761Z env: 2025-08-14T21:59:04.4020906Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:04.4021201Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:04.4021499Z DEVICE_NAME: 2025-08-14T21:59:04.4021650Z DEVICE_TYPE: 2025-08-14T21:59:04.4021791Z ##[endgroup] 2025-08-14T21:59:04.6403552Z No files were found with the provided path: debug-*.zip. No artifacts will be uploaded. 2025-08-14T21:59:04.6573001Z ##[group]Run # shellcheck disable=SC2156 2025-08-14T21:59:04.6573254Z # shellcheck disable=SC2156 2025-08-14T21:59:04.6573594Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \; 2025-08-14T21:59:04.6578035Z shell: /usr/bin/bash -e {0} 2025-08-14T21:59:04.6578216Z env: 2025-08-14T21:59:04.6578374Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:04.6578677Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:04.6579041Z DEVICE_NAME: 2025-08-14T21:59:04.6579191Z DEVICE_TYPE: 2025-08-14T21:59:04.6579329Z ##[endgroup] 2025-08-14T21:59:04.8289269Z Prepare all required actions 2025-08-14T21:59:04.8289646Z Getting action download info 2025-08-14T21:59:04.9560056Z ##[group]Run ./.github/actions/upload-utilization-stats 2025-08-14T21:59:04.9560284Z with: 2025-08-14T21:59:04.9560430Z job_id: 48128301909 2025-08-14T21:59:04.9560800Z job_name: linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T21:59:04.9561192Z workflow_name: inductor-periodic 2025-08-14T21:59:04.9561382Z workflow_run_id: 16976338999 2025-08-14T21:59:04.9561551Z workflow_attempt: 1 2025-08-14T21:59:04.9561699Z env: 2025-08-14T21:59:04.9561833Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:04.9562108Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:04.9562405Z DEVICE_NAME: 2025-08-14T21:59:04.9562543Z DEVICE_TYPE: 2025-08-14T21:59:04.9562683Z ##[endgroup] 2025-08-14T21:59:04.9574957Z ##[group]Run echo "workflow_id: 16976338999" 2025-08-14T21:59:04.9575188Z echo "workflow_id: 16976338999" 2025-08-14T21:59:04.9575386Z echo "workflow_attempt: 1" 2025-08-14T21:59:04.9575593Z echo "workflow_Name: inductor-periodic" 2025-08-14T21:59:04.9575802Z echo "job_id: 48128301909" 2025-08-14T21:59:04.9576205Z echo "job_name: linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)" 2025-08-14T21:59:04.9576621Z echo "artifact_prefix: " 2025-08-14T21:59:04.9576812Z python3 --version 2025-08-14T21:59:04.9581672Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:59:04.9581899Z env: 2025-08-14T21:59:04.9582047Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:04.9582324Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:04.9582613Z DEVICE_NAME: 2025-08-14T21:59:04.9582762Z DEVICE_TYPE: 2025-08-14T21:59:04.9582911Z ##[endgroup] 2025-08-14T21:59:04.9606259Z workflow_id: 16976338999 2025-08-14T21:59:04.9610548Z workflow_attempt: 1 2025-08-14T21:59:04.9610936Z workflow_Name: inductor-periodic 2025-08-14T21:59:04.9611225Z job_id: 48128301909 2025-08-14T21:59:04.9611656Z job_name: linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx) 2025-08-14T21:59:04.9612208Z artifact_prefix: 2025-08-14T21:59:04.9617467Z Python 3.9.23 2025-08-14T21:59:04.9647253Z ##[group]Run nick-fields/retry@v3.0.0 2025-08-14T21:59:04.9647455Z with: 2025-08-14T21:59:04.9647612Z shell: bash 2025-08-14T21:59:04.9647752Z timeout_minutes: 5 2025-08-14T21:59:04.9647907Z max_attempts: 5 2025-08-14T21:59:04.9648064Z retry_wait_seconds: 30 2025-08-14T21:59:04.9648373Z command: set -eu python3 -m pip install python-dateutil==2.8.2 boto3==1.35.42 pandas==2.1.3 dataclasses_json==0.6.7 2025-08-14T21:59:04.9648711Z polling_interval_seconds: 1 2025-08-14T21:59:04.9648894Z warning_on_retry: true 2025-08-14T21:59:04.9649054Z continue_on_error: false 2025-08-14T21:59:04.9649212Z env: 2025-08-14T21:59:04.9649349Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:04.9649620Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:04.9649902Z DEVICE_NAME: 2025-08-14T21:59:04.9650046Z DEVICE_TYPE: 2025-08-14T21:59:04.9650188Z ##[endgroup] 2025-08-14T21:59:05.2143184Z Defaulting to user installation because normal site-packages is not writeable 2025-08-14T21:59:05.2727692Z Collecting python-dateutil==2.8.2 2025-08-14T21:59:05.2896842Z Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB) 2025-08-14T21:59:05.9419294Z Collecting boto3==1.35.42 2025-08-14T21:59:05.9463263Z Downloading boto3-1.35.42-py3-none-any.whl (139 kB) 2025-08-14T21:59:06.2956500Z Collecting pandas==2.1.3 2025-08-14T21:59:06.3066922Z Downloading pandas-2.1.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.3 MB) 2025-08-14T21:59:06.4184629Z Requirement already satisfied: dataclasses_json==0.6.7 in /home/ec2-user/.local/lib/python3.9/site-packages (0.6.7) 2025-08-14T21:59:06.4196190Z Requirement already satisfied: six>=1.5 in /usr/lib/python3.9/site-packages (from python-dateutil==2.8.2) (1.15.0) 2025-08-14T21:59:06.4229067Z Requirement already satisfied: s3transfer<0.11.0,>=0.10.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.4) 2025-08-14T21:59:06.4230039Z Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /usr/lib/python3.9/site-packages (from boto3==1.35.42) (0.10.0) 2025-08-14T21:59:06.4235076Z Requirement already satisfied: botocore<1.36.0,>=1.35.42 in /home/ec2-user/.local/lib/python3.9/site-packages (from boto3==1.35.42) (1.35.99) 2025-08-14T21:59:07.0187650Z Collecting numpy<2,>=1.22.4 2025-08-14T21:59:07.0245932Z Downloading numpy-1.26.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB) 2025-08-14T21:59:07.1396399Z Requirement already satisfied: pytz>=2020.1 in /usr/lib/python3.9/site-packages (from pandas==2.1.3) (2022.7.1) 2025-08-14T21:59:07.1634239Z Collecting tzdata>=2022.1 2025-08-14T21:59:07.1672196Z Downloading tzdata-2025.2-py2.py3-none-any.whl (347 kB) 2025-08-14T21:59:07.1790028Z Requirement already satisfied: typing-inspect<1,>=0.4.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (0.9.0) 2025-08-14T21:59:07.1790764Z Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from dataclasses_json==0.6.7) (3.26.1) 2025-08-14T21:59:07.1833590Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/lib/python3.9/site-packages (from botocore<1.36.0,>=1.35.42->boto3==1.35.42) (1.25.10) 2025-08-14T21:59:07.1920272Z Requirement already satisfied: packaging>=17.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses_json==0.6.7) (25.0) 2025-08-14T21:59:07.1995191Z Requirement already satisfied: typing-extensions>=3.7.4 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (4.14.1) 2025-08-14T21:59:07.1996007Z Requirement already satisfied: mypy-extensions>=0.3.0 in /home/ec2-user/.local/lib/python3.9/site-packages (from typing-inspect<1,>=0.4.0->dataclasses_json==0.6.7) (1.1.0) 2025-08-14T21:59:07.3124673Z Installing collected packages: python-dateutil, tzdata, numpy, pandas, boto3 2025-08-14T21:59:11.0805996Z Attempting uninstall: boto3 2025-08-14T21:59:11.0811113Z Found existing installation: boto3 1.35.33 2025-08-14T21:59:11.0871139Z Uninstalling boto3-1.35.33: 2025-08-14T21:59:11.0878584Z Successfully uninstalled boto3-1.35.33 2025-08-14T21:59:11.1283424Z Successfully installed boto3-1.35.42 numpy-1.26.4 pandas-2.1.3 python-dateutil-2.8.2 tzdata-2025.2 2025-08-14T21:59:12.0322736Z Command completed after 1 attempt(s). 2025-08-14T21:59:12.0379658Z ##[group]Run python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \ 2025-08-14T21:59:12.0380104Z python3 -m tools.stats.upload_utilization_stats.upload_utilization_stats \ 2025-08-14T21:59:12.0380414Z  --workflow-run-id "16976338999" \ 2025-08-14T21:59:12.0380639Z  --workflow-name "inductor-periodic" \ 2025-08-14T21:59:12.0380865Z  --workflow-run-attempt "1" \ 2025-08-14T21:59:12.0381066Z  --job-id "48128301909" \ 2025-08-14T21:59:12.0381645Z  --job-name "linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)" \ 2025-08-14T21:59:12.0382085Z  --local-path "" \ 2025-08-14T21:59:12.0382274Z  --artifact-prefix "" 2025-08-14T21:59:12.0386888Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:59:12.0387116Z env: 2025-08-14T21:59:12.0387269Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:12.0387667Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:12.0387970Z DEVICE_NAME: 2025-08-14T21:59:12.0388246Z DEVICE_TYPE: 2025-08-14T21:59:12.0388445Z ##[endgroup] 2025-08-14T21:59:12.8801557Z repo: pytorch/pytorch 2025-08-14T21:59:12.8804396Z Search for test log in s3 bucket: ossci-utilization 2025-08-14T21:59:12.8805740Z Downloading logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301909.zip 2025-08-14T21:59:12.8806403Z extracting usage_log.txt from zip file logs-test-dynamic_cpu_inductor_huggingface-1-1-linux.8xlarge.amx_48128301909.zip 2025-08-14T21:59:12.8809336Z Converted Log Model: UtilizationMetadata: 2025-08-14T21:59:12.8810455Z UtilizationMetadata(level='metadata', workflow_id='16976338999', job_id='48128301909', workflow_name='inductor-periodic', job_name='linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)', usage_collect_interval=1.0, data_model_version=1.5, start_at=1755207308, gpu_count=0, cpu_count=32, gpu_type=None, error=None) 2025-08-14T21:59:12.8813417Z [Db Segments] detected pytest cmd: 9, generated segments: 9 2025-08-14T21:59:12.8813768Z [db model] Peek db timeseries 2025-08-14T21:59:12.8817733Z :{ 2025-08-14T21:59:12.8817990Z "created_at": 1755208752, 2025-08-14T21:59:12.8818191Z "type": "utilization", 2025-08-14T21:59:12.8818379Z "tags": [ 2025-08-14T21:59:12.8818527Z "record" 2025-08-14T21:59:12.8818667Z ], 2025-08-14T21:59:12.8818820Z "time_stamp": 1755207308, 2025-08-14T21:59:12.8819004Z "repo": "pytorch/pytorch", 2025-08-14T21:59:12.8819187Z "workflow_id": 16976338999, 2025-08-14T21:59:12.8819356Z "run_attempt": 1, 2025-08-14T21:59:12.8819514Z "job_id": 48128301909, 2025-08-14T21:59:12.8819701Z "workflow_name": "inductor-periodic", 2025-08-14T21:59:12.8820143Z "job_name": "linux-jammy-cpu-py3.9-gcc11-periodic-dynamo-benchmarks / test (dynamic_cpu_inductor_huggingface, 1, 1, linux.8xlarge.amx)", 2025-08-14T21:59:12.8820592Z "json_data": "{}" 2025-08-14T21:59:12.8820749Z } 2025-08-14T21:59:12.8821070Z Writing 1 documents to S3 ossci-utilization/util_metadata/v_1.5/pytorch/pytorch/16976338999/1/48128301909/metadata 2025-08-14T21:59:12.8821585Z Done! Finish writing document to S3 ossci-utilization/util_metadata/v_1.5/pytorch/pytorch/16976338999/1/48128301909/metadata 2025-08-14T21:59:12.8822101Z Writing 284 documents to S3 ossci-utilization/util_timeseries/v_1.5/pytorch/pytorch/16976338999/1/48128301909/time_series 2025-08-14T21:59:12.8822629Z Done! Finish writing document to S3 ossci-utilization/util_timeseries/v_1.5/pytorch/pytorch/16976338999/1/48128301909/time_series 2025-08-14T21:59:12.9822011Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main 2025-08-14T21:59:12.9822315Z with: 2025-08-14T21:59:12.9822470Z env: 2025-08-14T21:59:12.9822624Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:12.9822947Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:12.9823292Z DEVICE_NAME: 2025-08-14T21:59:12.9823449Z DEVICE_TYPE: 2025-08-14T21:59:12.9823608Z ##[endgroup] 2025-08-14T21:59:12.9834497Z ##[group]Run set -eou pipefail 2025-08-14T21:59:12.9834748Z set -eou pipefail 2025-08-14T21:59:12.9834938Z  2025-08-14T21:59:12.9835183Z echo "Holding runner for 2 hours until all ssh sessions have logged out" 2025-08-14T21:59:12.9835495Z for _ in $(seq 1440); do 2025-08-14T21:59:12.9835727Z  # Break if no ssh session exists anymore 2025-08-14T21:59:12.9835961Z  if [ "$(who)" = "" ]; then 2025-08-14T21:59:12.9836158Z  break 2025-08-14T21:59:12.9836361Z  fi 2025-08-14T21:59:12.9836519Z  echo "." 2025-08-14T21:59:12.9836694Z  sleep 5 2025-08-14T21:59:12.9836861Z done 2025-08-14T21:59:12.9842104Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:59:12.9842477Z env: 2025-08-14T21:59:12.9842669Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:12.9843034Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:12.9843401Z DEVICE_NAME: 2025-08-14T21:59:12.9843641Z DEVICE_TYPE: 2025-08-14T21:59:12.9843819Z ##[endgroup] 2025-08-14T21:59:12.9867776Z Holding runner for 2 hours until all ssh sessions have logged out 2025-08-14T21:59:12.9946281Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty 2025-08-14T21:59:12.9946846Z # ignore expansion of "docker ps -q" since it could be empty 2025-08-14T21:59:12.9947114Z # shellcheck disable=SC2046 2025-08-14T21:59:12.9947333Z docker stop $(docker ps -q) || true 2025-08-14T21:59:12.9947535Z # Prune all of the docker images 2025-08-14T21:59:12.9947737Z docker system prune -af 2025-08-14T21:59:12.9951377Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:59:12.9951600Z env: 2025-08-14T21:59:12.9951750Z GIT_DEFAULT_BRANCH: main 2025-08-14T21:59:12.9952037Z DOCKER_CONTAINER_ID: bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:12.9952330Z DEVICE_NAME: 2025-08-14T21:59:12.9952491Z DEVICE_TYPE: 2025-08-14T21:59:12.9952638Z ##[endgroup] 2025-08-14T21:59:23.7976369Z bbffe2680397 2025-08-14T21:59:24.0765837Z Deleted Containers: 2025-08-14T21:59:24.0766233Z bbffe2680397fb8e75b6b7d6504b35802fdecc4e3e9eaf00b2a24bf695f9c86b 2025-08-14T21:59:24.0766475Z 2025-08-14T21:59:30.9159263Z Deleted Images: 2025-08-14T21:59:30.9165274Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image:pytorch-linux-jammy-py3.9-gcc11-inductor-benchmarks-bfa89110622ba7202628e9faac705f183070defe 2025-08-14T21:59:30.9166260Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/ci-image@sha256:4236794baba289041d240d08fd393bbd57497c3012e5e0ccd9fd98f61ebf35c6 2025-08-14T21:59:30.9166783Z deleted: sha256:0899ae453036ee7a91795ea95b1db61000579eeb74b140edab5976919ee64bbe 2025-08-14T21:59:30.9167211Z deleted: sha256:aa7b544271e9ba3105dabd1afb12e315887018f3471e03135c1d50e64cc550c4 2025-08-14T21:59:30.9167577Z deleted: sha256:4c685831817cc2fc6dfdfda1726df1f402222d8cdccc40daad3198cf8b17e3f4 2025-08-14T21:59:30.9167951Z deleted: sha256:cedf3fb09a62e68c6d7e22cedbce12e77166a50649d0269200ee0efce8a57b88 2025-08-14T21:59:30.9168309Z deleted: sha256:1b3a9a237b4153f8f523a85cead9d36e29717eb57182e2f75069788681627d95 2025-08-14T21:59:30.9168669Z deleted: sha256:67bd313103dfbe7fe0172e6f4f7ee420fad9743a64a1cc1cd20bc22250d3602c 2025-08-14T21:59:30.9169032Z deleted: sha256:b17820137ada46a2a726c67aa08cce73d2ead7c95db08575cf5e69bedb4b600d 2025-08-14T21:59:30.9169832Z deleted: sha256:b16c9bc40cc1cf924638323aece4168d6332cfae212dad2a431a584a44fe967c 2025-08-14T21:59:30.9170211Z deleted: sha256:ab35ed781133eb4aaa1b2478aea73fb80dc71bceffbe474b55e1a60fc6c5ffbe 2025-08-14T21:59:30.9170598Z deleted: sha256:b9d0b0720dd9c0bcb4f174ae6770a7c2fe540c6983872180f3a5e18300434cdb 2025-08-14T21:59:30.9170985Z deleted: sha256:f5d1a4f32d90030cc174d73b579758d28f95c992a8cf21360e5addee99dea169 2025-08-14T21:59:30.9171341Z deleted: sha256:4af408141f8591f4b69cef9b425b6caa3c4cbc62ced38b5d08f3150f0c8ff449 2025-08-14T21:59:30.9171712Z deleted: sha256:e0019e5c461051e54a9af37ae22b49cfd2c2e5366da57a20304f6ef89171a3b3 2025-08-14T21:59:30.9172070Z deleted: sha256:542f999b2cfc965b97861645356840864e9946fa2fa40f1f5c4c45684e91c239 2025-08-14T21:59:30.9172425Z deleted: sha256:633629aa3d4ae6472e222a1c0b2ceb729b0d84ccb48e12d52ba2d2987c9063e1 2025-08-14T21:59:30.9172783Z deleted: sha256:ea645aba1ba54baac43713f3df7f1b89dd119764a747273897eb2931fea42856 2025-08-14T21:59:30.9173151Z deleted: sha256:1f50e367efff88c7182b9dc3ff618c1cf7bd34edf2f31805e268c50fac02a627 2025-08-14T21:59:30.9173515Z deleted: sha256:aff22d7ae43d842befa617e2e5f9878d09a82b67c362b0c44a40a4c88be92120 2025-08-14T21:59:30.9173866Z deleted: sha256:4275d4addb77b473ed40194e42918cf2aeb484d1d8e25cf54d374392643a095c 2025-08-14T21:59:30.9174217Z deleted: sha256:66471f6c8dc869455ff193909110d824b5d65f7383877a7d0face6331b21fff3 2025-08-14T21:59:30.9174638Z deleted: sha256:8cfd2d55570494ff2b993725f5eb13d0440a5698fa905823ca1677d2d16febb8 2025-08-14T21:59:30.9174994Z deleted: sha256:5c8cf8b9c4a76f679994decc8800bc6eefd258a8dc6293a714d5e100fea3a1bc 2025-08-14T21:59:30.9175402Z deleted: sha256:1acc162c6b9de62d13ce7fd33bb9b134458f7e7dbe996e5442e0047ec8f70c80 2025-08-14T21:59:30.9175766Z deleted: sha256:044bab98f3bceb1948c626ce6bdd19d3ec8f9c5ad42a4f635dd685a7ae9c9024 2025-08-14T21:59:30.9176134Z deleted: sha256:2acb11a9448f13c2c2d29c4d0d4013e046862bd019cf5ec9fe04bdf35299f1dd 2025-08-14T21:59:30.9176489Z deleted: sha256:8e7b56334416233f301944000dec16952e13bb69296cc80e1031bfecaf6e7f9d 2025-08-14T21:59:30.9176886Z deleted: sha256:4a4d1ec727c43389a601aefccdaeff6b3bf54c0daefb12e0c2098c3e18b383ba 2025-08-14T21:59:30.9177232Z deleted: sha256:8b9ca4276331196a2f03c2fa3a87422d2042cf06011b49368c2335be7da829c1 2025-08-14T21:59:30.9177572Z deleted: sha256:5076357fd3cc8b06ed54a0f692362a38f1ebafa4843c0b0bf8021f9021d2e583 2025-08-14T21:59:30.9177912Z deleted: sha256:f9451fa0842798e2a67c059fda5124cafb401801bb8c40d03ae736ff3ef5ed20 2025-08-14T21:59:30.9178252Z deleted: sha256:52b716f02091d6af6b79e7b2e1f5bbd7391235993d415c7a852d6752220c8b65 2025-08-14T21:59:30.9178589Z deleted: sha256:748225161c361d3779c96eb7ae5ea0c33d35311f9445c371d62616b98e3426e8 2025-08-14T21:59:30.9178929Z deleted: sha256:5eeda1478a46d8d58267e8917422eb0a182a40c8bdfb4bfe0869923f8114c770 2025-08-14T21:59:30.9179268Z deleted: sha256:66d4cebb04304f556dd191b425a876f7dbbcde8c3c647af4ef47c10804e51f5a 2025-08-14T21:59:30.9179643Z deleted: sha256:0b526447174d22890be2bc866228e40989483b1102a0430b4ab3ad16dc6c7787 2025-08-14T21:59:30.9179986Z deleted: sha256:1aa31d55f8f9bb51f1eb702ba7d46ceda8290ed90e8e8cf299bb8a9179bf2ae2 2025-08-14T21:59:30.9180338Z deleted: sha256:dd1f47c8dc7518f303a91fc8aae81a512caff53987d5a89a378bb24c1c6d7707 2025-08-14T21:59:30.9180681Z deleted: sha256:d60f9527fcb284e73795a37d4f536badd451a2eade4c9314ebe549d31efcc876 2025-08-14T21:59:30.9181024Z deleted: sha256:f23ad0355704751b0f71a8900169354e3bf23a7b3f5fa2cd9b2478a561bfbb45 2025-08-14T21:59:30.9181371Z deleted: sha256:10e7acf6460743fcad0c1fff0bbd01158fbeb88151621c1e15ae5994f1c8ef55 2025-08-14T21:59:30.9181716Z deleted: sha256:f674e3067e97f1407f4cd55202d4c0c8641f02811550e65a00a875fc19354b75 2025-08-14T21:59:30.9182053Z deleted: sha256:8a9c75c896425ccd25101f0cf39316bec7779111954f44df726842bf583e907b 2025-08-14T21:59:30.9182435Z deleted: sha256:9730d30edfcaa135287479d80f1720b39c6f728228df6d0eb7f095e917cc16b6 2025-08-14T21:59:30.9182776Z deleted: sha256:2787e13cf97e870ca65312526c3000163ebf3da20fe59e5f5d53b1aeb4fb424b 2025-08-14T21:59:30.9183184Z deleted: sha256:d61197909174795bd69f8d5f534f1b086065d36b7aa6c5a50744eca6f8d6b12b 2025-08-14T21:59:30.9183537Z deleted: sha256:ecdfbb81e95b2ae2c8e9ab4ca72ba8564095caabb0512a47da8f866923f71bff 2025-08-14T21:59:30.9183889Z deleted: sha256:cd2d7c644df243742a0c0349af0d37570c06fdd1711ddc367e79514757a6d5cc 2025-08-14T21:59:30.9184235Z deleted: sha256:6703ab1ced70b30a87660c0dd778fe95fb90b04ed8461c2a331272aa54eb3499 2025-08-14T21:59:30.9184581Z deleted: sha256:b7088ce49d7df1d6fb18eee5fc5664e637c5649c89e581d972c76a83f60d0a62 2025-08-14T21:59:30.9184928Z deleted: sha256:d0d2786658af9907d8c4ecfa84fa9e2bd07131257264395b804deef744a5c39c 2025-08-14T21:59:30.9185275Z deleted: sha256:d46baf72d8e570e6004c6f95131cea6ede27eb01c213d8c1e8b263ab95fdfe95 2025-08-14T21:59:30.9185613Z deleted: sha256:0219ea0bd0e38d169ed596ed80807b0f70b609ec5f886d671c249d10575dff2c 2025-08-14T21:59:30.9185953Z deleted: sha256:77d1a1f15cf8ae85a4c5495d800378c307967004360814810fd13b07a74aee5e 2025-08-14T21:59:30.9186291Z deleted: sha256:47c77d89ce8782a94a6f5435b1611a76b47f830153ba4b462d3e08dcbdaa40f7 2025-08-14T21:59:30.9186638Z deleted: sha256:d5120b2e61fb0ccc32a2ad02fc0b2b908bc69f1f174268bde3d26d79ce46f046 2025-08-14T21:59:30.9186978Z deleted: sha256:65626052fd7e03a8e90c72072a54f0eaa43788cfcb0835ffb98b700be89b0567 2025-08-14T21:59:30.9187320Z deleted: sha256:05c09c0832c35f0128e0258b1d3069d7bb4b94ce58239faba5d585e49c34e904 2025-08-14T21:59:30.9187676Z deleted: sha256:2d6749fb2c30585eebb1d97e99318434ec34e0f7a4414e552fd4a44175f86839 2025-08-14T21:59:30.9188013Z deleted: sha256:2d65e2932810021e5b3cfedd89cfd851dd47fce63fbe5dc6959e59f3d8a98499 2025-08-14T21:59:30.9188392Z deleted: sha256:b2e71ddacad35b6caa3a77429bab51b654f6acaccc9e9263f1cb43edb8c53ac3 2025-08-14T21:59:30.9188735Z deleted: sha256:632a43100a629c40972b4da95fbbb581f29fe8b073a96386c72931d27ffbbefa 2025-08-14T21:59:30.9189072Z deleted: sha256:11964e5f5833fdf2bcc61c52f33d5aebf9b5504c6792baf58beb96b90398d10a 2025-08-14T21:59:30.9189404Z deleted: sha256:f0c1cb4c9e4655464b9b62b6589ac5005c2392213765ab4175bd61e3f6462643 2025-08-14T21:59:30.9189749Z deleted: sha256:5113aaee4b4d5ee45b58bcee467ac314112b02e4c4e5e9c3cc7a236dd308e9de 2025-08-14T21:59:30.9190096Z deleted: sha256:9cdc88c7b7fe728e15c72d0e8eef813ace31905b4b317a0a23f1334b6a22e604 2025-08-14T21:59:30.9190432Z deleted: sha256:8056a3da01752a91095e2d0afd80b625172f0915f22f7d998b9b926b9462dc5f 2025-08-14T21:59:30.9190759Z deleted: sha256:8a99968112e0edd39c242f3452b05d167911724468fdd9b18d11a8f5fa9c3ac8 2025-08-14T21:59:30.9191097Z deleted: sha256:6f70653bcfea9c1dd39aba76713adac0ac8f6f4c202387ff86a3ffe45d2079f2 2025-08-14T21:59:30.9191447Z deleted: sha256:9a0ed45f26188ecbfcf7658f46e29922b441969b2aded64d1d6b287b6de2e49c 2025-08-14T21:59:30.9191775Z deleted: sha256:f84c75780b110e68f7593fe9592456387118761b365a954a105aee72016adeac 2025-08-14T21:59:30.9192111Z deleted: sha256:1a5a81f8cbb945eee96e25ee8b4958d7140bb6751b86bc2e4a6aa9e18a16846c 2025-08-14T21:59:30.9192450Z deleted: sha256:7e072dc6aa8c1831ddc97ba8229235081976cb8036c06ee1320b33606e03f9a4 2025-08-14T21:59:30.9192796Z deleted: sha256:369af3627df8ecb48c51ea4fd3267e561b2f6821075ddce314e9485494447f16 2025-08-14T21:59:30.9193144Z deleted: sha256:4d49b99f2eee0f82788e33a9c771f75b1411b0b70ce47771fc1b3bc160f23961 2025-08-14T21:59:30.9193500Z deleted: sha256:fe04dcb9c711f36f9ed1df5b2d0854d30dc5abaa6e6cd493b85d4c2e2d2c3e1b 2025-08-14T21:59:30.9193857Z deleted: sha256:4800771a0435c52d6e480540ffa8a65ecc51fdc82a91302c1a373e6021bc37ca 2025-08-14T21:59:30.9194194Z deleted: sha256:90a2bf02e851326fc70d05470553ed33e578342d6e06bfa0cfaf331c4079b7e4 2025-08-14T21:59:30.9194405Z 2025-08-14T21:59:30.9194493Z Total reclaimed space: 51.8GB 2025-08-14T21:59:30.9267230Z Post job cleanup. 2025-08-14T21:59:30.9294158Z Post job cleanup. 2025-08-14T21:59:31.0032286Z [command]/usr/bin/git version 2025-08-14T21:59:31.0066969Z git version 2.47.1 2025-08-14T21:59:31.0094264Z Copying '/home/ec2-user/.gitconfig' to '/home/ec2-user/actions-runner/_work/_temp/40d13371-2817-40c7-8067-98d377c6e982/.gitconfig' 2025-08-14T21:59:31.0109251Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/40d13371-2817-40c7-8067-98d377c6e982' before making global git config changes 2025-08-14T21:59:31.0111612Z Adding repository directory to the temporary git global config as a safe directory 2025-08-14T21:59:31.0116462Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch 2025-08-14T21:59:31.0163607Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand 2025-08-14T21:59:31.0197998Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :" 2025-08-14T21:59:31.0516875Z Entering 'android/libs/fbjni' 2025-08-14T21:59:31.0571572Z Entering 'third_party/FP16' 2025-08-14T21:59:31.0623036Z Entering 'third_party/FXdiv' 2025-08-14T21:59:31.0678292Z Entering 'third_party/NNPACK' 2025-08-14T21:59:31.0731429Z Entering 'third_party/NVTX' 2025-08-14T21:59:31.0790398Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-14T21:59:31.0842497Z Entering 'third_party/XNNPACK' 2025-08-14T21:59:31.0909833Z Entering 'third_party/aiter' 2025-08-14T21:59:31.0972968Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T21:59:31.1031942Z Entering 'third_party/benchmark' 2025-08-14T21:59:31.1086646Z Entering 'third_party/composable_kernel' 2025-08-14T21:59:31.1151795Z Entering 'third_party/cpp-httplib' 2025-08-14T21:59:31.1203250Z Entering 'third_party/cpuinfo' 2025-08-14T21:59:31.1260165Z Entering 'third_party/cudnn_frontend' 2025-08-14T21:59:31.1315490Z Entering 'third_party/cutlass' 2025-08-14T21:59:31.1372570Z Entering 'third_party/fbgemm' 2025-08-14T21:59:31.1421947Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-14T21:59:31.1474756Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-14T21:59:31.1532839Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-14T21:59:31.1586393Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-14T21:59:31.1644218Z Entering 'third_party/fbgemm/external/googletest' 2025-08-14T21:59:31.1697756Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-14T21:59:31.1751167Z Entering 'third_party/fbgemm/external/json' 2025-08-14T21:59:31.1803468Z Entering 'third_party/flash-attention' 2025-08-14T21:59:31.1860713Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T21:59:31.1914838Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-14T21:59:31.1981840Z Entering 'third_party/flatbuffers' 2025-08-14T21:59:31.2034802Z Entering 'third_party/fmt' 2025-08-14T21:59:31.2090376Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-14T21:59:31.2149910Z Entering 'third_party/gloo' 2025-08-14T21:59:31.2201882Z Entering 'third_party/googletest' 2025-08-14T21:59:31.2250811Z Entering 'third_party/ideep' 2025-08-14T21:59:31.2301356Z Entering 'third_party/ideep/mkl-dnn' 2025-08-14T21:59:31.2361993Z Entering 'third_party/ittapi' 2025-08-14T21:59:31.2418085Z Entering 'third_party/kineto' 2025-08-14T21:59:31.2471645Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T21:59:31.2518149Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T21:59:31.2574938Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T21:59:31.2632399Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T21:59:31.2683174Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T21:59:31.2731011Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T21:59:31.2788038Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T21:59:31.2842202Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T21:59:31.2893789Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T21:59:31.2948434Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T21:59:31.3002531Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T21:59:31.3055713Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T21:59:31.3115286Z Entering 'third_party/kleidiai' 2025-08-14T21:59:31.3170103Z Entering 'third_party/mimalloc' 2025-08-14T21:59:31.3224342Z Entering 'third_party/nlohmann' 2025-08-14T21:59:31.3277392Z Entering 'third_party/onnx' 2025-08-14T21:59:31.3341678Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-14T21:59:31.3401329Z Entering 'third_party/opentelemetry-cpp' 2025-08-14T21:59:31.3453211Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T21:59:31.3514614Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T21:59:31.3567904Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T21:59:31.3616299Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T21:59:31.3673102Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T21:59:31.3724472Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T21:59:31.3778455Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T21:59:31.3832034Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T21:59:31.3887531Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T21:59:31.3936029Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T21:59:31.4011861Z Entering 'third_party/pocketfft' 2025-08-14T21:59:31.4070430Z Entering 'third_party/protobuf' 2025-08-14T21:59:31.4117097Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-14T21:59:31.4170797Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-14T21:59:31.4227700Z Entering 'third_party/psimd' 2025-08-14T21:59:31.4281416Z Entering 'third_party/pthreadpool' 2025-08-14T21:59:31.4331085Z Entering 'third_party/pybind11' 2025-08-14T21:59:31.4396597Z Entering 'third_party/python-peachpy' 2025-08-14T21:59:31.4452814Z Entering 'third_party/sleef' 2025-08-14T21:59:31.4505992Z Entering 'third_party/tensorpipe' 2025-08-14T21:59:31.4554234Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-14T21:59:31.4606164Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-14T21:59:31.4656687Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-14T21:59:31.4711596Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T21:59:31.4762037Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T21:59:31.4841774Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader 2025-08-14T21:59:31.4869693Z http.https://github.com/.extraheader 2025-08-14T21:59:31.4880781Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader 2025-08-14T21:59:31.4913698Z [command]/usr/bin/git submodule foreach --recursive sh -c "git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :" 2025-08-14T21:59:31.5229903Z Entering 'android/libs/fbjni' 2025-08-14T21:59:31.5267071Z http.https://github.com/.extraheader 2025-08-14T21:59:31.5303503Z Entering 'third_party/FP16' 2025-08-14T21:59:31.5331717Z http.https://github.com/.extraheader 2025-08-14T21:59:31.5364789Z Entering 'third_party/FXdiv' 2025-08-14T21:59:31.5400648Z http.https://github.com/.extraheader 2025-08-14T21:59:31.5432770Z Entering 'third_party/NNPACK' 2025-08-14T21:59:31.5468776Z http.https://github.com/.extraheader 2025-08-14T21:59:31.5503286Z Entering 'third_party/NVTX' 2025-08-14T21:59:31.5534255Z http.https://github.com/.extraheader 2025-08-14T21:59:31.5568580Z Entering 'third_party/VulkanMemoryAllocator' 2025-08-14T21:59:31.5603241Z http.https://github.com/.extraheader 2025-08-14T21:59:31.5636649Z Entering 'third_party/XNNPACK' 2025-08-14T21:59:31.5673806Z http.https://github.com/.extraheader 2025-08-14T21:59:31.5716430Z Entering 'third_party/aiter' 2025-08-14T21:59:31.5749671Z http.https://github.com/.extraheader 2025-08-14T21:59:31.5782887Z Entering 'third_party/aiter/3rdparty/composable_kernel' 2025-08-14T21:59:31.5817888Z http.https://github.com/.extraheader 2025-08-14T21:59:31.5863408Z Entering 'third_party/benchmark' 2025-08-14T21:59:31.5894069Z http.https://github.com/.extraheader 2025-08-14T21:59:31.5928203Z Entering 'third_party/composable_kernel' 2025-08-14T21:59:31.5963699Z http.https://github.com/.extraheader 2025-08-14T21:59:31.6007169Z Entering 'third_party/cpp-httplib' 2025-08-14T21:59:31.6039619Z http.https://github.com/.extraheader 2025-08-14T21:59:31.6077191Z Entering 'third_party/cpuinfo' 2025-08-14T21:59:31.6109671Z http.https://github.com/.extraheader 2025-08-14T21:59:31.6146307Z Entering 'third_party/cudnn_frontend' 2025-08-14T21:59:31.6177332Z http.https://github.com/.extraheader 2025-08-14T21:59:31.6211793Z Entering 'third_party/cutlass' 2025-08-14T21:59:31.6241432Z http.https://github.com/.extraheader 2025-08-14T21:59:31.6292789Z Entering 'third_party/fbgemm' 2025-08-14T21:59:31.6325973Z http.https://github.com/.extraheader 2025-08-14T21:59:31.6364684Z Entering 'third_party/fbgemm/external/asmjit' 2025-08-14T21:59:31.6396490Z http.https://github.com/.extraheader 2025-08-14T21:59:31.6431992Z Entering 'third_party/fbgemm/external/composable_kernel' 2025-08-14T21:59:31.6465153Z http.https://github.com/.extraheader 2025-08-14T21:59:31.6505617Z Entering 'third_party/fbgemm/external/cpuinfo' 2025-08-14T21:59:31.6536854Z http.https://github.com/.extraheader 2025-08-14T21:59:31.6576189Z Entering 'third_party/fbgemm/external/cutlass' 2025-08-14T21:59:31.6607548Z http.https://github.com/.extraheader 2025-08-14T21:59:31.6653424Z Entering 'third_party/fbgemm/external/googletest' 2025-08-14T21:59:31.6683032Z http.https://github.com/.extraheader 2025-08-14T21:59:31.6726345Z Entering 'third_party/fbgemm/external/hipify_torch' 2025-08-14T21:59:31.6755294Z http.https://github.com/.extraheader 2025-08-14T21:59:31.6787191Z Entering 'third_party/fbgemm/external/json' 2025-08-14T21:59:31.6821577Z http.https://github.com/.extraheader 2025-08-14T21:59:31.6859821Z Entering 'third_party/flash-attention' 2025-08-14T21:59:31.6894828Z http.https://github.com/.extraheader 2025-08-14T21:59:31.6931057Z Entering 'third_party/flash-attention/csrc/composable_kernel' 2025-08-14T21:59:31.6963076Z http.https://github.com/.extraheader 2025-08-14T21:59:31.7005782Z Entering 'third_party/flash-attention/csrc/cutlass' 2025-08-14T21:59:31.7036374Z http.https://github.com/.extraheader 2025-08-14T21:59:31.7087672Z Entering 'third_party/flatbuffers' 2025-08-14T21:59:31.7121144Z http.https://github.com/.extraheader 2025-08-14T21:59:31.7159449Z Entering 'third_party/fmt' 2025-08-14T21:59:31.7188471Z http.https://github.com/.extraheader 2025-08-14T21:59:31.7225724Z Entering 'third_party/gemmlowp/gemmlowp' 2025-08-14T21:59:31.7261041Z http.https://github.com/.extraheader 2025-08-14T21:59:31.7294112Z Entering 'third_party/gloo' 2025-08-14T21:59:31.7327604Z http.https://github.com/.extraheader 2025-08-14T21:59:31.7363267Z Entering 'third_party/googletest' 2025-08-14T21:59:31.7399580Z http.https://github.com/.extraheader 2025-08-14T21:59:31.7433623Z Entering 'third_party/ideep' 2025-08-14T21:59:31.7470182Z http.https://github.com/.extraheader 2025-08-14T21:59:31.7505306Z Entering 'third_party/ideep/mkl-dnn' 2025-08-14T21:59:31.7536173Z http.https://github.com/.extraheader 2025-08-14T21:59:31.7582622Z Entering 'third_party/ittapi' 2025-08-14T21:59:31.7613715Z http.https://github.com/.extraheader 2025-08-14T21:59:31.7643228Z Entering 'third_party/kineto' 2025-08-14T21:59:31.7680722Z http.https://github.com/.extraheader 2025-08-14T21:59:31.7713462Z Entering 'third_party/kineto/libkineto/third_party/dynolog' 2025-08-14T21:59:31.7749277Z http.https://github.com/.extraheader 2025-08-14T21:59:31.7785356Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/DCGM' 2025-08-14T21:59:31.7818296Z http.https://github.com/.extraheader 2025-08-14T21:59:31.7852832Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/cpr' 2025-08-14T21:59:31.7884447Z http.https://github.com/.extraheader 2025-08-14T21:59:31.7917545Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/fmt' 2025-08-14T21:59:31.7950405Z http.https://github.com/.extraheader 2025-08-14T21:59:31.7989515Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags' 2025-08-14T21:59:31.8023179Z http.https://github.com/.extraheader 2025-08-14T21:59:31.8054818Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/gflags/doc' 2025-08-14T21:59:31.8084404Z http.https://github.com/.extraheader 2025-08-14T21:59:31.8122618Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/glog' 2025-08-14T21:59:31.8161734Z http.https://github.com/.extraheader 2025-08-14T21:59:31.8195679Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/googletest' 2025-08-14T21:59:31.8224891Z http.https://github.com/.extraheader 2025-08-14T21:59:31.8263650Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/json' 2025-08-14T21:59:31.8294199Z http.https://github.com/.extraheader 2025-08-14T21:59:31.8330865Z Entering 'third_party/kineto/libkineto/third_party/dynolog/third_party/pfs' 2025-08-14T21:59:31.8366956Z http.https://github.com/.extraheader 2025-08-14T21:59:31.8407662Z Entering 'third_party/kineto/libkineto/third_party/fmt' 2025-08-14T21:59:31.8439526Z http.https://github.com/.extraheader 2025-08-14T21:59:31.8475750Z Entering 'third_party/kineto/libkineto/third_party/googletest' 2025-08-14T21:59:31.8507299Z http.https://github.com/.extraheader 2025-08-14T21:59:31.8547026Z Entering 'third_party/kleidiai' 2025-08-14T21:59:31.8584862Z http.https://github.com/.extraheader 2025-08-14T21:59:31.8619057Z Entering 'third_party/mimalloc' 2025-08-14T21:59:31.8645310Z http.https://github.com/.extraheader 2025-08-14T21:59:31.8682611Z Entering 'third_party/nlohmann' 2025-08-14T21:59:31.8716916Z http.https://github.com/.extraheader 2025-08-14T21:59:31.8753086Z Entering 'third_party/onnx' 2025-08-14T21:59:31.8795714Z http.https://github.com/.extraheader 2025-08-14T21:59:31.8835874Z Entering 'third_party/onnx/third_party/pybind11' 2025-08-14T21:59:31.8876544Z http.https://github.com/.extraheader 2025-08-14T21:59:31.8917774Z Entering 'third_party/opentelemetry-cpp' 2025-08-14T21:59:31.8956406Z http.https://github.com/.extraheader 2025-08-14T21:59:31.8990364Z Entering 'third_party/opentelemetry-cpp/third_party/benchmark' 2025-08-14T21:59:31.9022403Z http.https://github.com/.extraheader 2025-08-14T21:59:31.9058067Z Entering 'third_party/opentelemetry-cpp/third_party/googletest' 2025-08-14T21:59:31.9089825Z http.https://github.com/.extraheader 2025-08-14T21:59:31.9128446Z Entering 'third_party/opentelemetry-cpp/third_party/ms-gsl' 2025-08-14T21:59:31.9161611Z http.https://github.com/.extraheader 2025-08-14T21:59:31.9200673Z Entering 'third_party/opentelemetry-cpp/third_party/nlohmann-json' 2025-08-14T21:59:31.9226747Z http.https://github.com/.extraheader 2025-08-14T21:59:31.9263634Z Entering 'third_party/opentelemetry-cpp/third_party/opentelemetry-proto' 2025-08-14T21:59:31.9296095Z http.https://github.com/.extraheader 2025-08-14T21:59:31.9327505Z Entering 'third_party/opentelemetry-cpp/third_party/opentracing-cpp' 2025-08-14T21:59:31.9362962Z http.https://github.com/.extraheader 2025-08-14T21:59:31.9394786Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp' 2025-08-14T21:59:31.9425764Z http.https://github.com/.extraheader 2025-08-14T21:59:31.9460494Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/civetweb' 2025-08-14T21:59:31.9493097Z http.https://github.com/.extraheader 2025-08-14T21:59:31.9530929Z Entering 'third_party/opentelemetry-cpp/third_party/prometheus-cpp/3rdparty/googletest' 2025-08-14T21:59:31.9562774Z http.https://github.com/.extraheader 2025-08-14T21:59:31.9600530Z Entering 'third_party/opentelemetry-cpp/tools/vcpkg' 2025-08-14T21:59:31.9636110Z http.https://github.com/.extraheader 2025-08-14T21:59:31.9690022Z Entering 'third_party/pocketfft' 2025-08-14T21:59:31.9721152Z http.https://github.com/.extraheader 2025-08-14T21:59:31.9755213Z Entering 'third_party/protobuf' 2025-08-14T21:59:31.9792899Z http.https://github.com/.extraheader 2025-08-14T21:59:31.9828179Z Entering 'third_party/protobuf/third_party/benchmark' 2025-08-14T21:59:31.9859634Z http.https://github.com/.extraheader 2025-08-14T21:59:31.9896563Z Entering 'third_party/protobuf/third_party/googletest' 2025-08-14T21:59:31.9929732Z http.https://github.com/.extraheader 2025-08-14T21:59:31.9971578Z Entering 'third_party/psimd' 2025-08-14T21:59:32.0001994Z http.https://github.com/.extraheader 2025-08-14T21:59:32.0033913Z Entering 'third_party/pthreadpool' 2025-08-14T21:59:32.0070681Z http.https://github.com/.extraheader 2025-08-14T21:59:32.0105276Z Entering 'third_party/pybind11' 2025-08-14T21:59:32.0137009Z http.https://github.com/.extraheader 2025-08-14T21:59:32.0170719Z Entering 'third_party/python-peachpy' 2025-08-14T21:59:32.0200280Z http.https://github.com/.extraheader 2025-08-14T21:59:32.0237100Z Entering 'third_party/sleef' 2025-08-14T21:59:32.0273889Z http.https://github.com/.extraheader 2025-08-14T21:59:32.0314598Z Entering 'third_party/tensorpipe' 2025-08-14T21:59:32.0346742Z http.https://github.com/.extraheader 2025-08-14T21:59:32.0383431Z Entering 'third_party/tensorpipe/third_party/googletest' 2025-08-14T21:59:32.0416839Z http.https://github.com/.extraheader 2025-08-14T21:59:32.0453967Z Entering 'third_party/tensorpipe/third_party/libnop' 2025-08-14T21:59:32.0488893Z http.https://github.com/.extraheader 2025-08-14T21:59:32.0526545Z Entering 'third_party/tensorpipe/third_party/libuv' 2025-08-14T21:59:32.0560505Z http.https://github.com/.extraheader 2025-08-14T21:59:32.0596301Z Entering 'third_party/tensorpipe/third_party/pybind11' 2025-08-14T21:59:32.0629928Z http.https://github.com/.extraheader 2025-08-14T21:59:32.0665952Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang' 2025-08-14T21:59:32.0704729Z http.https://github.com/.extraheader 2025-08-14T21:59:32.0855426Z A job completed hook has been configured by the self-hosted runner administrator 2025-08-14T21:59:32.0867508Z ##[group]Run '/home/ec2-user/runner-scripts/after_job.sh' 2025-08-14T21:59:32.0870611Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0} 2025-08-14T21:59:32.0870887Z ##[endgroup] 2025-08-14T21:59:32.0946988Z [!ALERT!] Swap in detected! [!ALERT!] 2025-08-14T21:59:40.6136749Z [!ALERT!] Swap out detected [!ALERT!] 2025-08-14T21:59:55.6675903Z Cleaning up orphan processes